*Yann LeCuns commentary.* https://facebook/yann.lecun/posts/10152184295832143 My comments on the IBM TrueNorth neural net chip. IBM has an article in Science about their TrueNorth neural net chip. The chip has 4096 processors, each of which has 256 integrate-and-fire spiking neurons (with binary output) each with 256 inputs. Each processor can compute all 256 neurons 1000 times per second. Neuron states are binary (spikes) and synaptic weights are 9-bit signed integers with a 4-bit time delay. The overall peak performance is 4096x256x256x1000 = 266 GSops (billion synaptic operations per second). The power consumption is supposed to be around 100mW. The NYT and Wired have articles about the announcement in which I express my skepticism of the approach. I feel that I need to clarify my position. First, Im all in favor of building special-purpose chips to run neural nets. In fact, I have worked on this myself. My NYU lab started the NeuFlow project, which implements convolutional neural nets in hardware (see neuflow.org/ ). the NeuFlow project was initially implemented at NYU on a Xilinx Virtex-6 FPGA (yann.lecun/exdb/publis/index.html#farabet-suml-11 ), and later turned into an ASIC design by Eugenio Culurciello and his team at Purdue (yann.lecun/exdb/publis/index.html#pham-mwscas-12 ). The NeuFlow design implements convolutional networks which we know can produce state of the art performance on a number of vision tasks such as object recognition, semantic segmentation, obstacle detection, etc. So we know that the hardware could be practically useful and was worth the effort and the cost (mostly paid for by ONR). Now, what wrong with TrueNorth? My main criticism is that TrueNorth implements networks of integrate-and-fire spiking neurons. This type of neural net that has never been shown to yield accuracy anywhere close to state of the art on any task of interest (like, say recognizing objects from the ImageNet dataset). Spiking neurons have binary outputs (like neurons in the brain). The advantage of spiking neurons is that you dont need multipliers (since the neuron states are binary). But to get good results on a task like ImageNet you need about 8 bit of precision on the neuron states. To get this kind of precision with spiking neurons requires to wait multiple cycles so the spikes average out. This slows down the overall computation. Getting excellent results on ImageNet is easily achieved with a convolutional net with something like 8 to 16 bit precision on the neuron states and weights. This requires to implement hardware multipliers. But the structure of convolutional nets make it easy to organize the computation so the traffic with the memory is minimized. An ASIC implementing the NeuFlow architecture with the same fabrication technology as TrueNorth would be capable of about 3000 GSops (billion synaptic operations per second) in about 1 Watt. Thats 3.3e-13 Joules/Sop. This is similar to the TrueNorths 3.7e-13 J/Sop (266 GSops in 0.1 W). These estimates are somewhat speculative, but fairly conservative. Im assuming 16 bit states and weights for NeuFlow, which is considerably more accurate than TrueNorths 1-bit states and 9-bit weights. Further improvements can be obtained with dynamic precision (variable bit depth). We know we can get away with 8 bits in the early layers without much loss of accuracy. NYT: nytimes/2014/08/08/science/new-computer-chip-is-designed-to-work-like-the-brain.html Wired: wired/2014/08/ibm-unveils-a-brain-like-chip-with-4000-processor-cores/ Science (paywall): sciencemag.org/content/345/6197/668 For a comparison of NeuFlow ConvNets with AER (spiking neuron) ConvNet implementation, look at this: yann.lecun/exdb/publis/index.html#farabet-frontiersin-12
Posted on: Fri, 08 Aug 2014 17:59:12 +0000
Trending Topics
Recently Viewed Topics
© 2015