<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-forward-container">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
Dear colleagues,<br>
<br>
This is a discussion about well known techniques, not specifically
about whose work. We have had many papers about neural
networks. But we did not have sufficiently honest discussion on
well-known techniques. At least I hesitated very much to discuss
such a subject, because Profs. X, Y, Z used such techniques. This
lack of honesty has caused a lot of waste in resources, including
time (of our professors, researchers, postdocs, and graduate
students) and money (governments, private foundations, and
companies). Still, I am afraid that the following paragraphs
will make some well known researchers angry. For that reason, the
following discussion has identified myself (J. Weng) who should be
blamed for using some of the well-known techniques. I also made
mistakes. Please accept my apology. <br>
<br>
Please reply with your comments.<br>
<br>
---- some new paragraphs in the Brain Principles Manifesto ----<br>
<br>
Industrial and academic interests have been keen on a combination
of two things — easily understandable tests (e.g., G. Hinton et
al. NIPS 2012, congratulations!) and major companies are involved
(e.g., Google, thanks!). We have read statements like “our
results can be improved simply by waiting for faster GPUs and
bigger datasets to become available” (G. Hinton et al. NIPS
2012). However, the newly known brain principles have told us
that the ways to conduct such tests (e.g., ImageNet) will give
only vanishing gains that do not lead to a human-like zero error
rate, regardless how long the Moore’s Law can continue and how
many more static images are added to the training set. Why? All
such tests used static images in which objects mix with the
background. Such tests therefore prevent participating groups from
seriously considering autonomous object segmentation (free of
handcrafted object model). Through synapse maintenance (Y. Wang
et al. ICBM 2012), neurons in a human brain automatically cut off
inputs from background pixels if background pixels matched badly
compared with attended object pixels. Our babies spend much more
time in dynamic physical world than seeing static photos.<br>
<br>
Our industry should learn more powerful brain mechanisms that went
beyond conventional well-known, well-tested techniques. The
following gives some examples: <br>
<br>
(1) Deep Learning Networks (e.g., J. Weng et al. IJCNN 1992, Y.
LeCun et al. Proceedings of IEEE 1998, G. Hinton et al. NIPS 2012)
are not only biologically implausible but also functionally weak.
The brain uses a rich network of processing areas (e.g., Felleman
& Van Essen, Cerebral Cortex 1991) where connections are
almost always two-way (J. Weng, Natural and Artificial
Intelligence, 2012), not a cascade of modules as in the Deep
Learning Networks. Such a Deep Learning Network is not able to
conduct top-down attention in a cluttered scene (e.g., attention
to location or type in J. Weng, Natural and Artificial
Intelligence, 2012 or attention to more complex object shape as
reported in L. B. Smith et al. Developmental Science 2005). <br>
<br>
(2) Convolution (e.g., J. Weng et al. IJCNN 1992, Y. LeCun et al.
Proceedings of IEEE 1998, G. Hinton et al. NIPS 2012) is not only
biologically implausible, but also computationally weak. Why? All
feature neurons in the brain carry not only sensory information
but also motor information (e.g., Felleman & Van Essen,
Cerebral Cortex 1991) so that later-processing neurons become less
concrete and more abstract --- which is impossible to accomplish
using the shift-invariant convolution. Namely, convolution is
always location-concrete (even using max-pulling) and never
location-abstract. <br>
<br>
(3) Error back-propagation in neural networks (e.g., G. Hinton et
al. NIPS 2012) is not only biologically implausible (e.g., a baby
does not have error in his motors) but also damaging to long-term
memory because of its lack of match-based competition for
error-causality (such as those in SOM, LISSOM, and LCA as optimal
SOM). Even though the gradient vector identifies a neuron that
can reduce the current error, the current error is not the
business of that neuron at all and it must keep its own long-term
memory unchanged. That is why error back-propagation is well
known to be bad for incremental learning and requires research
assistants to try many guesses of initial weights (i.e., using the
test set as the training set!). Let us not be blinded by
artificial low error rates. <br>
<br>
Do our industry and public need another 20 years? <br>
<br>
---- end of the new paragraphs -----<br>
Full text:<br>
<br>
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="ProgId" content="Word.Document">
<meta name="Generator" content="Microsoft Word 14">
<meta name="Originator" content="Microsoft Word 14">
<link rel="File-List"
href="file://localhost/Users/weng/Library/Caches/TemporaryItems/msoclip/0/clip_filelist.xml">
<!--[if gte mso 9]><xml>
<o:DocumentProperties>
<o:Revision>0</o:Revision>
<o:TotalTime>0</o:TotalTime>
<o:Pages>1</o:Pages>
<o:Words>2116</o:Words>
<o:Characters>12065</o:Characters>
<o:Company>Michigan State University</o:Company>
<o:Lines>100</o:Lines>
<o:Paragraphs>28</o:Paragraphs>
<o:CharactersWithSpaces>14153</o:CharactersWithSpaces>
<o:Version>14.0</o:Version>
</o:DocumentProperties>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<link rel="themeData"
href="file://localhost/Users/weng/Library/Caches/TemporaryItems/msoclip/0/clip_themedata.xml">
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>JA</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
<w:UseFELayout/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="276">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Times;
panose-1:2 0 5 0 0 0 0 0 0 0;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:Times;
panose-1:2 0 5 0 0 0 0 0 0 0;
mso-font-charset:0;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:3 0 0 0 1 0;}
@font-face
{font-family:华文黑体;
mso-font-charset:80;
mso-generic-font-family:auto;
mso-font-pitch:variable;
mso-font-signature:16777863 135200768 235143169 0 262303 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-parent:"";
margin:0in;
margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:华文黑体;
mso-font-kerning:1.0pt;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;
text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-noshow:yes;
mso-style-priority:99;
color:purple;
mso-themecolor:followedhyperlink;
text-decoration:underline;
text-underline:single;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:华文黑体;
mso-font-kerning:1.0pt;}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:华文黑体;
mso-font-kerning:1.0pt;}
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:华文黑体;
mso-font-kerning:1.0pt;}
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast
{mso-style-priority:34;
mso-style-unhide:no;
mso-style-qformat:yes;
mso-style-type:export-only;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
mso-add-space:auto;
mso-pagination:widow-orphan;
font-size:12.0pt;
font-family:"Times New Roman";
mso-fareast-font-family:华文黑体;
mso-font-kerning:1.0pt;}
span.st
{mso-style-name:st;
mso-style-unhide:no;}
.MsoChpDefault
{mso-style-type:export-only;
mso-default-props:yes;
font-size:10.0pt;
mso-ansi-font-size:10.0pt;
mso-bidi-font-size:10.0pt;
mso-fareast-font-family:华文黑体;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;
mso-header-margin:.5in;
mso-footer-margin:.5in;
mso-paper-source:0;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:663899949;
mso-list-type:hybrid;
mso-list-template-ids:1431704628 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:78.0pt;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:114.0pt;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:150.0pt;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:186.0pt;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:222.0pt;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:258.0pt;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:294.0pt;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:330.0pt;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style><!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0in 5.4pt 0in 5.4pt;
mso-para-margin:0in;
mso-para-margin-bottom:.0001pt;
mso-pagination:widow-orphan;
font-size:10.0pt;
font-family:"Times New Roman";}
</style>
<![endif]--><!--StartFragment--><!--EndFragment-->The Brain Principles
Manifesto <br>
(Draft Version 4.5)<br>
<br>
March 21, 2015<br>
<br>
Historically, public acceptance of science was slow. For example,
Charles Darwin waited about 20 years (from the 1830s to 1858) to
publish his theory of evolution for fear of public reaction.
About 20 years later (by the 1870s) the scientific community and
much of the general public had accepted evolution as a fact. Of
course, the debate on evolution still goes on today. <br>
<br>
Is the public acceptance of science faster in modern days? Not
necessarily so, even though we have now better and faster means to
communicate. The primary reason is still the same but much more
severe—the remaining open scientific problems are more complex and
the required knowledge goes beyond a typical single person. <br>
<br>
For instance, network-like brain computation — connectionist
computation (e.g., J. McClelland and D. Rumelhart, Parallel
Distributed Processing, 1986) — has been long doubted and ignored
by industry. Deep convolutional networks appeared by at least
1980 (K. Fukushima). Max-pooling technique for deep convolutional
networks was published by 1992 (J. Weng et al.). However, Apple,
Baidu, Google, Microsoft, Samsung, and other major related
companies did not show considerable interest till after 2012.
That is a delay of about 20 years. The two techniques above are
not very difficult to understand. However, these two suddenly hot
techniques have already been proved obsolete by the discoveries of
more fundamental and effective principles of the brain, six of
which are intuitively explained below. <br>
<br>
Industrial and academic interests have been keen on a combination
of two things — easily understandable tests (e.g., G. Hinton et
al. NIPS 2012, congratulations!) and major companies are involved
(e.g., Google, thanks!). We have read statements like “our
results can be improved simply by waiting for faster GPUs and
bigger datasets to become available” (G. Hinton et al. NIPS
2012). However, the newly known brain principles have told us
that the ways to conduct such tests (e.g., ImageNet) will give
only vanishing gains that do not lead to a human-like zero error
rate, regardless how long the Moore’s Law can continue and how
many more static images are added to the training set. Why? All
such tests used static images in which objects mix with the
background. Such tests therefore prevent participating groups from
seriously considering autonomous object segmentation (free of
handcrafted object model). Through synapse maintenance (Y. Wang
et al. ICBM 2012), neurons in a human brain automatically cut off
inputs from background pixels if background pixels matched badly
compared with attended object pixels. Our babies spend much more
time in dynamic physical world than seeing static photos.<br>
<br>
Our industry should learn more powerful brain mechanisms that went
beyond conventional well-known, well-tested techniques. The
following gives some examples: <br>
<br>
(1) Deep Learning Networks (e.g., J. Weng et al. IJCNN 1992, Y.
LeCun et al. Proceedings of IEEE 1998, G. Hinton et al. NIPS 2012)
are not only biologically implausible but also functionally weak.
The brain uses a rich network of processing areas (e.g., Felleman
& Van Essen, Cerebral Cortex 1991) where connections are
almost always two-way (J. Weng, Natural and Artificial
Intelligence, 2012), not a cascade of modules as in the Deep
Learning Networks. Such a Deep Learning Network is not able to
conduct top-down attention in a cluttered scene (e.g., attention
to location or type in J. Weng, Natural and Artificial
Intelligence, 2012 or attention to more complex object shape as
reported in L. B. Smith et al. Developmental Science 2005). <br>
<br>
(2) Convolution (e.g., J. Weng et al. IJCNN 1992, Y. LeCun et al.
Proceedings of IEEE 1998, G. Hinton et al. NIPS 2012) is not only
biologically implausible, but also computationally weak. Why? All
feature neurons in the brain carry not only sensory information
but also motor information (e.g., Felleman & Van Essen,
Cerebral Cortex 1991) so that later-processing neurons become less
concrete and more abstract --- which is impossible to accomplish
using the shift-invariant convolution. Namely, convolution is
always location-concrete (even using max-pulling) and never
location-abstract. <br>
<br>
(3) Error back-propagation in neural networks (e.g., G. Hinton et
al. NIPS 2012) is not only biologically implausible (e.g., a baby
does not have error in his motors) but also damaging to long-term
memory because of its lack of match-based competition for
error-causality (such as those in SOM, LISSOM, and LCA as optimal
SOM). Even though the gradient vector identifies a neuron that
can reduce the current error, the current error is not the
business of that neuron at all and it must keep its own long-term
memory unchanged. That is why error back-propagation is well
known to be bad for incremental learning and requires research
assistants to try many guesses of initial weights (i.e., using the
test set as the training set!). Let us not be blinded by
artificial low error rates. <br>
<br>
Do our industry and public need another 20 years? <br>
<br>
On the other hand, neuroscience and neuropsychology have made many
advances by providing experimental data (e.g., Felleman & Van
Essen, Cerebral Cortex 1991). However, it has been well
recognized that these disciplines are data-rich and theory-poor.
The phenomena of brain circuits and brain behavior are extremely
rich. Many researchers in these areas use only local tools (e.g.,
attracters that can only be attracted into local extrema) and
consequently have been overwhelmed by the richness of brain
phenomena. A fundamental reason is that they miss the guidance
of the global automata theory of computer science, although
previous automata do not emerge. For example, X. -J. Wang et al.
Nature 2013 stated correctly that neurons of mixed selectivity
were rarely analyzed but have widely observed. However, the mixed
selectivity has already been well explained, as a special case, by
the new Emergent Turing Machine in Developmental Networks in a
theoretically complete way. The traditional Universal Turing
Machine is a theoretical model for modern-day computers --- how
computers work --- but they do not emerge. The mixed selectivity
of neurons in such a new kind of Turing Machine are caused by
emergent and beautiful brain circuits, but each neuron still uses
a simple similarity of inner product in its high dimensional and
dynamic input space. <br>
<br>
October 2011, a highly respected multi-disciplinary professor
kindly wrote: “I tell these students that they can work on brains
and do good science, or work on robots and do good engineering.
But if they try to do both at once, the result will be neither
good science nor good engineering.” How long does it take for the
industry and public to accept that the pessimistic view of the
brain was no longer true even then?<br>
<br>
The brain principles that have already been discovered could bring
fundamental changes in the way humans live, the way countries and
societies are organized, our industry, our economy, and the way
humans treat one another. <br>
<br>
The known brain principles have told us that the brain of anybody,
regardless of his education and experience, is fundamentally
shortsighted, in both space and time. Prof. Jonathan Haidt
documented well such shortsightedness in his book “The Righteous
Mind: Why Good People Are Divided by Politics and Religion”,
although not in terms of brain computation. <br>
<br>
In terms of brain computation, the circuits in your brain
self-wire beautifully and precisely according to your real-time
experience (the genome only regulates) and their various
invariance properties required for abstraction also largely depend
on experience. Serotonin (caused by, e.g., threats), dopamine
(caused by e.g., praise), and other neural transmitters quickly
bias these circuits so that neurons for more long-term thoughts
lost in competition to fire. Furthermore, such bias has a
long-term effect. Therefore, you make long-term mistakes but you
still feel you are right. Everybody is like that. Depending on
experience, shortsightedness varies in terms of subject matter. <br>
<br>
Traditionally, many domain experts think that computers and brain
appear to use very different principles. Naturally emerging
Turing Machine in Developmental Networks that has been
mathematically proved (see J. Weng, Brain as an Emergent Finite
Automaton: A Theory and Three Theorems, IJIS, 2015) should change
our intuition.<br>
The new result proposed the following six brain principles:<br>
1. The developmental program (genome-like, task-nonspecific)
regulates the development (i.e., lifetime learning) of a
task-nonspecific “brain-like” network —— Developmental Network.
The Developmental Network is of general-purpose—can learn any
body-capable tasks, in principle. Not only pattern recognition.<br>
2. The brain’s images are naturally sensed images of cluttered
scenes where many objects mix. In typical machine training (e.g.,
Krizhevsky et al. NIPS 2012), each training image has a bounding
box drawn around each object to learn, which is not the case for
a human baby. Neurons in the Developmental Network automatically
learn object segmentation through synapse maintenance. <br>
3. The brain’s muscles have multiple subareas where each
subarea represents either declarative knowledge (e.g., abstract
concepts such as location, type, scale, etc.) or non-declarative
knowledge (e.g., driving a car or riding a bicycle). Not just
discrete class labels in global classification. <br>
4. Each brain in the physical world is at least is a Super
Turing Machine in a Developmental Network. Every area in the
network emerges (does not statically exist, see M. Sur et al.
Nature 2000 and P. Voss, Frontiers in Psychology 2013) using a
unified area function whose feature development is nonlinear but
free of local minima, contrary to engineering intuition --- not
convolution; not error back-propagation.<br>
5. The brain’s Developmental Network learns
incrementally—taking one-pair of sensory pattern and motor pattern
at a time to update the “brain” and discarding the pair
immediately after. Namely, a real brain has only one pair of
stereoscopic retinas which cannot store more than one pair of
image. Batch learning (i.e., learn before test) is not scalable:
Without a mistake in an early test, a student cannot learn how to
correct the mistake later. <br>
6. The brain’s Developmental Network is always optimal—Each
network update in real time computes the maximum likelihood
estimate of the “brain”, conditioned on the limited computational
resources and the limited learning experience in its “life” so
far. One should not use the test set as a training set: report
only the best network after trying many networks on the test set.
<br>
<br>
The logic completeness of a brain is (partially, not all)
understood by a Universal Turing Machine in a Developmental
Network. This emergent automaton brain model proposes that each
brain is an automaton, but also very different from all
traditional symbolic automata because it programs
itself—emergent. No traditional Turing Machine can program
itself but a brain Turing Machine does. <br>
<br>
The automaton brain model has predicted that brain circuits
dynamically and precisely record the statistics of experience,
roughly consistent with neural anatomy (e.g., Felleman & Van
Essen, Cerebral Cortex, 1991). In particular, the model predicted
that “shifting attention between `humans’ and `vehicles’
dramatically changes brain representation of all categories” (J.
Gallant et al. Nature Neuroscience, 2013) and that human attention
“can regulate the activity of their neurons in the medial temporal
lobe” (C. Koch et al. Nature, 2010). The “place” cells work of
the 2014 Nobel Prize in Physiology or Medicine implies that
neurons encode exclusively bottom-up information (place). The
automaton brain model challenges such a view: Neurons represent a
combination of both bottom-up (e.g., place) and top-down context
(e.g., goal) as reported by Koch et al. and Gallant et al. <br>
<br>
Unfortunately, the automaton brain model implies that all
neuroscientists and neural network researchers are unable to
understand the brain of their studies without a rigorous training
in automata theory. For example, traditional models for nervous
systems and neural networks focus on pattern recognition and do
not have the capabilities of a grounded symbol system (e.g.,
“rulefully combining and recombining,” Stevan Harnad, Physica D,
1990). The automata theory deals with such capabilities. Does
this new knowledge stun our students and researchers or guide them
so their time is better spent?<br>
<br>
Brain automata would enable us to see answers to a wide variety of
important questions, some of which are raised below. The automaton
brain model predicts that there is no absolute right or wrong in
any brain but its environmental experiences wire and rewire the
brain. We do not provide yes/no answers here, only raise
questions. <br>
<br>
How can our industry and public understand that the door for
understanding brains has opened for them? How can they see the
economical outlooks that this opportunity leads them to?<br>
<br>
How should our educational system reform to prepare our many
bright minds for the new brain age? Has our government been
prompt to properly respond to this modern call from the nature?<br>
<br>
How should our young generation act for the new opportunity that
is unfolding before their eyes? Is a currently narrowly defined
academic degree sufficient for their career?<br>
<br>
How can everybody take advantage of the new knowledge about his
own brain so that he is more successful in his career, including
statesmen, officials, educators, attorneys, entrepreneurs,
doctors, technicians, artists, workers, drivers, and other mental
and manual workers?<br>
<br>
Regardless where we are and what we do, we are all governed by the
same set of brain principles. Everybody’s brain automatically
programs itself.<br>
<br>
---- end of the manifesto ----<br>
<br>
-John<br>
<pre class="moz-signature" cols="72">--
--
Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:weng@cse.msu.edu">weng@cse.msu.edu</a>
URL: <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.cse.msu.edu/%7Eweng/">http://www.cse.msu.edu/~weng/</a>
----------------------------------------------
</pre>
<br>
</div>
<br>
</body>
</html>