Neural Network applications using Beowulf

[Robert G. Brown]

On Wed, 20 Nov 2002, Eray Ozkural wrote:

> Personally, I think our brains, logically, operate in a way more like a GA and 
> less like an ANN; however I'm also sure that it is unlike anything we have 
> come up with. 

Encouraged by some positive feedback, he continues, but of course feel
free to delete:-)

It isn't necessarily hard science, but it seems like a reasonable time
to plug the book "The Lucifer Principle" by Bloom, which is one of the
most interesting books I've read in years (and I read a LOT:-).  The
book's theme is that much of human and animal affairs, including the
parts that are difficult to impossible to explain by means of strict
Darwinian evolution (altruism, suicide) is the result of memetic
evolution, not just, or even mostly, genetic.  Rather genetic and
memetic co-evolution.  One of his more interesting extensions of this
idea (which is not of course original to Bloom, although he does a
lovely job of presenting it) is that "identifiable" organizations in
human society itself are structured like a neural net, with humans
functioning as very complex "neurons", with a very generalized parallel
processing power engendered by the interhuman connections and
information pathways.  These human neural nets (HNNs) are themselves
memetic superorganisms, in constant competition with other memetic
superorganisms and struggling to maintain identity, attract "food" in
the form of new followers and other resources, reinforce and strengthen
its most important neural entities, part of which comes from the
superorganism diverting rewards and resources to them and part of which
is a >>genetic<< bonus -- we are genetically co-evolved to live longer
and remain stronger if we are "important", and to die when we are not.

Very interesting and provocative ideas, although obviously
superorganisms like the many churches and organized religions would
strongly rebel against the notion that they have "evolved" into their
current states by survival of the fittest along with very identifiable
instances of historical "crossover and mutation" while the losers at
some point proved weaker, less flexible, and hence were "killed" by the
winners.  The interminable crusades, the cold war, the gulf war, all
interpretable as battles between superorganisms.

I personally think that the metaphor extends strongly into both open
source software and the beowulf phenomenon.  Open source software is
informationally strong and robust because its development process fully
exploits both genetic optimization algorithms and the NN organization.

The GA component is driven by the open exchange of information and
patches (crossover), a broad base of smart people contributing new ideas
and codons (Lysenko-heretic mutations), and "fatal" competition both
between program revisions and similar programs (leading to a modestly
ruthless elimination of the unfit both locally and in time globally, as
bugs are fixed, useless programs disappear, and new programs appear).
The NN component is important in several ways.  Without many layers of
contribution and development in some sort of hierarchy, programs would
be random and chaotic and fail to work together -- the NN-like hierarchy
associated with this list, with the distribution organizations, with
Linus and the kernel development team keeps this from happening and by
feeding BACK constraints, creates the "survival/fitness landscape" that
ensure that everything maintains an acceptable level of integrability
and mutuality.  

This list tolerates some diversity of discussion (as robust evolution
requires sometimes radical or not obviously connected new
ideas/mutations), but also contains mechanisms for "pruning" truly
irrelevant discussions.  We are all having meta-sex as we read and
write, with the basic codons for beowulf construction constantly being
passed down to new beowulfs, new codons being fed back to the list from
existing beowulfs and other (parallel) codon sources, the codon-pool
gradually broadening and co-evolving with needs and technology.  codons
that were appropriate for 10BT ethernet hubs are discarded; new codons
appropriate for switched GbE are experimented with, all in a
non-stationary fitness landscape with an underlying economic basis and a
highly nonlinear and variable reward structure.

This is basically Open Source's fundamental advantage.  Open Source
development is structurally a GA with widespread, even promiscuous,
exchange of information.  Closed source has a much narrower information
base, and these days that base is very much driven by influx from the
open source community.

Ordinarily, one sex models with high mutation rates evolve toward
optimal fitness more rapidly initially than multisex models, but then
the high mutation rate (the only source of genetic variability) trades
off against genetic stability and one sex models often fail to approach
a local optimum closely or the global optimum at all.  Multisex models
take longer to get going as selective gene sorting can initially be
slower than a random search, but it gently explores space in a much more
thorough way, with an N^3 advantage over N random samplings in the
volume explored.  In a high dimensional model, this can make a big
difference, but slowly.

I personally think that this is very much evident in the ongoing
evolution of and competition between operating systems and software.
The open source universe creates the web.  The web grows and open source
web products improve.  Closed source companies embrace the web, stealing
the codons, and immediately begin to mutate them WITHOUT sharing the
mutations back into the open source pool.  In doing so, they create
protected feeding grounds where their products are less optimized and
gradually drift from the open source evolution, but their mutation
eliminates the need to compete directly.  And many other examples, of
course -- more than just a metaphor, a real model.

There are some important and interesting nuances to the process.
Software co-evolves with hardware (which in turn is driven by a
superoganismal competition between corporate entities interacting with
LOTS of superentities in a ruthlessly competitive landscape) and madness
lies in trying to fully untangle the web of dependencies .  Also, the
creation of "new" software products, although arguably in many cases
informed by an informational crossover, is more arguably a higher order
event (or lower order event, as you prefer:-) driven by wetware, often
in a single individual.  Humans are very complex neuron analogs.

All of which leads me to my actual reply and comments:  I think human
brains are more precisely like a MIX of GAs and NNs, not unlike this
picture of human society and its intertwined superorganisms.  Some
elements of the neurology of the brain are very homologous to FFNNs --
the pathways from ears to auditory cortex, for example, where there
exist layered, tonotopic maps in the cortex that are connected to and
driven by the nerves in the ears.  However, I suspect that the real
reason that "simple" NN models fail to explain much in the way of brain
function is that the brain uses "simple" FFNN's to build higher order
structures, and then links these structures into still higher order
structures, with lots of feedBACK loops.  It is hard to view brain
function per se (not higher order reasoning, but functioning at the
neural level) as a simple two-sex GA, as there is no mechanism, at least
with which I'm aware, that constitutes any sort of crossover or
mutation.

There does seem to be competition-driven pruning as I noted before, and
since there are single-sex genetic algorithms WITHOUT crossover (else
how would our humble friends the bacteria and the viruses evolve:-) one
could argue that this is something of a GA-driven process.  And of
course I firmly believe that it was GAs that led to the hard-soft wiring
of the brain itself -- the higher order structures referred to above
exist in posse at birth, and are then "activated" (further
self-organized and pruned) by use.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu