Return to MODULE PAGE

McCulloch-Pitts Neurons: Intermediate Level (page 8)

Michael Marsalli: Author

Hebbian Learning

The next important idea in the theory of neural networks was contained in the book The Organization of Behavior written by Donald O. Hebb in 1949.  In this book Hebb proposed a rule for the modification of a synapse which would provide a physiological basis for learning. Here is Hebb's rule, as presented in his book.

"When an axon of cell A is near enough to excite cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased."

In our terms, if an input to an MCP neuron is repeatedly or persistently 1 when the output is 1, then the weight of that synapse is increased. Hebb's rule provides a way to explain how a neuron can "learn" to produce a desired output from given inputs. The neuron increases the strength of synapses which fire when the neuron should fire.  Thus we have a rule by which neural networks can "learn" to produce desired outputs. We call this rule a Hebbian learning rule.

This same principle can be applied to interconnected collections of neurons, which Hebb called "cell assemblies."  These cell assemblies might represent information in the brain, and groups of interconnected cell assemblies might corrsepond to perceptions or thoughts.  Indeed, Hebb is perhaps the first person to use the term "connectionism" in the context of brain function.

While Hebb did not state a rule for weakening a synaptic connection, it is easy to provide such a rule in our terms.  If an input to an MCP neuron is repeatedly or persistently the opposite of the output, then the weight of that synapse is decreased.

In order to apply Hebb's idea to our MCP neuron, we need a mathematical form of Hebbian learning.

Two input MCP neurons with hebbian learning

Let's explore the use of a learning rule in the case of a two input MCP neuron.  As before we use x1 and x2 for the two inputs, w1 and w2 for the corresponding weights, and T for the threshold.  In order to simplify the computations, we introduce an input x0, which will always be 1, and its associated weight  w0, which will be -T.  Note that
w1*x1 + w2*x2 * T  is the same as  -T*1 +  w1*x1 + w2*x2 * 0.  So the output of the MCP neuron will be 1 if  -T*1 +  w1*x1 + w2*x2  * 0, and it will be 0 if  -T*1 +  w1*x1 + w2*x2 < 0.  We will also use A for the actual output of the MCP neuron and D for the desired output.

Now for any pair of inputs x1 and x2, there are three possiblilites for the output corresponding to these inputs: D > A, D < A, or D = A.  Note that  D > A is the same as D = 1 and A = 0, because D and A can only be 1 or 0. Likewise D < A is the same as  D = 0 and A = 1.

We will use the following "learning" rule for the modification of the weights.

If D > A, then we replace wi by wi + xi
If D < A, then we replace wi by wi - xi
If D = A, do nothing.

Let's examine the consequences in each of the three cases to see how this rule implements "learning."

Case 1. D > A   (D = 1 and A = 0)  In this case, if xi = 1, then wi will increase by 1.  And if  if xi = 0, then wi will remain the same.  This is a model of Hebbian learning, because the strength of the synaptic connections, i.e. the weights, are increased when the input is  1 and the desired out put is 1.

Case 2.  D < A   (D = 0 and A = 1)  In this case, if xi = 1, then wi will decrease by 1.  And if  if xi = 0, then wi will remain the same.  So in this case the strength of the synaptic connections, i.e. the weights, are decreased when the input is 1 and the desired out put is  0.

Case 3. If D = A, then no change is made in the weights, because the desired output matches the actual output.

We proceed as follows.  We will give the inputs (1,1), (1,0), (0,1), and (0,0) to the MCP neuron in this order, sarting with (1,1).  After each input we will compare the actual output A for this input to the desired output D for this same input.  Then we will apply the learning rule to modify the weights.  We now technically have a new MCP neuron if the weights are modified.  We now give the next input to the modified MCP neuron, and again apply the learning rule. After we reach the input (0,0), we start with (1,1) again. When will this procedure stop ?  When the desired output matches the actual output for each pair of inputs on one complete cycle through (1,1) to (0,0). Let's look at an example.