Learning Algorithm

Artificial Neural Networks

by Tim Dorney

Learning Algorithm

In practical circumstances, ANNs must be able to adapt to disturbances or variations. It is essential that any in situ ANN be able to learn as quickly as possible. A learning rate that is too slow can be detrimental in the sense that the ANN may take too long to respond to the environment. The learning scheme must provide a fairly rapid ability to adjust; otherwise, the ANN will be too unresponsive to be of use in time-critical applications. Several learning schemes are investigated in Dorney. The Delta Rule Updating was used for this investigation since real-time results were not required, and its ease of implementation.

The error, in supervised learning, is the sum of the squared difference between the target outputs, t_pk, and the ANN outputs, o_pk, for each of the input patterns, p. This error is also averaged for the total number of input patterns, P.

Error = 1/2P {SUM(for p = 1 to P) SUM(for k = 1 to K) (t_pk - o_pk)^2}  (8)

The objective is to reduce the system error below some given tolerance or reach the global minimum. The technique outlined here attempts to reduce the error using information from the first order derivative of the error. The approach belongs to a class of gradient methods.

For each iteration, all of the input patterns are sent to the neural network. The error is summed for all input patterns. The summed difference between the outputs and the target is used to adjust the weights on each of the links between the inputs and the neural network nodes.

The emphasis in increasing the rate of learning is to adjust the learning step, eta. The learning step determines how large a step is made from the present state to the next. If eta is small, then the learning rate is slow. If a larger eta is selected, the learning rate increases, but the possibility the system will enter into an undesired state which oscillates or is stuck at a fixed value is also increased.

Using delta rule updating introduced by Rumelhart, Hinton and Williams, the weights for each of the links of the FLN are updated after all patterns are presented to the network. The method attempts to decrease the error between the outputs of the ANN and the target patterns. By following the slope of the error curve, an appropriate change is made to the weights. A simple outline is offered.

0) Generate weights for all links based on a random number generator. Calculate the functional enhancements for all of the input patterns.

START LOOP 1

START LOOP 2

1) Get an input pattern, its associated functional enhancement, and desired target.

2) Feed the functional-link net ANN with the input pattern and the functional enhancement, and store the output.

3) Sum the ERROR by ERROR = ERROR + (OUT - TARGET)^2.

4) Sum delta update by DELTA = DELTA + ETA*(TARGET-OUT)*OUT*(1-OUT) for each of the weights.

5) Goto LOOP 2 until all patterns and targets have been presented once.

6) Update the weights by WEIGHTS = WEIGHTS + DELTA/P, where P is the number of patterns.

7) Goto LOOP 1 until the ERROR minimum threshold is reached, or a certain number of iterations has been exceeded.

The Delta rule updating maintains a fixed eta, and has the drawback that it does not guarantee that the error is reduced at each iteration. In fact, depending on the learning step, the error can get stuck at a fixed value (a ridge) or even oscillate. The fixed eta scheme represents only one method of updating the weights. A more appropriate question might be how to vary eta to speed the learning process. This poses an optimization problem which is addressed in Dorney.

Previous Page Next Page

jchen@micro.ti.com
tdorney@ti.com
sparr@owlnet.rice.edu