Publications

The principle of maximizing mutual information is applied to learning overcomplete and recurrent representations. The underlying model con- sists of a network of input units driving a larger number of output units with recurrent interactions. In the limit of zero noise, the network is de- terministic and the mutual information can be related to the entropy of the output units. Maximizing this entropy with respect to both the feed- forward connections as well as the recurrent interactions results in simple learning rules for both sets of parameters. The conventional independent components (ICA) learning algorithm can be recovered as a special case where there is an equal number of output units and no recurrent con- nections. The application of these new learning rules is illustrated on a simple two-dimensional input example.