Forward Forward Algorithm in Python

Training Neural Networks Without Backprop via Hinton's Forward Forward Algorithm

Quick Summary

We code the paper that introduced the Forward-Forward deep learning algorithm in Python.

Here are some free gpu credits.

Here’s 10 dollars off your next international transfer.

Forward Forward Algorithm Summary

1.0 Paper Introduction

Forward-Forward (Hinton, 2022)¹ is a biologically-plausible backpropagation alternative that achieves ~96% (Löwe, 2023)² accuracy on MNIST without flowing gradients.

Abstract for The Forward-Forward Algorithm: Some Preliminary Investigations (Hinton, 2022)

It vastly outperforms Target Propagation (Bengio, 2014)³, another biologically plausible backprop alternative, that reaches ~39% accuracy on MNIST.

This is the third part in our Alternatives To Backpropagation series:

Target Propagation: A Biologically Plausible Neural Network Training Algorithm

Belief Propagation is an Obscure Alternative to Backpropagation for Training Reasoning Models

1.1 Hinton’s Algorithm

Forward-Forward algorithm summary. Taken from (Hinton, 2022)

Hinton’s idea is super simple: we run two forward passes. The first pass permits the network learn correct data points. The second pass is inundated with incorrect data to permit the network conceptualize false data*.

*It’s somewhat analogous to training a GAN that discriminates between good and bad inputs.

Each layer is trained in isolation (using gradient descent), so gradients don’t flow backwards through the network.

Goodness is the sum of square outputs of ReLU

(Hinton, 2022) proposes goodness as a metric for measuring an individual layer’s ability to contrast correct and false data.