ffnn-vs-rnn

Adaptation of the untyped neural network implementation at the start of Justin Le's blog series on dependent types for machine learning.

I adapt the feed-forward neural network (FFNN) to implement a modified neural network in the style of recurrent neural networks (RNN.)

The problem that the model is trained to solve in all three examples: learn the infinite repeating sequence 0, 1, 2, 0, 1, 2, ...

Note that the back-propagation method used for both RNN examples (the logistic-final and softmax-final ones) is not back-propagation through time (BPTT) as described in 9. 1. 2. Training of Jurafsky. See also this blog post

-- FFNN on directly encoded numerals; sample inputs are 3-grams
> displayTraining 0.25 training_data       
-- RNN on directly encoded numerals; sample inputs are single integers 
> displayRNNTraining 0.25 rnn_training_data
-- RNN on indirectly encoded numerals; sample inputs / outputs are one-hot 3-by-1 vectors
> displayRNNTraining_softmax 0.25 rnn_training_data_onehot

(11/25/2020) The BPTT module via src/ now contains an implementation, from scratch, of back-propagation through time (BPTT) along with a discussion, in-line with the code of backward_phase, of how each quantity adjusting the parameters and propagating error attribution to the prior hidden state, from the current time, is derived.

For demoing the behavior of these simple RNNs in general, observeUntrainedPredictions, observeTrainedPredictions and observeUntrainedVsTrained are defined in BPTT module.

Sources

The following resources were valuable in helping me form an understanding of backpropagation for neural nets:

• 3Blue1Brown neural networks playlist