Add backpropagation and first round of learning

[handwriting-recognition.git] / README.md

diff --git a/README.md b/README.md
index ad1494553d16fddaf4c96faa725c0d27c196c459..760002f714dd303e786e5d55989023d9a15d2ede 100644 (file)
--- a/README.md
+++ b/README.md
@@ -5,6 +5,7 @@ Basics:
  - [MNIST database of handwritten digits](http://yann.lecun.com/exdb/mnist/)
  - [Neuron](https://en.wikipedia.org/wiki/Artificial_neuron)
  - [Perceptron](https://en.wikipedia.org/wiki/Perceptron)
+ - [Backpropagation](https://en.wikipedia.org/wiki/Backpropagation)
  - [3Blue1Brown video series](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)
 
 Too high-level for first-time learning, but apparently very abstract and powerful for real-life:
@@ -27,9 +28,42 @@ plt.imshow(grad, cmap='gray')
 plt.show()
 
 plt.imshow(np.sin(np.linspace(0,10000,10000)).reshape(100,100) ** 2, cmap='gray')
-# does not work with QT_QPA_PLATFORM=wayland
+# non-blocking does not work with QT_QPA_PLATFORM=wayland
 plt.show(block=False)
 plt.close()
 ```
 
  - Get the handwritten digits training data with `./download-mnist.sh`
+
+ - Read the MNIST database into numpy arrays with `./read_display_mnist.py`. Plot the first ten images and show their labels, to make sure the data makes sense:
+
+   ![visualize training data](screenshots/mnist-visualize-training-data.png)
+
+ - Define the structure of the neural network: two hidden layers with parametrizable sizes. Initialize weights and biases randomly. This gives totally random classifications of course, but at least makes sure that the data structures and computations work:
+
+```
+$ ./train.py
+output vector of first image: [    0.         52766.88424917     0.             0.
+ 14840.28619491 14164.62850135     0.          7011.882333
+     0.         46979.62976127]
+classification of first image: 1 with confidence 52766.88424917019; real label 5
+correctly recognized images after initialization: 10.076666666666668%
+```
+
+ - Add backpropagation algorithm and run a first training round. This is slow, as expected:
+ ```
+ $ time ./train.py
+output vector of first image: [    0.         52766.88424917     0.             0.
+ 14840.28619491 14164.62850135     0.          7011.882333
+     0.         46979.62976127]
+classification of first image: 1 with confidence 52766.88424917019; real label 5
+correctly recognized images after initialization: 10.076666666666668%
+round #0 of learning...
+./train.py:18: RuntimeWarning: overflow encountered in exp
+  return 1 / (1 + np.exp(-x))
+correctly recognized images: 14.211666666666666%
+
+real   0m37.927s
+user   1m19.103s
+sys    1m10.169s
+```