Process many images in parallel

[handwriting-recognition.git] / README.md

diff --git a/README.md b/README.md
index 96974c990119387ee722e8f5d809536d0885233c..3af0686d6972af783a60d8ca377d150c54e01cad 100644 (file)
--- a/README.md
+++ b/README.md
@@ -6,6 +6,7 @@ Basics:
  - [Neuron](https://en.wikipedia.org/wiki/Artificial_neuron)
  - [Perceptron](https://en.wikipedia.org/wiki/Perceptron)
  - [Backpropagation](https://en.wikipedia.org/wiki/Backpropagation)
  - [Neuron](https://en.wikipedia.org/wiki/Artificial_neuron)
  - [Perceptron](https://en.wikipedia.org/wiki/Perceptron)
  - [Backpropagation](https://en.wikipedia.org/wiki/Backpropagation)

+ - [Understanding & Creating Neural Networks with Computational Graphs from Scratch](https://www.kdnuggets.com/2019/08/numpy-neural-networks-computational-graphs.html)

  - [3Blue1Brown video series](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)
 
 Too high-level for first-time learning, but apparently very abstract and powerful for real-life:
  - [3Blue1Brown video series](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi)
 
 Too high-level for first-time learning, but apparently very abstract and powerful for real-life:


@@ -49,3 +50,38 @@ output vector of first image: [    0.         52766.88424917     0.
 classification of first image: 1 with confidence 52766.88424917019; real label 5
 correctly recognized images after initialization: 10.076666666666668%
 ```
 classification of first image: 1 with confidence 52766.88424917019; real label 5
 correctly recognized images after initialization: 10.076666666666668%
 ```

+
+ - Add backpropagation algorithm and run a first training round. This is slow, as expected:
+ ```
+ $ time ./train.py
+output vector of first image: [    0.         52766.88424917     0.             0.
+ 14840.28619491 14164.62850135     0.          7011.882333
+     0.         46979.62976127]
+classification of first image: 1 with confidence 52766.88424917019; real label 5
+correctly recognized images after initialization: 10.076666666666668%
+round #0 of learning...
+./train.py:18: RuntimeWarning: overflow encountered in exp
+  return 1 / (1 + np.exp(-x))
+correctly recognized images: 14.211666666666666%
+
+real   0m37.927s
+user   1m19.103s
+sys    1m10.169s
+```
+
+ - This is way too slow. I found an [interesting approach](https://www.kdnuggets.com/2019/08/numpy-neural-networks-computational-graphs.html) that harnesses the power of numpy by doing the computations for lots of images in parallel, instead of spending a lot of time in Python on iterating over tens of thousands of examples. Now the accuracy computation takes only negligible time instead of 6 seconds, and each round of training takes less than a second:
+```
+$ time ./train.py
+output vector of first image: [0.50863223 0.50183558 0.50357349 0.50056673 0.50285531 0.5043152
+ 0.51588292 0.49403    0.5030618  0.51006963]
+classification of first image: 6 with confidence 0.5158829224337754; real label 7
+correctly recognized images after initialization: 9.58%
+cost after training round 0: 1.0462266880961681
+[...]
+cost after training round 99: 0.4499245817840479
+correctly recognized images after training: 11.35%
+
+real   1m51.520s
+user   4m23.863s
+sys    2m31.686s
+```