Two demos where the training happens on the chip itself, not on a laptop.
Demo 2 after training on 10 examples of each sound (0.9 s of training on the board): a finger clap (SOUND 1), a normal clap (SOUND 2) and a tap on the table (SOUND 3), each recognised with its confidence on the round display.
Inspired by Bill McCartney's field note Neural nets on chips that 'can't do' neural nets, which trains the network on a laptop and runs it on 16 microcontrollers. Here the ESP32-S3 does the training too.
| Demo 1, 80 Adam steps on 4,096 examples, 1 core | 35.1 s |
| Demo 1, same on 2 cores | 18.0 s (1.95x) |
| Demo 1, test accuracy (board and NumPy, identical) | 84.8 % |
| Demo 1, training work | ~1.33 billion multiply-adds |
| Demo 1, inference | 20,488 decisions/s, 6.9 cycles per multiply-add (the field note lists 36 for the S3) |
| NumPy on a laptop, same 80 steps | 0.08 s |
| Demo 2, 3 sounds x 10 examples, 40 epochs | 0.9 s |
| What happens | Where the data comes from | |
|---|---|---|
apps/01_signal_train |
Retrains the "sine / square / chirp / noise" classifier from the Thingdone field note (32-32-16-8-4 MLP, 1,756 weights, 80 full-batch Adam steps) on one core, then on both cores, then benchmarks inference | The board generates its own 4,096 training windows |
apps/02_sound_teacher |
Starts with random weights. You record a few examples of each sound through an I2S mic, type train, and a few seconds later it recognises them |
Your microphone |
Everything is plain C: dense layers, ReLU, softmax, backprop and Adam in
components/tinytrain (about 250 lines, no SIMD, no vendor ML library).
components/
tinytrain/ MLP + backprop + Adam (float32, portable C)
sigdata/ synthetic 8-bit signal generator for demo 1
audiofeat/ FFT, 20 mel bands, onset detector, 161-byte clip features
teacher/ record / train / predict / save logic for demo 2
roundlcd/ GC9A01 240x240 round display: framebuffer, 5x7 font, progress ring
trainview/ demo 1's live view on the round display
apps/
00_menu/ both demos in one firmware, picked with the button at boot
01_signal_train/
02_sound_teacher/
host/ the same code compiled for your laptop, with tests
tools/
reference_numpy.py laptop version of demo 1 (identical data and init)
render_training.py turns a training log into the "watch it learn" video
A 1.28" GC9A01 240x240 round SPI LCD shows what is happening. Without one, everything still runs and prints to the serial console.
| Display pin | ESP32-S3 (default) |
|---|---|
| VCC | 3V3 |
| GND | GND |
| SCL | GPIO 12 |
| SDA | GPIO 11 |
| CS | GPIO 10 |
| DC | GPIO 9 |
| RST | GPIO 8 |
| BLK (if present) | 3V3 |
Change the pins, or turn the display off, under idf.py menuconfig →
Round display (GC9A01 240x240). If the picture is mirrored or the colours
are off, adjust esp_lcd_panel_mirror / invert_color / the RGB order in
components/roundlcd/roundlcd_hw.c.
cd apps/00_menu
idf.py set-target esp32s3
idf.py -p /dev/ttyACM0 flash monitorAt boot the round display shows a menu:
| Button | Starts |
|---|---|
| double press | demo 1, the signal classifier (trains on 1 core, then 2) |
| single press | demo 2, the sound teacher |
Typing 1 or 2 in the serial monitor works too. The board's reset button
brings you back to the menu; after demo 1 finishes, a button press does as
well. The console runs at 921,600 baud here. Recordings saved with save in
the sound teacher are kept, since the partition layout is the same.
The two apps below still build on their own.
cd apps/01_signal_train
idf.py set-target esp32s3
idf.py -p /dev/ttyACM0 flash monitor | tee board.logThe console runs at 921,600 baud (set in sdkconfig.defaults, and
idf.py monitor picks it up automatically) so the prediction dump is quick.
The printing happens between steps and is not counted in the training time.
At the end it prints a results block like:
training, 1 core : X.XX s for 80 steps, test accuracy 84.8%
training, 2 cores: X.XX s for 80 steps, test accuracy 84.8% (1.9x faster)
training work : 1.33 billion multiply-adds, ~NN CPU cycles each (1 core)
inference : NNNN decisions/s, NNNNN cycles each, NN.N cycles per MAC
The inference line is directly comparable to the field note's ESP32-S3 row (60,668 cycles, 36 cycles per MAC, 3,956 decisions/s).
On the round display you watch it learn: a 2x2 grid with one square per real class (sine, square, chirp, noise), where each of the 1,024 windows in a square is coloured by what the network currently calls it, plus the step, the training accuracy and a progress ring. At the end it shows the 1-core vs 2-core times. Drawing happens between steps and is not counted.
Render the animation from the board's own run:
pip install pillow numpy # ffmpeg must be on your PATH
python3 tools/render_training.py apps/01_signal_train/board.log -o s3_training.mp4That writes the MP4 plus _start.png and _final.png stills. The log
contains two runs; the renderer uses the last one (the dual-core run, which
is the only one that dumps predictions).
Laptop comparison:
python3 tools/reference_numpy.py --dump laptop.log
python3 tools/render_training.py laptop.log -o laptop_training.mp4 --title "Laptop (NumPy)"The Python generator is a bit-exact port of the C one (verified: 100% of the 4,096 windows identical), and the weights start from the same RNG, so the laptop and the board learn along almost the same path. Expect around 89.7% training / 84.8% test accuracy on both.
| Mic pin | ESP32-S3 (default) |
|---|---|
| VDD | 3V3 |
| GND | GND |
| L/R | GND (left channel) |
| SCK / BCLK | GPIO 5 |
| WS / LRCL | GPIO 6 |
| SD / DOUT | GPIO 4 |
A push button between GPIO 7 and GND (internal pull-up, no resistor needed) lets you run it without typing:
| Button | Does |
|---|---|
| press | record the next sound: 10 examples, hands-free, then it stops by itself (press again to stop early) |
| hold 1 s | train for 40 epochs, then listen |
Sounds recorded with the button are called sound 1, sound 2, ... unless
you type name clap first. The click of the button itself is ignored: after a
press the display says GET READY for about a second, then GO.
Change the pins under idf.py menuconfig → Sound teacher (button GPIO -1
= no button). If your L/R pin is tied to 3V3, enable "Mic L/R pin is tied
HIGH" there.
The partition table needs a 4 MB (or larger) flash; it adds a 128 KB model
partition for saving. On modules with octal PSRAM (N8R8, N16R8) PSRAM is
enabled and holds the display's 115 KB framebuffer and the large temporary
buffer save uses; the network and its training state stay in internal RAM.
Boards without PSRAM still boot.
What the round display shows:
- idle: the sounds it knows and how many examples each has
- record: a live spectrum, and after each capture the 8 x 20 fingerprint of that clip (exactly what the network learns from), with the count
- train: one bar per epoch for accuracy on the examples, plus loss and time
- listen: the name of the sound it heard, its confidence, and a bar per sound
- level: live spectrum and how far above the background it is
cd apps/02_sound_teacher
idf.py set-target esp32s3
idf.py -p /dev/ttyACM0 flash monitorteach> level # 5 s meter: make some sounds, check you see "<- onset"
teach> listen # before training: "nothing has been taught yet"
teach> rec clap # clap 10 times, with a short pause between each
teach> rec snap
teach> rec whistle
teach> rec knock
teach> train # ~40 epochs, prints progress, then starts listening
teach> stop
teach> rec snap 5 # it mixes up snap and clap? add five more snaps...
teach> train 20 # ...and train a bit more from where it left off
teach> save # keeps the recordings and the weights in flash
Or with the button only: press, clap 10 times; press, snap 10 times; press, whistle 10 times; hold the button for a second to train; then make the sounds.
reset puts the weights back to random but keeps the recordings, which is
useful on camera: you can show it being clueless, then train it again.
How it works: every 32 ms frame becomes 20 log-mel energies. When the energy jumps about 13 dB above the tracked background level, the board grabs a 512 ms clip (including 64 ms from before the onset), averages pairs of frames to an 8 x 20 grid, normalises it against the loudest point, and adds a loudness value: 161 numbers. The network (161-32-16-6, 5,814 weights, 23 KB) learns from those, with mini-batches of 16, a small random time shift and a bit of noise added to each example so ten examples per sound are enough.
Tips: keep each recording session in the room you will demo in. If it
triggers on keyboard clicks, raise thresh (e.g. thresh 4). If it misses
quiet sounds, lower it. Up to 6 sounds and 30 examples each.
make -C host
./host/test_host # gradient check + demo 1 training on your PC
./host/test_sound # demo 2 pipeline with synthetic clap/snap/whistle/knock soundstest_sound pushes synthetic audio through the same onset detector,
features and trainer that run on the board. It should report about 99% on
new clips after training on 12 examples per sound. Real sounds through a
real mic are messier than synthetic ones, so expect lower numbers in a real room.
docs/WORKLOG.md has the build log: the bugs that were fixed on the way (the 921,600-baud console setting, the
2-core start order that first gave no speed-up, internal RAM vs. PSRAM), the display and button wiring, and the
gotchas of the native USB port.
MIT

