SpikeCalls/FlyDrones

_{Recorded from the live browser demo: the swat game. Simulation running MiniFly, see the honest part.}

Open palm, it climbs. Fist, it holds. Drop your hand, it comes down. Rush at the camera, the giant fiber fires and it escapes.
The brain never gets a "climb" command. It gets an optic-flow illusion, and its own reflexes do the rest.

In 2026 the complete central nervous system of a male fruit fly was published: MaleCNS v1.0, about 166,000 neurons reconstructed by HHMI Janelia, Cambridge, MRC LMB and Google Research, free under CC-BY. People have plugged it into DOOM, Mario, trading bots and walking robots.

FlyDrones gives it wings. It turns a drone camera (or your hand in front of a webcam) into activity on the fly's own visual neurons, runs the connectome as a spiking network, and reads out the descending neurons a real fly uses to steer its wings as flight commands for a real drone.

spikecalls.github.io/FlyDrones. No install. A 3D bedroom, a blue quadcopter with a fly riding on top, and the whole loop running live in your browser: a ray-cast drone camera, per-cell optic flow, 850 spiking neurons, the descending-neuron read-out, the safety governor and the drone physics.

Play

When the drone really climbs, its camera sees real downward flow, which cancels the illusion, so the loop settles on its own.

• This is a computational model, not a living or resurrected fly. The wiring is biological (synapse counts from electron microscopy). The dynamics are a standard leaky integrate-and-fire model (Shiu et al., Nature 2024) using their published parameters.
• The bridge is engineered. Which camera features feed which neurons, the tonic "I am flying" drive, and the linear read-out from descending neurons to stick commands were designed by us. Each choice is written down in defaults.yaml and docs/SCIENCE.md, with the paper it leans on.
• The drone's own flight controller keeps it level. FlyDrones sends high-level stick commands (vertical speed, yaw rate, forward), like a pilot with a remote. A safety governor outside the brain always has the last word.
• The browser demo and the GIFs use MiniFly, an 850-neuron, hand-wired stand-in with real fly cell-type names, so it runs anywhere in seconds. It is not the real connectome. The fly sitting on the 3D drone is a mascot; the pilot is the simulated brain. In the swat game the giant fiber triggers a short jump (a stand-in for a fly's takeoff jump) instead of the gentle climb used for real drones. The real 166k MaleCNS connectome loads with two commands (below), and you calibrate its read-out with flydrones calibrate.
• Hardware adapters are written against the official SDKs but have not been flight-tested by us yet. Start in the simulator, then props-off, then a net or an empty room.

git clone https://github.com/SpikeCalls/FlyDrones.git
cd FlyDrones
python -m venv .venv && source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e ".[vision]"

flydrones demo --live                 # simulated room + scripted hand, live dashboard
flydrones demo --record my_demo.gif   # or save it
flydrones swarm --live                # one connectome, three pilots
flydrones inspect                     # poke every input group, watch the motor neurons

No camera, drone or GPU needed for any of the above.

The simulator is event-driven: each step only touches the synapses of neurons that actually spiked. Details in docs/ARCHITECTURE.md.

Hand tracking uses MediaPipe if installed (pip install -e ".[gestures]"), otherwise an OpenCV skin-colour detector.

Everything below is a dry run (commands printed, nothing sent) until you add --send.

Setup and wiring for each drone: docs/HARDWARE.md. Step-by-step guide: docs/GUIDE.md.

pip install -e ".[data]"
flydrones download malecns                   # ~1.2 GB from Janelia's public bucket (CC-BY)
flydrones build-brain --out data/malecns_brain.npz
flydrones inspect   --brain data/malecns_brain.npz
flydrones calibrate --brain data/malecns_brain.npz --out readout_malecns.json
flydrones demo      --brain data/malecns_brain.npz --config configs/malecns.yaml

build-brain keeps connections with ≥ 3 synapses, signs them by predicted neurotransmitter and resolves every input/output group by cell type and side. Exact counts depend on the filters. For reference, DOOMFLY reports 166,700 neurons and 25,582,938 directed connections after its own filtering.

For a faster model, keep only neurons within a few synapses of both the eyes and the descending neurons: flydrones build-brain --core-hops 3.

flydrones swarm copies one connectome into three independent brains: same wiring, separate spikes. We give each drone a different job, and the brain decides the rest. Output of the run above:

fly-1  hover job        -> holds ~1.0 m, 0 collisions
fly-2  cruise at chair  -> 2 giant-fiber escapes, gets past the chair, 0 collisions
fly-3  hand dropped     -> descends to the 0.3 m safety floor and holds

A desktop CPU is faster. The control loop adapts dt to wall time and warns if the brain falls behind. Use --core-hops or run on a faster machine for real-time full-brain flight. Measure yours with flydrones bench --brain ....

src/flydrones/
  brain/       connectome.py (MaleCNS loader, groups, subgraphs) · lif.py (simulator) · synthetic.py (MiniFly)
  senses/      retina.py (optic flow, looming) · gestures.py (hand -> illusions) · encoder.py · webcam.py
  motor/       decoder.py (descending neurons -> commands) · command.py
  drones/      sim.py · tello.py · crazyflie.py · mavlink.py · udp_bridge.py (ESP32/MSP)
  safety.py    limits, ceiling, floor, geofence, watchdog, battery
  runtime.py   the closed loop
  calibrate.py fit the read-out on your connectome
  viz/         live dashboard and GIF recorder
  cli.py       `flydrones ...`
docs/index.html + docs/live/     the browser demo (three.js, JS port of the engine)
firmware/esp32_msp_bridge/   Arduino sketch: UDP -> MSP_SET_RAW_RC
configs/     tello, crazyflie, mavlink SITL, esp32, malecns
docs/        GUIDE · HARDWARE · SCIENCE · ARCHITECTURE · CONNECTOME_DATA · SAFETY · FAQ
examples/    small scripts to poke neurons, write your own decoder, replay a video
tests/       pytest suite (simulator, retina, decoder, safety, protocol, MaleCNS loader)
tools/       export the browser brain, check the JS engine against Python

Drones cut fingers. The brain is a research model and can do unexpected things.

• Every hardware command is a dry run until --send.
• The safety governor clamps speed, fades out climbing near the ceiling, blocks descent below the floor, hovers if the brain stalls, lands on low battery or after max_flight_s.
• Ctrl+C lands. Keep the vendor app or a second remote ready as a kill switch.
• Props off for the first bench test. Fly indoors only with prop guards, over soft ground, away from people and pets.

Read docs/SAFETY.md before the first real flight.

• Event-driven LIF simulator, MaleCNS v1.0 loader, MiniFly
• Optic flow, looming, gesture illusions, haltere feedback
• Tello, Crazyflie, MAVLink, ESP32/Betaflight adapters, safety governor
• Live dashboard, GIF recorder, swarm mode, read-out calibration
• Browser demo on GitHub Pages: 3D room, swat game, clickable neurons, webcam hand tracking
• First real Tello flight video with the full MaleCNS brain
• Retinotopy from MaleCNS optic-lobe hex coordinates
• PyTorch / CUDA backend for full-brain real time on a laptop GPU
• Dopamine (PPL1) reinforcement on crashes, like DOOMFLY
• ROS 2 node
• Onboard: brain core on a Raspberry Pi 5 carried by the drone

See ROADMAP.md. Ideas and PRs are welcome: CONTRIBUTING.md.

• MaleCNS v1.0 connectome: HHMI Janelia FlyEM, University of Cambridge, MRC LMB, Google Research. male-cns.janelia.org · data CC-BY 4.0 · paper in Cell (2026).
• LIF whole-brain model: Shiu P.K. et al., A Drosophila computational brain model reveals sensorimotor processing, Nature 2024. code
• Flight circuitry: Namiki et al. (DNg02), Ache et al. 2019 (LPLC2/LC4 → giant fiber), DNp03 flight-saccade papers in Current Biology 2024-2025, Maisak et al. 2013 (T4/T5 directions).
• Full reference list and what is literature vs. engineering: docs/SCIENCE.md. Data licences: THIRD_PARTY.md.

Related projects that inspired this one: DOOMFLY (MaleCNS plays DOOM) · flybrain (FlyWire on a Tello) · flybrain-robot-bridge · Eon fly-brain · NeuroMechFly / FlyGym · flybody · awesome-fly.

Code: MIT. Connectome data keeps its own licence (MaleCNS: CC-BY 4.0), see THIRD_PARTY.md. If you use FlyDrones in a paper or video, please credit the connectome authors and cite this repo.

_{The fly is a model. The drone is real. Fly carefully.}