LightFlow

Audio-reactive lighting. Two surfaces in one Next.js app:

• /player: the automated show. Songs are analyzed offline (Demucs stem separation + librosa), then an attention layer picks the dominant stem per frame and a deterministic Director chooses a pulse rate per bar. An optional Claude Scout pass reads the analysis once and returns song-level moments (drops, lifts, breakdowns) and per-section hints. The screen previews the result as a flat full-screen light or a 3D rig; with the DMX bridge running, the same signal goes to real fixtures.
• /live: the live console for events. Mic or tap-tempo input drives a scene engine (keyboard-triggered scenes and colours, strobe/pulse/blackout, AutoPilot) rendered on an on-screen 3D rig and, when output is switched to live, out over DMX via an Enttec USB Pro.

Hosted demo: https://lightflow-emunns-projects.vercel.app (three pre-analyzed songs; the full library stays local).

Architecture

[audio file]
     │
     ├─► Python analyzer (Demucs + librosa)  ~60s/song, local only
     │     └─► public/songs/<slug>_analysis.json  (stems, sections, BPM, onsets)
     │
     ├─► Scout (src/lib/scout/, /api/scout)  optional, Claude, once per song
     │     └─► public/songs/<slug>_scout.json  (moments, section annotations, primary stem)
     │
     ├─► /player  (src/app/player/page.tsx)
     │     ├─ Attention layer (src/lib/attention.ts): one dominant stem per frame, shaped envelope
     │     ├─ Director (src/lib/modulators.ts): pulse rate per bar — raw / pulse:1–16, step-down ladder
     │     ├─ Scout apply (src/lib/scout/apply.ts): moment cues + stem weights + section hints
     │     └─ Render: flat screen | Screen3D | sendDMX()
     │
     └─► /live  (src/app/live/page.tsx)
           ├─ liveAudio.ts: mic energy / bands / onsets / auto-BPM
           ├─ tapTempo.ts: phase-continuous tempo clock (tap or typed BPM)
           ├─ scenes.ts: living scenes + beat-synced motion layer
           ├─ autopilot.ts: hysteresis state machine → scene/colour selection
           └─ stageRender.ts + fixtures.ts → universe → Rig3D preview | sendDMX()

dmxClient.ts ──ws://localhost:7777──► scripts/dmx-bridge.mjs ──serial──► Enttec USB Pro

Quick start

npm install
echo 'ANTHROPIC_API_KEY=sk-ant-...' >> .env   # optional, only for Scout generation
npm run dev                                   # http://localhost:3000

/player with no local song library shows the three demo songs tracked in git. To use your own library, drop <slug>.mp3 + <slug>_analysis.json (+ optional <slug>_scout.json) into public/songs/ and regenerate public/songs/manifest.json with scripts/ingest_audio_dir.py. Everything in public/songs/ except the three demo songs is gitignored and excluded from Vercel uploads (.vercelignore).

DMX output (events)

node scripts/dmx-bridge.mjs      # auto-detects the Enttec; DMX_PORT=/dev/tty.usbserial-XXXX to force

Open /live on the same laptop, press L (or the preview/live button) to send the universe to the bridge. On localhost the page connects to the bridge automatically; on the hosted URL it only dials the bridge after that user gesture, which is what triggers Chrome's local-network permission prompt.

Analyzing new songs (local only)

Dropping an audio file onto /player posts it to the Python analyzer at http://localhost:8000 (NEXT_PUBLIC_API_URL to override). Without it, the file plays in raw mode (audio only, no Director).

conda create -n lightflow python=3.10 && conda activate lightflow
pip install demucs librosa soundfile fastapi uvicorn pydantic
conda install -c conda-forge 'ffmpeg>=6,<7'
cd archive/v3-python && KMP_DUPLICATE_LIB_OK=TRUE PYTHONUNBUFFERED=1 python server.py

Known quirks: torchaudio wants ffmpeg 4–7 (Homebrew ships 8, use conda's); KMP_DUPLICATE_LIB_OK=TRUE silences the OpenMP duplicate-lib abort on macOS; num_workers=0 is required on macOS.

How the player decides

1. Attention (attention.ts): per frame, reads stem energies, picks the loudest/most-interesting stem, shapes its envelope with stem-specific attack/release (kick = exp decay, hat = square, bass = plateau, vocals = sigmoid). v1.5 is bass-first by default; Scout can flip that per song.
2. Director (modulators.ts): maps the attention signal to a pulse rate on a continuum (raw, pulse:1 per beat, :2, :4/:5, :8/:16 strobe). Decisions are held for bars, not frames. Strobes start only on downbeats, any fast rate is capped at 2 bars before a mandatory step-down, hat-dominant sections run raw. DIRECTOR_VERSION tags the algorithm.
3. Scout (scout/): one Claude call per song turns the analysis into a small JSON file: an arc sentence, primaryStem, modulatorBias, timestamped moments with anticipation/duration, and per-section annotations (semantic label, stem focus, intensity scale). apply.ts turns those into attention weights and cue boosts at render time. No key or no file = Director-only baseline; the engine's taste is always the floor.
4. Render: signal = attention × modulator × intensity (+ moment boost) → screen brightness / 3D rig / DMX channels.

Key files

Deploying

The Vercel project deploys from GitHub. Pushes to live build a Preview; promote it (vercel promote <url>) or set the project's production branch to live so pushes go straight to production. Set ANTHROPIC_API_KEY in the Vercel environment for Scout. Do not deploy with vercel --prod from a working tree that has the full song library unless .vercelignore is present: the CLI uploads public/ wholesale.

Director version history

• v1.0 — unified pulse-rate continuum, smooth decay envelope, beatless→raw, hat→raw, transient detector, beat-offset grid fitting, 2-bar min commit.
• v1.1 / v1.2 — Claude-as-editor layer (DraftPlan + PlanPatch) and composed bar patterns; later replaced by the simpler single-shot Scout.
• v1.5 — bass-first attention + perceptual decay + simplified UI.

LightFlow ML Roadmap

Plan for evolving LightFlow from a deterministic, per-song precomputed light show into a learned model that can drive live shows from microphone input — and continue to be tweaked and improved over time.

Where we are today

• Webapp (src/) — Next.js UI with stem-aware attention layer, Director (auto-picks raw / strobe / pulse / blackout per bar), Program Focus Timeline, energy-waveform scrubber, drag-and-drop with server-side analysis.
• Analysis pipeline (archive/v3-python/analyze.py) — htdemucs stem separation, librosa BPM + section detection, per-frame stem energies / onsets / pans / centroid / spectral flux at 30fps.
• Director (src/lib/modulators.ts) — deterministic state machine that picks effects from stem/centroid/variance/beat-phase, bar-locked, with rate-limiting.
• Output — for any song with analysis we precompute a Program Focus Timeline (programBlocks) of stem focus + effect choice across the entire track.

This is the deterministic teacher that everything below builds on.

Goal

A model that takes live audio (mic, line in) and produces brightness + stem-focus + effect decisions in real time, with comparable taste to the current deterministic Director — and a workflow for tweaking it over time.

Phase 1 — Data pipeline at scale

Storage layout

songs/
  {audio_sha256}/
    audio.mp3              # source
    analysis.json          # demucs stems + bpm + sections + per-frame features
    show.json              # precomputed program timeline (current Director output)
    metadata.json          # title, artist, genre, source, license, bpm, duration

• Hash by audio content → free dedup.
• Blob storage on R2 / B2 / S3.
• SQLite or Postgres for metadata + search.

Batch analysis

• Demucs on CPU: ~60–90s/song. On a single GPU: ~5–10s.
• 5000 songs on a 4090 ≈ 7–14 hrs. Easy overnight job.
• Lambda Labs A100 ≈ $1/hr → 5000 songs in ~5 hrs for ~$5.
• Refactor needed: turn analyze_cli.py into a directory walker that hashes inputs, skips already-analyzed entries, and writes the hashed layout above.

Sourcing songs

• Free Music Archive (~150k CC tracks) — best legal corpus.
• Jamendo, ccMixter, MTG-Jamendo (used by Spotify research).
• Personal library for taste calibration.
• ⚠️ Do not train on commercial music if there's any chance of shipping it.

Phase 2 — The model

Two architectures, one to start with, one to add later.

Option A — Imitation learning (start here)

Map audio features → Director decisions, learning from precomputed shows.

Inputs (per ~30ms frame):

• Mel spectrogram, last 1–2 seconds (gives the model "ears")
• Beat phase, BPM estimate, spectral features
• Optionally stem energies during training (teacher-forcing); model learns to predict without them at inference time

Outputs (per frame):

• Brightness (0–1, regression)
• Dominant stem (5-way softmax: bass / kick / snare / hat / other)
• Effect choice (4-way softmax: raw / strobe / pulse / blackout)

Architecture:

• Small transformer or 1D conv-net on a sliding mel window
• ~5–20M parameters
• Trains in a few hours on a single GPU
• 30fps CPU inference is trivial

Training data:

• 5000 songs × 4min × 30fps = ~36M (frame_features → director_choice) pairs
• Each programBlocks entry contributes labels at 30fps over its [start, end] interval
• Massively over-determined relative to model size

Key trick — stem distillation:

• The deterministic Director uses live stems, but a mic feed doesn't have stems
• Train a "stemless" student: input = raw mel only, target = stem-aware Director output
• This is the same idea as demucs distillation
• Works well in practice; the model effectively learns an implicit cheap stem separator

Option B — Preference / RL learning (later)

• Watch shows live, thumbs-up/down moments
• Tiny dataset (~hundreds of labeled segments) goes a long way once the imitation baseline exists
• Use as a fine-tuning layer on top of Option A — don't start here

Phase 3 — Live mic inference

Pipeline

mic ──► circular buffer (2s)
    ──► mel spectrogram (~10ms)
    ──► trained model (~5ms)
    ──► existing modulators (raw / strobe / pulse / blackout)
    ──► LED output / WebGL render

Latency budget: 30–80ms total. Achievable on a Mac mini.

Live BPM tracking

• Offline beat trackers (librosa.beat.beat_track) are too slow / wobbly live.
• Use BeatNet (PyTorch streaming, ~30ms latency) — or fold beat estimation into the model's output head.

Audio I/O

• WebAudio API in browser (works for laptop mic / line in).
• For LED output, the existing Three.js Screen3D already drives a per-frame brightness — same hook works.

Phase 4 — Tweak & improve workflow

This is where having a learned model pays off vs. a static deterministic one.

1. Show review UI

• Load a song, watch the model's predicted timeline alongside the deterministic Director's.
• Flag disagreements. (We already have the Program Focus Timeline component — extend it to render two tracks side-by-side.)

2. Manual override capture

• Every time the user pins a sub-pill manually during playback, log (song_hash, time, predicted_choice, user_choice) as a correction example.
• Stored in a corrections table; queryable.

3. Periodic fine-tune

• Weekly job: replay all corrections + a sample of original training data through a small LoRA-style update on the model.
• The model gradually shifts toward the user's taste, not just the deterministic teacher's.

4. A/B mode

• Split-screen rendering: deterministic Director vs. learned model on the same audio.
• Tracking metric: how often the user prefers the learned version → improvement signal.

Hard parts

• Stem → stemless distillation stability — the most important and least solved-by-default piece.
• Live BPM at song start / on quiet sections — beat trackers wobble. Mitigate with smoothing + uncertainty output.
• Genre coverage — a model trained only on EDM is terrible on jazz. Curate intentionally; diversity > volume.
• The "average show" trap — imitation learners trained on many sources regress to a boring mean. Counter with:
  • Train on a single curated style first
  • Add a sampling temperature / variance term
  • Layer on Phase-4 preference learning

Concrete first milestone (1 focused week)

Target: 500 curated electronic tracks → working learned model running side-by-side with the deterministic Director in the existing webapp.

This deliverable is enough to decide whether the approach is worth scaling.

Deployment notes (from current discussion)

• The current Next.js /api/analyze route shells out to analyze_cli.py over a venv. Works locally and on any container host (Render, Railway, Fly.io).
• Won't work on Vercel/Netlify-style serverless — torch + demucs is ~2GB and takes 30s–2min per call.
• Recommended deployment shape:
  • Webapp on Vercel
  • Analysis worker on Render or Fly Machines (or Modal/Replicate for GPU on-demand)
  • Results cached in object storage by audio hash → instant on re-load
• Once the model exists, inference (the mic-driven path) is small enough to run client-side via ONNX Runtime Web. No backend needed for the live show.

Cheat code

The single biggest reason this is feasible: we already have a deterministic teacher that produces good output. Most ML projects have to figure out what "good" looks like first. We don't.

Everything in this roadmap is plumbing on top of that core asset.