Audio

An audio-reactive video generator using ModernGL and ffmpeg.

av.py renders a fragment shader for every frame of an audio file, analyzes the audio with librosa (RMS, bands, onset, beat, spectral centroid, and more), streams RGB frames to ffmpeg, and muxes H.264 + AAC into an MP4.

Prerequisites

• Python 3 with venv
• ffmpeg on your PATH (the script shells out to encode video)
• Optional: a still image (PNG/JPEG, etc.) when you want the shader to sample a texture instead of a black canvas

Quick start

python3 -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Basic usage

Two modes, controlled by how many positional arguments you pass:

1. Image + audio → video — resolution follows the image; the shader receives it as sampler2D u_tex (when the shader declares that uniform).

   python av.py photo.jpg track.wav outputs/out.mp4

2. Audio only → video — solid black RGB buffer at --width × --height (default 1280×720). Use this for shaders that do not need a source image.

   python av.py track.wav outputs/out.mp4

Command-line options

Flag	Default	Meaning
--fps	30	Frame rate; also drives audio feature hop length.
--shader	shaders/sliding_bars.glsl	Fragment shader path.
--vertex-shader	shaders/vertex.glsl	Vertex shader path.
--width, --height	1280, 720	Canvas size in audio-only mode (ignored when an image is provided).
--grid	1	Subdivisions of a clip-space triangle mesh (1 = one fullscreen quad). Higher values feed a denser mesh to the vertex shader for deformation.
--flip-vertical	off	Flip each frame vertically after readback so “up” in the shader matches “up” in the encoded file (OpenGL’s origin is bottom-left).
--ripples	off	Blank canvas only + --shader shaders/pin_impression_bw.glsl: center ripples (water-stone) pin heights (u_pin_pattern = 1). Mutually exclusive with --waves.
--waves	off	Same pairing: smooth interference-wave pin heights (u_pin_pattern = 2).

Examples

Pin board examples with image input:

python av.py flower.jpg song.wav outputs/flower_pins.mp4 \
  --shader shaders/pin_impression.glsl \

Black-and-white pin board (luminance only, same idea as pin_impression.glsl). With an image, av.py uploads Rec.709 grayscale (same weights as the shader) so the texture is black-and-white while pin length still follows luminance.

python av.py photo.png music.wav outputs/bw.mp4 \
  --shader shaders/pin_impression_bw.glsl \

Blank canvas + procedural pins (no image; still pass <audio> <output>):

python av.py music.wav outputs/pins_ripples.mp4 \
  --shader shaders/pin_impression_bw.glsl \
  --ripples

python av.py music.wav outputs/pins_waves.mp4 \
  --shader shaders/pin_impression_bw.glsl \
  --waves

Shaders and uniforms

The Python side sets u_time (seconds) every frame. It also sets optional uniforms only if your shader declares them:

• u_audio — normalized RMS envelope
• u_bass, u_mids, u_treble — band energies
• u_onset — onset strength
• u_beat, u_bpm_phase — beat pulses and phase vs. estimated tempo
• u_centroid, u_flatness — spectral descriptors
• u_harmonic, u_percussive — HPSS split
• u_pin_pattern — used by pin_impression_bw.glsl: 0 texture luma, 1 ripples, 2 waves (set from --ripples / --waves in blank-canvas mode).
• u_reveal_mask_flip — 0 default; 1 with --organic-mask-flip inverts vertical crawl for mandala_organic_reveal.glsl.

Other fragment shaders in shaders/ include character_pulse.glsl, rectangles.glsl, tethered_nodes.glsl, organic_reveal.glsl, and others; try them with --shader.

python av.py photo.jpg track.wav outputs/organic_reveal.mp4 \
  --shader shaders/organic_reveal.glsl

Vertex shaders

Default shaders/vertex.glsl passes clip-space positions through. For motion tied to the mesh, you can point --vertex-shader at something like shaders/vertex.glsl (if your shader pair expects the same varyings).

Output

Rendering writes a temporary temp.mp4 in the working directory, then renames it to your output path. Ensure you have write permission there and enough disk space for the encode.