spiced-mini — Build with Paper

A tiny, runnable reproduction of the idea behind "Human-like autonomy emerges from self-play and a pinch of human data" (SPICED — arXiv 2606.19370, Emerge Lab), at toy scale, on a Mac (Apple M5).

Core idea: a minimal reward plus a pinch of human demonstrations as a regularizer yields a policy that is both competent and able to coordinate with people — instead of learning "alien" conventions no human can drive alongside.

The original paper uses PPO + a KL term toward a human policy, on GPUDrive (needs NVIDIA/CUDA → cloud-only on a Mac). This repo isolates exactly that regularizer mechanism at minimal scale, so it runs on a laptop and you can see it work.

---

Results (measured on an Apple M5)

Every number below is produced by the scripts in this repo.

1) Lever game — a pinch of human demos buys "coordination with people":

λ (knob)	self-play	coord-with-human	AI picks
0	0.998	0.034	an arbitrary 1.0 lever (alien)
0.05	0.893	0.807	the human's lever
2.0	0.770	0.751	the human's lever

Trade −0.29 task reward for +0.69 human-compatibility.

2) RLHF on a from-scratch tiny LLM (MLX, M5 GPU) — reward = count of "good":

β (KL)	reward	KL→ref	what the model generates
0	7.00	13.15	good good good good … (reward hacking)
0.15	1.00	0.23	the good cat sat (fluent and scores)
1.0	0.43	0.02	~stays at the reference

Switch the reward to "sentence length" → a different hack: β=0 gives a a big fox slept and a a fox …; β=0.2 gives the blue car ate and the blue cat saw … (same max score, but KL 0.10 → still fluent).

3) A REAL model (Qwen2.5-0.5B) + LoRA — 36 demos, 0.3% of params, ~20s on GPU:

New prompt (held out)	Before	After (LoRA)
"How do I stay productive…"	"To stay productive… 1. Set clear goals…"	"Ahoy! Ye best tackle… one plank at a time… Arrr, yarrr!"

Val loss 5.66 → 0.15. A pinch of demonstration data is enough for a real model to adopt a new convention and generalize to unseen prompts.

4) M5 flex — pretraining throughput (batch 1024, seq 24):

config	params	throughput	peak mem
small	0.6M	234k tok/s	1.7 GB
big	25.3M	11.1k tok/s	10.6 GB

GPU applegpu_g17g, 24 GB unified memory.

---

01 — coordination_demo (numpy · runs anywhere)

Other-Play lever game: several levers worth 1.0 plus one unique lever worth 0.9 (a Schelling point humans tend to pick). Pure self-play (λ=0) settles on an arbitrary 1.0 lever → fails when paired with a human. Adding λ · log p(human demo) shifts the policy onto the human's lever → they coordinate.

python3 01_coordination_demo/coordination_demo.py

Prints a λ → self-play reward, human-compat table and saves tradeoff.png. You'll see: trading a little task reward (1.0 → 0.9) buys the jump in human-compat (≈0.03 → ≈0.8).

Before/after still (same algorithm, different λ):

02 — rlhf_mini_mlx (MLX · uses the Apple M5 GPU via Metal)

Same mechanism but on a neural net: maximize E[reward] − β · KL(π ‖ reference). This is the KL term of RLHF. β=0 → reward hacking ("alien"); high β → stays natural, like people.

.venv/bin/python 02_rlhf_mini_mlx/rlhf_mini.py

Prints a β → reward, KL, natural-mass table, reports the MLX device (GPU) and timing, and saves rlhf_tradeoff.png.

Real-LLM version — llm_rlhf_mlx.py: pretrains a mini Transformer from scratch (no download) then RLHFs it with reward − β·KL(π‖reference). β=0 → reward hacking ("good good good…"); β=0.15 → still fluent ("the good cat sat") while scoring. This is exactly the KL mechanism behind RLHF / ChatGPT.

.venv/bin/python 02_rlhf_mini_mlx/llm_rlhf_mlx.py

---

Mapping to the paper

Paper (driving)	spiced-mini
Minimal reward (+1 reach goal, −1 collision)	self_play_reward / reward model
30 minutes of human demos	30 lever samples / reference distribution
Regularizer pulling toward human behavior	λ · log p(demo) / β · KL(π‖ref)
Self-play's "alien" convention	arbitrary 1.0 lever / reward-hacking action
Coordinating with held-out humans	crossplay_human / natural-mass

Advanced — M5 flex + real RLHF

In 02_rlhf_mini_mlx/:

• mini_gpt.py — shared tiny-GPT + corpus.
• flex_m5.py — benchmark small vs big GPT on the M5 (params, throughput, peak mem).
• reward_hacks.py — switch the reward (count "good" vs sentence length) → the AI hacks differently, while KL always reins it back toward the reference.

In 03_real_llm_mlx/ — a REAL model (Qwen2.5-0.5B-Instruct) + LoRA (needs mlx-lm):

.venv/bin/python 03_real_llm_mlx/make_data.py            # create a "pinch" of demos
.venv/bin/python -m mlx_lm lora --model mlx-community/Qwen2.5-0.5B-Instruct-4bit \
  --train --data 03_real_llm_mlx/data --fine-tune-type lora --num-layers 8 \
  --batch-size 4 --iters 120 --learning-rate 1e-4 --mask-prompt \
  --adapter-path 03_real_llm_mlx/adapters
.venv/bin/python 03_real_llm_mlx/compare.py              # compare before vs after

Before: a normal assistant. After ~36 demos (0.3% of params): it answers like a pirate and generalizes to unseen prompts → the SPICED "pinch of human data" idea, on a real LLM.

Setup

numpy and matplotlib are usually already present. MLX lives in .venv (installed: mlx). To recreate the environment:

python3 -m venv .venv && ./.venv/bin/pip install mlx mlx-lm numpy matplotlib

License

MIT — see LICENSE.