Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ INFERENCE_BACKEND=fireworks
# (see docs/amd_proof/ — serve it on the AMD MI300X notebook first)
MODEL_ID=accounts/your-account/deployments/your-deployment-id

# Fireworks (AMD-hosted) — used when INFERENCE_BACKEND=fireworks
# Fireworks — used when INFERENCE_BACKEND=fireworks
FIREWORKS_API_KEY=
FIREWORKS_BASE_URL=https://api.fireworks.ai/inference/v1

Expand Down
8 changes: 4 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ open-weight family, **self-hosted on AMD** so eval data never leaves your hardwa
judge isn't a metered closed API. Do NOT claim "nobody does adversarial generation" —
Promptfoo already does; our angle is the self-hosted single-family closed loop on AMD.

- **P0 failure mode: hallucination only.** Jailbreak = P1, bias = P2. Do not add P0 scope.
- **Failure modes supported:** hallucination, jailbreak, and bias are all already implemented.
- **Demo target (system-under-test):** a deliberately weak model (e.g. Gemma 3 4B) so
failures are dramatic. Must be verified on Day 1 that it actually hallucinates on our
seeded attacks (the "Day-1 spike").
Expand Down Expand Up @@ -101,7 +101,7 @@ and tested — reuse, don't redefine.
1. **Response time < 30s per request** (all-tracks rule). → async fan-out; never run a huge
batch on the live path. Instrument wall-clock per run. Batch of ~20 is the demo size.
2. **AMD compute usage is REQUIRED or the project is disqualified.** Gemma must run on AMD:
self-hosted on MI300X (vLLM+ROCm) and/or Gemma on Fireworks' AMD-hosted infra. Proof
self-hosted on MI300X (vLLM+ROCm). Proof
(config + logs + screenshot) must be committed under `docs/amd_proof/` by end of Day 3.
Comment on lines 103 to 105
3. **No secrets in git.** `.env` is gitignored; read all keys from env. We supply our OWN
Fireworks keys — the `FIREWORKS_*`/`ALLOWED_MODELS` harness injection is a **Track 1**
Expand All @@ -113,7 +113,7 @@ and tested — reuse, don't redefine.
## 6. Inference backends (env-selected)

`INFERENCE_BACKEND` picks where Attacker+Judge Gemma runs:
- `fireworks` → Gemma on Fireworks (AMD-hosted). Powers the **live URL** (managed uptime).
- `fireworks` → Gemma on Fireworks. Powers the **live URL** (managed uptime).
Exact available Gemma model IDs must be confirmed from Fireworks (launch day was 6 Jul —
confirm and set `MODEL_ID`).
- `mi300x` → Gemma self-hosted on AMD Dev Cloud MI300X via vLLM+ROCm. Powers the
Expand Down Expand Up @@ -156,7 +156,7 @@ Do these in order. Each should be small, typed, and testable.
it; it imports **only** `run_eval` + schemas.
- Tests: 68 passing, `ruff` clean, CI green (`pythonpath=["."]` so bare `pytest` resolves).

**P1 (only if ahead):** jailbreak mode (`prompts/*_jailbreak.md` + AdvBench seed), full
**Future plans:** full
self-consistency (every case ×3), leaderboard data, SQLite run history, PDF export data.

## 8. Stack & conventions
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Not required for Track 3, but useful for a reproducible one-command run.
#
# Builds the Streamlit demo (app.py). The engine talks to Gemma over an
# OpenAI-compatible endpoint - Fireworks (AMD-hosted) or self-hosted vLLM on AMD +
# OpenAI-compatible endpoint - Fireworks or self-hosted vLLM on AMD +
# ROCm - configured entirely via environment variables (see .env.example). With no
# engine configured, the app still renders the committed real runs and the clearly
# labelled simulated demo, so the container is always demonstrable.
Expand Down
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,7 @@
GemmaJudge uses one open-weight model family (Gemma) in two adversarial roles — an
**Attacker** that generates targeted adversarial test cases for a chosen failure mode,
and a **Judge** that scores a target model's responses against a rubric — running the
whole closed loop on AMD (self-hosted on AMD GPUs via vLLM + ROCm, or Gemma on Fireworks'
AMD-hosted infrastructure).
whole closed loop on AMD (self-hosted on AMD GPUs via vLLM + ROCm, or Gemma on Fireworks).
Comment on lines 7 to +8

Built for the **AMD Developer Hackathon: ACT II — Track 3 (Unicorn)**.

Expand Down Expand Up @@ -112,7 +111,7 @@ All configuration is via environment variables — nothing is hardcoded. See
[`.env.example`](.env.example) for the full list. Two inference backends, selected by
`INFERENCE_BACKEND`:

- `fireworks` — Gemma on Fireworks' AMD-hosted infra (powers the live demo URL).
- `fireworks` — Gemma on Fireworks (powers the live demo URL).
- `mi300x` — Gemma self-hosted on an AMD GPU (AMD Developer Cloud) via vLLM + ROCm.
The committed proof used an AMD Radeon PRO W7900 (gfx1100); the same backend serves
an AMD Instinct MI300X unchanged.
Expand Down
2 changes: 1 addition & 1 deletion app.py
Original file line number Diff line number Diff line change
Expand Up @@ -726,7 +726,7 @@ def _render_amd_proof(settings: Any | None, config_error: str | None) -> None:
st.dataframe(pd.DataFrame(rows), use_container_width=True, hide_index=True)
st.caption(
"Pending files are shown honestly. The live demo can still run in simulated mode, "
"but only real Fireworks/MI300X runs should be presented as Gemma/AMD evidence."
"but only real MI300X/W7900 runs should be presented as Gemma/AMD evidence."
)


Expand Down
4 changes: 2 additions & 2 deletions deck/build_deck.py
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ def slide_solution(c):
# AMD band under attacker+judge
c.setFillColor(AMD)
c.setFont("Helvetica-Bold", 9)
c.drawString(x0, y - 0.28 * inch, "▲ both Gemma roles run on AMD (self-hosted via vLLM + ROCm, or Fireworks' AMD-hosted infra)")
c.drawString(x0, y - 0.28 * inch, "▲ both Gemma roles run on AMD (self-hosted via vLLM + ROCm)")
_bullets(c, [
("Config → generate N adversarial prompts → run the target → judge each with "
"written reasoning → report. Fully automated, no human in the loop.", INK, False),
Expand Down Expand Up @@ -257,7 +257,7 @@ def slide_amd(c):
("Proof-of-compute on AMD Radeon PRO W7900 (gfx1100 / ROCm 7.2): Gemma served via "
"vLLM — a real run with committed rocm-smi + serve command + vLLM logs in the repo "
"(hallucination ASR 80%, judge self-consistency stdev 0.00).", INK, False),
("Live URL backed by Fireworks (AMD-hosted) for uptime; self-hosted AMD (Radeon W7900, "
("Live URL backed by Fireworks for uptime; self-hosted AMD (Radeon W7900, "
"ROCm) for the screenshottable hardware proof.", INK, False),
("On-screen: model id + inference backend on every run, plus a cost/throughput "
"panel driven by actual measured token usage.", INK, False),
Expand Down
2 changes: 1 addition & 1 deletion gemmajudge/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ class ConfigError(RuntimeError):
class InferenceBackend(StrEnum):
"""Where the Attacker + Judge Gemma runs (env ``INFERENCE_BACKEND``)."""

FIREWORKS = "fireworks" # Gemma on Fireworks' AMD-hosted infra → live URL
FIREWORKS = "fireworks" # Gemma on Fireworks → live URL
MI300X = "mi300x" # Gemma self-hosted on AMD Dev Cloud MI300X → AMD proof


Expand Down
2 changes: 1 addition & 1 deletion gemmajudge/orchestrator.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ def _build_cost(
"""Turn measured per-role usage into a CostReport with a $ figure.

The $ figure prices the **engine** (Attacker+Judge) tokens, which is what runs
on the AMD-hosted Gemma; the target is a separate system whose price we don't
on the Gemma engine; the target is a separate system whose price we don't
assume. If no pricing is configured, ``usd`` is a truthful ``0.0``."""
pricing = settings.pricing if settings else None
if pricing is not None:
Expand Down
Loading