groq-cascade

A resilient multi-model fallback chain for the Groq API. When one model fails, it tries the next. Users always get a response.

Built from production infrastructure at zambo.dev, powering 17 AI products since 2026.

---

The problem

AI apps that call a single model break in production:

• Rate limits hit at peak hours
• Models go offline during maintenance
• Context windows exceeded on edge cases
• Empty responses returned silently

Most engineers handle this with a try/catch that shows users an error. This is worse.

---

The solution

A cascade that tries 6 models in order. If model 1 fails for any reason, it falls to model 2. Down the chain until something works. If everything fails, you can define a hardcoded fallback so users still get something.

llama-3.3-70b-versatile  ← try first (best quality)
  ↓ fails
llama-3.1-8b-instant     ← fast, high availability
  ↓ fails
llama-4-scout-17b        ← Meta's latest
  ↓ fails
gemma2-9b-it             ← Google, different infra
  ↓ fails
qwen-qwq-32b             ← Alibaba, different limits
  ↓ fails
mixtral-8x7b-32768       ← Mistral, long context
  ↓ fails
"your hardcoded fallback" ← always works

---

Install

npm install groq-cascade
# or
pnpm add groq-cascade

---

Usage

Basic

import { groqCascade } from "groq-cascade";

const result = await groqCascade({
  apiKey: process.env.GROQ_API_KEY,
  system: "You are a helpful assistant.",
  user: "Summarize this document in 3 bullet points.",
  fallback: "I'm having trouble right now. Please try again in a moment.",
});

console.log(result.text);   // the response
console.log(result.model);  // which model answered
console.log(result.attempts); // how many models were tried

JSON responses

import { groqCascadeJson } from "groq-cascade";

const { data, model } = await groqCascadeJson<{ score: number; reason: string }>({
  apiKey: process.env.GROQ_API_KEY,
  system: "Return only valid JSON.",
  user: "Score this idea from 0-100 and explain why: autonomous drone delivery",
});

console.log(data.score, data.reason);

Custom model list

import { groqCascade } from "groq-cascade";

const result = await groqCascade({
  apiKey: process.env.GROQ_API_KEY,
  user: "Quick question: what's 2+2?",
  models: ["llama-3.1-8b-instant", "gemma2-9b-it"], // just use fast models
  maxTokens: 64,
  temperature: 0,
  fallback: "4",
});

Log failures (monitoring)

const result = await groqCascade({
  apiKey: process.env.GROQ_API_KEY,
  user: "...",
  onModelFailure: (model, error) => {
    console.warn(`[cascade] ${model} failed:`, error);
    // send to Datadog, Sentry, etc.
  },
});

---

API

groqCascade(options) → Promise<CascadeResult>

Option	Type	Default	Description
apiKey	string	required	Groq API key
user	string	required	User message
system	string	—	System prompt
messages	CascadeMessage[]	—	Full messages array (overrides system + user)
models	string[]	6-model default	Models to try in order
maxTokens	number	1024	Max tokens per attempt
temperature	number	0.4	Sampling temperature
minLength	number	40	Min response chars before treating as failure
timeoutMs	number	12000	Per-model timeout in ms
fallback	string	—	Returned if all models fail (instead of throwing)
onModelFailure	fn(model, err)	—	Called on each failure

groqCascadeJson<T>(options) → Promise<{ data: T, model: string, attempts: number }>

Same options. Automatically strips markdown code fences and parses JSON. Throws if no valid JSON found.

---

License

MIT

Built by Brennan Zambo · @zambodotdev