Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions Approx tokens/sec: 38.08
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@

### 2.4 Failure Triage

When a real‑QVAC run exits with a non‑zero code or unexpected output, use the
following steps to investigate.

1. **Validate connectivity** – Verify the endpoint URL with `curl -v $QVAC_URL`.
If the server is reachable, a `200 OK` response should be returned.

2. **Confirm the model name** – Ensure the value passed to `--model` matches
a model that the endpoint actually supports. Check the provider’s
documentation or query the `/models` endpoint if available.

3. **Verify the API key** – Double‑check that the key stored in
`QVAC_API_KEY` (or `OPENAI_API_KEY`) is correct and has not expired.
Rotate keys if necessary.

4. **Check for rate limits** – Repeated runs may trigger `429 Too Many Requests`
responses. Wait a short time and retry, or ask your provider to increase
the quota.

5. **Inspect sanitised output** – The CLI prints error details to stderr.
Sensitive tokens are redacted, but the remaining information usually
contains the HTTP status code and a human‑readable reason.
```bash
node dist/cli.js --url ... --model ... 2> error.log
```

6. **Adjust timeouts** – If the error code is `timeout`, increase the request
timeout with `--timeout-ms`. This is especially helpful on high‑latency
connections or for slow models.

7. **Interpret common error codes**
- `server_unavailable` – The upstream host could not be reached. Check
network access and firewall rules.
- `http_error` – A non‑2xx HTTP status was returned. Look at the sanitised
message for the code (e.g., 401/403 for auth problems, 5xx for
server‑side issues).
- `malformed_stream` – The server sent an unexpected SSE payload. Use
`curl` to inspect the raw JSON lines and confirm the server is
compatible with the OpenAI‑style streaming API.
- `timeout` – The request took longer than the configured timeout. Try
relaxing the limit or checking network latency.

8. **Test with minimal parameters** – Use a short prompt (`"Hello"`) and a
small `--max-tokens` value to isolate whether the issue is specific to a
longer request.

9. **Collect logs for further investigation** – Save both stdout and stderr
(e.g., `node dist/cli.js ... > out.txt 2> err.txt`). Share the sanitised
output when seeking help – **never** share raw API keys.

10. **Compare baseline** – As a last resort, use `curl` to send a simple
request to the same endpoint. If the raw `curl` also fails, the problem
lies with the server or network, not with `qvac-bench`.
1 change: 1 addition & 0 deletions docs/validation-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,4 @@ pipeline that does not have access to the real endpoint.
7. If the run exits with a non‑zero code or produces unexpected output, refer to the [Failure Triage](#24-failure-triage) section for debugging guidance.

**Example expected stdout (single iteration)**

Loading