Support OpenAI verbose_json for /v1/audio/transcriptions (+ multipart field fix)

[ademasi]

Support OpenAI verbose_json for /v1/audio/transcriptions (+ multipart field fix)

Summary

• Adds support for response_format=verbose_json to POST /v1/audio/transcriptions, returning an OpenAI-compatible payload with per-segment timestamps (start / end) extracted from Whisper's internal <|X.XX|> markers.
• Fixes a multipart-dispatcher bug where everything except model and file was silently dropped on the way to the transcription handler (so response_format, language, prompt, stream, and temperature had no effect when sent via multipart).
• Fully backward-compatible: callers that don't ask for verbose_json (or don't set response_format at all) get the same {model, text} payload shape as before.

Changes

• src/server/server.cpp — the multipart handler for /v1/audio/transcriptions now forwards response_format, language, prompt, stream, and temperature from the parsed multipart body into the request JSON before dispatching to handle_openai_audio_transcriptions. Previously only model and file were forwarded.
• src/server/rest_handler.cpp — handle_openai_audio_transcriptions now reads response_format from the request. When it is verbose_json, the Whisper engine is invoked with timestamps enabled and the raw output is split on <\|(\d+\.\d+)\|> to build a segments array; otherwise the existing {model, text} behavior is preserved (with timestamp markers stripped as a defensive step). Adds #include <regex>.

Before / after

Before

Request (either JSON body or multipart):

POST /v1/audio/transcriptions
Content-Type: multipart/form-data

model=whisper-...
file=<audio>
response_format=verbose_json

Response (every format returned the same shape; response_format was ignored by the multipart path entirely):

{
  "model": "whisper-...",
  "text": "This is the transcribed text of the audio clip..."
}

After

Same request with response_format=verbose_json:

{
  "task": "transcribe",
  "language": "en",
  "duration": 12.34,
  "text": "This is the transcribed text of the audio clip split into two segments.",
  "segments": [
    {
      "id": 0,
      "seek": 0,
      "start": 0.00,
      "end": 6.20,
      "text": " This is the transcribed text of the audio",
      "tokens": [],
      "temperature": 0.0,
      "avg_logprob": 0.0,
      "compression_ratio": 0.0,
      "no_speech_prob": 0.0
    },
    {
      "id": 1,
      "seek": 0,
      "start": 6.20,
      "end": 12.34,
      "text": " clip split into two segments.",
      "tokens": [],
      "temperature": 0.0,
      "avg_logprob": 0.0,
      "compression_ratio": 0.0,
      "no_speech_prob": 0.0
    }
  ],
  "model": "whisper-..."
}

Same request with response_format=json (or omitted): byte-for-byte identical to the old behavior, i.e. {model, text}.

Backward compatibility

• Plain response_format=json callers (and callers that omit response_format) see the same {model, text} response shape as before. The only change to that path is that any stray <|X.XX|> timestamp markers are stripped from text before returning — but because the underlying generator is only asked to emit timestamps when verbose_json is requested, this strip is a belt-and-suspenders pass and is a no-op for the plain path.
• The multipart fix is additive: fields that callers weren't previously setting via multipart continue to use the handler's defaults; fields that callers were setting (but which were being dropped) now actually take effect. Anyone relying on the old "dropped" behavior would have had to be setting these fields via a JSON body instead, which already worked correctly.
• No changes to URL, method, required fields, or the shape of any pre-existing response format.

Test plan

• Tested end-to-end against a long-form (~1 hour) multi-speaker Opus recording; response_format=verbose_json returned 200+ segments with monotonically increasing start / end timestamps and a matching flat text field.
• Verified the duration field in the verbose response matches the expected clip length (within one segment boundary).
• Plain response_format=json path manually verified to return {model, text} with no behavioral change and no timestamp markers in text.
• Verified that language, prompt, temperature, and stream sent via multipart now reach the handler (previously dropped).
• CI build on maintainer's platform matrix.

Notes for reviewers

• Timestamp parsing is regex-based: segment boundaries are extracted from Whisper's internal <|X.XX|> markers using std::regex with the pattern <\|(\d+\.\d+)\|>. This is intentionally conservative — it assumes the markers are emitted verbatim in the decoded string and can't straddle a Unicode boundary. In practice this holds today because the markers are ASCII, but it is fragile if the tokenizer or decoder output format changes. A more robust long-term fix would be to do the split at the tokenizer level (i.e. have whisper_engine->generate(...) return structured segment data directly, rather than reconstructing it from the flat string), which is out of scope for this PR.
• The per-segment tokens, avg_logprob, compression_ratio, and no_speech_prob fields are emitted with zero/empty defaults because the Whisper engine doesn't currently surface them; they are present in the response so the shape matches the OpenAI spec and downstream clients don't have to special-case missing keys. Filling them in is a follow-up.
• The two commits are split so the multipart-dispatcher bug fix (fix(server): forward multipart audio fields...) can be taken independently of the verbose_json feature commit if desired.

[Copilot]

Pull request overview

Adds OpenAI-compatible response_format=verbose_json output to /v1/audio/transcriptions and fixes multipart forwarding so non-file/model fields reach the transcription handler.

Changes:

• Forward additional multipart fields (response_format, language, prompt, stream, temperature) into the transcription request JSON.
• Add verbose_json response mode that parses Whisper timestamp markers into OpenAI-style segments, while preserving the existing {model, text} response for default/json mode.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/server/server.cpp	Forwards more multipart form fields into the transcription request payload.
src/server/rest_handler.cpp	Implements verbose_json formatting by extracting segments from `<

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

temperature is parsed with std::stof(parts["temperature"].content) directly in the request handler. If the multipart field is present but empty/invalid (or uses a locale-specific format), this throws and is caught by the generic handler exception path, returning a 500 instead of a client error. Consider validating/parsing defensively (e.g., try/catch and return a structured 400 error, or ignore invalid values and let the handler apply defaults).

Suggested change

	if (parts.count("temperature")) request_json["temperature"] = std::stof(parts["temperature"].content);
	if (parts.count("temperature")) {
	const std::string& temperature_content = parts["temperature"].content;
	std::istringstream temperature_stream(temperature_content);
	temperature_stream.imbue(std::locale::classic());
	float temperature = 0.0f;
	char extra = '\0';
	if (!(temperature_stream >> temperature) || (temperature_stream >> extra)) {
	json error_response = {
	{"error", {
	{"message", "Invalid temperature: expected a numeric value."},
	{"type", "invalid_request_error"},
	{"param", "temperature"}
	}}
	};
	send_response(error_response);
	return;
	}
	request_json["temperature"] = temperature;
	}

[Copilot]

In the FASTFLOWLM_LINUX_LIMITED_MODELS branch, throw std::runtime_error(...) is followed by declarations of raw_output / language. This code is unreachable and can trigger compiler warnings (and break builds when warnings are treated as errors). Move the declarations before the throw (or remove them entirely and restructure the conditional so the variables exist in both branches without unreachable statements).

[Copilot]

verbose_json segmentation assumes there are at least 2 timestamp markers; if markers.size() < 2, the loop never runs and the response returns empty text, empty segments, and duration stays 0. This can happen for very short outputs where the model emits only the initial timestamp marker before EOS. Add a fallback for insufficient markers (e.g., strip markers and return a single segment or fall back to the non-verbose {model,text} response) so verbose_json never drops transcription text.

[ademasi]

Hi FastFlowLM team 👋

Friendly nudge on this one. I've rebased the branch onto the latest main (v0.9.43) — it's currently mergeable — and addressed all three points from the Copilot review:

• Defensive temperature parsing in the multipart dispatch, so a malformed value no longer throws a 500; it's dropped and the handler applies its default.
• Hoisted the raw_output/language declarations above the FASTFLOWLM_LINUX_LIMITED_MODELS #ifdef to remove the unreachable-code path.
• Fallback single segment when Whisper emits fewer than two <|timestamp|> markers, so verbose_json never drops text on very short clips.

Quick recap of what the PR does:

• Adds response_format=verbose_json to POST /v1/audio/transcriptions, returning an OpenAI-compatible payload with per-segment start/end timestamps (parsed from Whisper's <|X.XX|> markers), plus top-level task/language/duration.
• Fixes a multipart bug where every field except model and file was silently dropped, so response_format, language, prompt, stream, and temperature had no effect when sent over multipart.
• Fully backward-compatible: the default json path is unchanged ({model, text}).

I've been running it locally against long multi-speaker recordings to feed a diarization pipeline, and having the timestamps over the standard OpenAI API surface makes alignment trivial.

Would you be open to merging this? Happy to make any changes you'd like — for instance, if you'd prefer the segment split done at the tokenizer level rather than via regex on the decoded string, I'm glad to rework it that way. Thanks for the great project!