Skip to content

Support OpenAI verbose_json for /v1/audio/transcriptions (+ multipart field fix) #510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ademasi wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Support OpenAI verbose_json for /v1/audio/transcriptions (+ multipart field fix) #510

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
71abe2b
fix(server): forward multipart audio fields to transcription handler
ademasi Apr 23, 2026
f4672a8
feat(server): support OpenAI verbose_json for audio transcriptions
ademasi Apr 23, 2026
e1c2516
fix(server): defensive temperature parse in multipart dispatch
ademasi Apr 25, 2026
90c6a51
refactor(server): hoist raw_output/language above transcription #ifdef
ademasi Apr 25, 2026
17ad979
fix(server): fallback segment for verbose_json with <2 markers
ademasi Apr 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 109 additions & 20 deletions src/server/rest_handler.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
#include <iomanip>
#include <locale>
#include <random>
#include <regex>
#include "server.hpp"

///@brief Normalize messages by merging consecutive user messages (like Ollama does)
Expand Down Expand Up @@ -1259,35 +1260,123 @@ void RestHandler::handle_openai_audio_transcriptions(const json& request,
bool stream = request.value("stream", false);
json response;
if (this->asr) {
std::string response_format = request.value("response_format", std::string("json"));
bool want_verbose = (response_format == "verbose_json");
std::string raw_output;
std::string language;
#ifndef FASTFLOWLM_LINUX_LIMITED_MODELS
this->whisper_engine->load_audio(audio_raw);
header_print("FLM", "Transforming audio to text...");
// Show text
std::cout << "Audio content: " << std::flush;
std::pair<std::string, std::string> audio_result = this->whisper_engine->generate(Whisper::whisper_task_type_t::e_transcribe, true, false, std::cout);
std::string audio_context = audio_result.first;
std::pair<std::string, std::string> audio_result = this->whisper_engine->generate(
Whisper::whisper_task_type_t::e_transcribe,
true,
want_verbose,
std::cout);
raw_output = audio_result.first;
language = audio_result.second;
std::cout << std::endl;
#else
throw std::runtime_error("ASR models are not supported in this build");
std::string audio_context;
#endif
Comment on lines 1279 to 1281

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the FASTFLOWLM_LINUX_LIMITED_MODELS branch, throw std::runtime_error(...) is followed by declarations of raw_output / language. This code is unreachable and can trigger compiler warnings (and break builds when warnings are treated as errors). Move the declarations before the throw (or remove them entirely and restructure the conditional so the variables exist in both branches without unreachable statements).

Copilot uses AI. Check for mistakes.

response = {
{"model", model},
{"text", audio_context}
//{"usage", {
// {"type", "tokens"},
// {"input_tokens", 0},
// {"input_tokens_details", json::array({
// {
// {"text_tokens", 0},
// {"audio_tokens", 0}
// }
// })},
// {"output_tokens", 0},
// {"total_tokens", 0}
//}}
};
static const std::regex ts_regex(R"(<\|(\d+\.\d+)\|>)");

if (want_verbose) {
std::vector<std::tuple<float, size_t, size_t>> markers;
auto it = std::sregex_iterator(raw_output.begin(), raw_output.end(), ts_regex);
auto end = std::sregex_iterator();
for (; it != end; ++it) {
markers.emplace_back(
std::stof((*it)[1].str()),
static_cast<size_t>(it->position()),
static_cast<size_t>(it->position() + it->length()));
}

json segments = json::array();
std::string plain_text;
float max_end = 0.0f;
for (size_t i = 0; i + 1 < markers.size(); ++i) {
float seg_start = std::get<0>(markers[i]);
float seg_end = std::get<0>(markers[i + 1]);
size_t text_begin = std::get<2>(markers[i]);
size_t text_end = std::get<1>(markers[i + 1]);
if (text_end < text_begin) continue;
std::string seg_text = raw_output.substr(text_begin, text_end - text_begin);
size_t a = seg_text.find_first_not_of(" \t\n\r");
if (a == std::string::npos) continue;
size_t b = seg_text.find_last_not_of(" \t\n\r");
seg_text = seg_text.substr(a, b - a + 1);
if (seg_text.empty()) continue;

segments.push_back({
{"id", (int)segments.size()},
{"seek", 0},
{"start", seg_start},
{"end", seg_end},
{"text", std::string(" ") + seg_text},
{"tokens", json::array()},
{"temperature", 0.0},
{"avg_logprob", 0.0},
{"compression_ratio", 0.0},
{"no_speech_prob", 0.0}
});
if (!plain_text.empty()) plain_text += " ";
plain_text += seg_text;
if (seg_end > max_end) max_end = seg_end;
}

// Fallback: if no segments were emitted (e.g. fewer than two
// timestamp markers were produced for very short clips, or every
// candidate segment trimmed to empty), surface the raw transcript
// as a single segment so verbose_json never drops text the model
// actually produced.
if (segments.empty()) {
std::string fallback_text = std::regex_replace(raw_output, ts_regex, "");
size_t fa = fallback_text.find_first_not_of(" \t\n\r");
if (fa != std::string::npos) {
size_t fb = fallback_text.find_last_not_of(" \t\n\r");
fallback_text = fallback_text.substr(fa, fb - fa + 1);
float fb_start = markers.empty() ? 0.0f : std::get<0>(markers.front());
float fb_end = markers.empty() ? 0.0f : std::get<0>(markers.back());
segments.push_back({
{"id", 0},
{"seek", 0},
{"start", fb_start},
{"end", fb_end},
{"text", std::string(" ") + fallback_text},
{"tokens", json::array()},
{"temperature", 0.0},
{"avg_logprob", 0.0},
{"compression_ratio", 0.0},
{"no_speech_prob", 0.0}
});
plain_text = fallback_text;
if (fb_end > max_end) max_end = fb_end;
}
}

response = {
{"task", "transcribe"},
{"language", language},
{"duration", max_end},
{"text", plain_text},
{"segments", segments},
{"model", model}
};
Comment on lines +1285 to +1366

Copilot AI Apr 24, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verbose_json segmentation assumes there are at least 2 timestamp markers; if markers.size() < 2, the loop never runs and the response returns empty text, empty segments, and duration stays 0. This can happen for very short outputs where the model emits only the initial timestamp marker before EOS. Add a fallback for insufficient markers (e.g., strip markers and return a single segment or fall back to the non-verbose {model,text} response) so verbose_json never drops transcription text.

Copilot uses AI. Check for mistakes.
} else {
std::string plain_text = std::regex_replace(raw_output, ts_regex, "");
size_t a = plain_text.find_first_not_of(" \t\n\r");
if (a == std::string::npos) plain_text.clear();
else {
size_t b = plain_text.find_last_not_of(" \t\n\r");
plain_text = plain_text.substr(a, b - a + 1);
}
response = {
{"model", model},
{"text", plain_text}
};
}
}
else {
header_print("Warning", "No asr model loaded, cannot load audio file");
Expand Down
11 changes: 11 additions & 0 deletions src/server/server.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1034,6 +1034,17 @@ std::unique_ptr<WebServer> create_lm_server(model_list& models, ModelDownloader&
json request_json;
request_json["model"] = parts["model"].content;
request_json["file"] = parts["file"].content;
if (parts.count("response_format")) request_json["response_format"] = parts["response_format"].content;
if (parts.count("language")) request_json["language"] = parts["language"].content;
if (parts.count("prompt")) request_json["prompt"] = parts["prompt"].content;
if (parts.count("stream")) request_json["stream"] = (parts["stream"].content == "true");
if (parts.count("temperature")) {
try {
request_json["temperature"] = std::stof(parts["temperature"].content);
} catch (const std::exception&) {
// malformed temperature: drop it and let the handler apply its default
}
}
rest_handler->handle_openai_audio_transcriptions(request_json, send_response, send_streaming_response, cancellation_token);
});

Expand Down