ASR: /v1/audio/transcriptions ignores response_format — timestamps are computed internally but discarded

[LalaDango]

Hi! Thanks for FLM — the NPU ASR path has been really useful for me.

While reading the source I noticed that /v1/audio/transcriptions always returns a minimal {model, text} object, and that most of the pieces needed for OpenAI's verbose_json response already exist inside the Whisper engine but are dropped at the REST layer. Filing this in case it's simply "not wired up yet" rather than a deliberate decision — apologies if it's already on the roadmap.

Checked against master @ 8b8399c. Environment: FLM v0.9.45, Windows, AMD Ryzen AI 5 340 — though this comes from reading the source rather than a runtime failure, so the environment probably isn't relevant here.

Current behavior

response_format is never read on the server side (grepping for response_format / verbose_json in src/ returns nothing), so a client asking for verbose_json silently gets plain text back. Anything that needs segment timing has to re-chunk the audio client-side and stitch the offsets back together.

Handler:

FastFlowLM/src/server/rest_handler.cpp

Lines 1290 to 1301 in 8b8399c

	std::pair<std::string, std::string> audio_result = this->whisper_engine->generate(Whisper::whisper_task_type_t::e_transcribe, true, false, std::cout);
	std::string audio_context = audio_result.first;
	std::cout << std::endl;
	#else
	throw std::runtime_error("ASR models are not supported in this build");
	std::string audio_context;
	#endif
	response = {
	{"model", model},
	{"text", audio_context}
	//{"usage", {

std::pair<std::string, std::string> audio_result =
    this->whisper_engine->generate(Whisper::whisper_task_type_t::e_transcribe, true, false, std::cout);
//                                                                            ^^^^  ^^^^^
//                                                    enable_time_stamp=true, return_time_stamp=false

std::string audio_context = audio_result.first;
...
response = { {"model", model}, {"text", audio_context} };

What the engine already produces

1. Timestamps are computed. When return_time_stamp is true, decoded timestamps are offset per chunk and appended to the result string:

FastFlowLM/src/common/whisper/modeling_whisper.cpp

Lines 170 to 174 in 8b8399c

	if (return_time_stamp){
	std::string time_stamp = this->tokenizer->run_time_decoder(last_idx);
	std::string offset_time_stamp = this->_offset_time_stamp(time_stamp, time_offset);
	result += offset_time_stamp;
	os << offset_time_stamp << std::flush;

2. enable_time_stamp cannot be the exposure switch — it is structurally required for the sliding-window chunking, since the next chunk boundary is derived from the last timestamp:

FastFlowLM/src/common/whisper/modeling_whisper.cpp

Line 233 in 8b8399c

float end_time = _get_time(last_time_stamp);

So the only knob that controls exposure is return_time_stamp, and it is hardcoded to false at the single call site.

3. The detected language is already returned, but the handler only uses .first:

FastFlowLM/src/common/whisper/modeling_whisper.cpp

Line 247 in 8b8399c

return std::make_pair(result, langmap::to_language_name(language_detected));

return std::make_pair(result, langmap::to_language_name(language_detected));

4. Audio duration is already known — load_audio() logs Length of audio: N seconds.

Suggestion

Read response_format in handle_openai_audio_transcriptions, and for verbose_json pass return_time_stamp=true, parse the <|x.xx|> markers out of the result string into segments[], and fill language / duration from what the engine already hands back:

{
  "task": "transcribe",
  "language": "japanese",
  "duration": 123.4,
  "text": "...",
  "segments": [
    { "id": 0, "start": 0.00, "end": 4.20, "text": "..." }
  ]
}

json and text formats would keep the current behavior, so this should be backward compatible.

Why it matters

Without segments, subtitle generation and long-audio alignment require re-implementing chunking on the client side, which duplicates work the engine is already doing (and can drift from the engine's own chunk boundaries).

Related: #234 — surfacing the detected language would partly address that request as well.

Thanks for the great work!