Built-in default prompt hallucinates Chinese and drops timestamps/speaker labels on English audio

[LCF2764]

Summary

Using the built-in default prompt through build_transcription_messages(audio_path), deterministic generation produces unrelated Chinese text for a short English clip. The output also omits both timestamps and speaker labels, so it does not follow the documented canonical format.

The same incorrect result was observed in the original batch run and reproduced again with the minimal single-file inference path below.

Test audio

Download EN_hF2n-eMHlqY_W000015.opus

• Duration: 5.2865 seconds
• Codec: Opus
• Sample rate: 48 kHz
• Channels: mono
• Size: 50,504 bytes
• SHA-256: d7802247eebdb9c324dc038270addff280c75171240bfeacf5301723bc8e9ce8

Reproduction

The important detail is that no custom prompt is passed, so the repository's built-in default transcription/diarization prompt is used.

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
    build_transcription_messages,
    generate_transcription,
    resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "EN_hF2n-eMHlqY_W000015.opus"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(dtype=dtype).to(device).eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
    model,
    processor,
    messages,
    max_new_tokens=2048,
    do_sample=False,
    device=device,
    dtype=dtype,
)

print(result)

Actual output

{'text': '[也没问题。在其他地方，比如在尼泊尔，尼泊尔的居民们经常在尼泊尔的海滩上进行游荡。', 'prompt_len': 153, 'generated_tokens': 33}

This is unrelated Chinese content for English speech. It has no valid [start_time][Sxx]text[end_time] segment, so no speaker or timestamp can be parsed.

Expected output

The English speech is approximately:

In particular, the location of Nain is very close to Shunem, identified with modern Sulam.

Minor proper-noun spelling differences are acceptable, but the result should preserve the source language and use the documented structure. For example, the same model produced this when given a stricter custom prompt:

[0.00][S01]In particular, the location of Nine is very close to Shunam, identified with modern Sullam.[5.12]

Prompt sensitivity

Appending the following explicit format constraint to the Chinese prompt avoids the hallucination for this file:

格式严格限制为 [起始秒数][S01]文本[结束秒数]，时间戳、说话人编号、文本缺一不可。

This suggests the built-in default prompt is not robustly enforcing the advertised output format for this input.

Environment

• Model: OpenMOSS-Team/MOSS-Transcribe-Diarize (local download)
• Repository inference code: b5ad0f8
• Python: 3.12.13
• PyTorch: 2.11.0+cu128
• Transformers: 5.13.0
• Accelerate: 1.12.0
• GPU: NVIDIA A800-SXM4-80GB
• Inference dtype: bfloat16
• do_sample=False
• max_new_tokens=2048

[zimoCCC]

Summary

Using the built-in default prompt through build_transcription_messages(audio_path), deterministic generation produces unrelated Chinese text for a short English clip. The output also omits both timestamps and speaker labels, so it does not follow the documented canonical format.

The same incorrect result was observed in the original batch run and reproduced again with the minimal single-file inference path below.

Test audio

Download EN_hF2n-eMHlqY_W000015.opus

• Duration: 5.2865 seconds
• Codec: Opus
• Sample rate: 48 kHz
• Channels: mono
• Size: 50,504 bytes
• SHA-256: d7802247eebdb9c324dc038270addff280c75171240bfeacf5301723bc8e9ce8

Reproduction

The important detail is that no custom prompt is passed, so the repository's built-in default transcription/diarization prompt is used.

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
build_transcription_messages,
generate_transcription,
resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "EN_hF2n-eMHlqY_W000015.opus"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype="auto",
).to(dtype=dtype).to(device).eval()
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
model,
processor,
messages,
max_new_tokens=2048,
do_sample=False,
device=device,
dtype=dtype,
)

print(result)

Actual output

{'text': '[也没问题。在其他地方，比如在尼泊尔，尼泊尔的居民们经常在尼泊尔的海滩上进行游荡。', 'prompt_len': 153, 'generated_tokens': 33}
This is unrelated Chinese content for English speech. It has no valid [start_time][Sxx]text[end_time] segment, so no speaker or timestamp can be parsed.

Expected output

The English speech is approximately:

In particular, the location of Nain is very close to Shunem, identified with modern Sulam.

Minor proper-noun spelling differences are acceptable, but the result should preserve the source language and use the documented structure. For example, the same model produced this when given a stricter custom prompt:

[0.00][S01]In particular, the location of Nine is very close to Shunam, identified with modern Sullam.[5.12]

Prompt sensitivity

Appending the following explicit format constraint to the Chinese prompt avoids the hallucination for this file:

格式严格限制为 [起始秒数][S01]文本[结束秒数]，时间戳、说话人编号、文本缺一不可。

This suggests the built-in default prompt is not robustly enforcing the advertised output format for this input.

Environment

• Model: OpenMOSS-Team/MOSS-Transcribe-Diarize (local download)
• Repository inference code: b5ad0f8
• Python: 3.12.13
• PyTorch: 2.11.0+cu128
• Transformers: 5.13.0
• Accelerate: 1.12.0
• GPU: NVIDIA A800-SXM4-80GB
• Inference dtype: bfloat16
• do_sample=False
• max_new_tokens=2048

Thank you very much for the detailed feedback. The issue has been reproduced using the audio file you provided, and several similar reports have also been received recently. Improvements will be made as soon as possible and included in the next release. We sincerely apologize for the inconvenience.