Align

On-device word-timestamp refinement for Apple's Speech framework. Align corrects the word-level timings that SpeechTranscriber and SpeechAnalyzer return, without replacing them. It observes the same audio the analyzer already receives, runs a small Core ML cascade on the CPU and Neural Engine, and returns the familiar result surface with tightened audioTimeRange values. Everything runs locally, so audio never leaves the device.

Align augments Apple's pipeline instead of competing with it. When a correction would be structurally invalid, lacks streaming context, or falls on the search-window edge, Align keeps Apple's original timestamp.

Apple:   "world"  start 2.61s  end 3.04s
Align:   "world"  start 2.57s  end 2.98s

• Features
• Install
• Usage
  • SpeechAnalyzer pipeline
  • Callback-based audio
  • File input
• Result type
• Accuracy
• Languages
• Model and resources
• License

Features

• Runs fully on device. Audio never leaves the machine.
• Augments the standard SpeechTranscriber / SpeechAnalyzer pipeline with one input modifier and one result modifier. Apple stays the source of transcription.
• Fixed batch-16 Core ML cascade on CPU and Neural Engine, with an Accelerate/vDSP log-mel frontend.
• Nine languages: English, Spanish, French, Italian, Portuguese, German, Japanese, Korean, and Chinese. Unsupported languages pass through unchanged.
• Structural safety: invalid ranges, search-window edge predictions, and missing streaming context keep Apple's original timestamp.
• Streaming, callback, and file input, with no deliberate lookahead delay.
• Small compiled models (~0.7 MB), bundled by default for offline apps or supplied from your own directory.

Install

Requirements: iOS 16+, iPadOS 16+, Mac Catalyst 16+, macOS 13+, tvOS 16+, or visionOS 1+, and Swift 6.2+. The package adds cleanly to apps with these deployment targets. The SpeechAnalyzer refinement APIs are gated with @available and run on iOS 26+, iPadOS 26+, Mac Catalyst 26+, macOS 26+, tvOS 26+, and visionOS 26+ (the versions SpeechAnalyzer / SpeechTranscriber require).

Add Align with Swift Package Manager:

.package(url: "https://github.com/Desert-Ant-Labs/align.git", from: "0.1.0")

Then add the Align product to your app target. The compiled models and calibration resources are bundled by default, so the SDK works fully offline after installation.

To manage the model files yourself, disable the default BundledModel trait and supply a directory containing align_coarse.mlmodelc, align_fine.mlmodelc, mel_filters.bin, refiner_config.json, and calibrator.bin:

.package(url: "https://github.com/Desert-Ant-Labs/align.git", from: "0.1.0", traits: [])

let refiner = try SpeechTimestampRefiner(locale: locale, resourceDirectory: modelDirectory)

Applications can also depend on the AlignCoreMLResources product explicitly and pass AlignCoreMLResourcesBundle.bundle to the bundle initializer.

Usage

Create one refiner and reuse it. Construction loads the Core ML models, so build it while preparing the Apple analyzer rather than on the first result.

import Align
import Speech

let refiner = try SpeechTimestampRefiner(locale: locale)

SpeechAnalyzer pipeline

Record the audio already going into SpeechAnalyzer with one input modifier, then refine the result loop with one result modifier. SpeechTranscriber and SpeechAnalyzer remain the source of transcription; Align only observes their audio and corrects finalized timestamps.

let transcriber = SpeechTranscriber(
    locale: locale,
    transcriptionOptions: [],
    reportingOptions: [],
    attributeOptions: [.audioTimeRange]
)
let analyzer = SpeechAnalyzer(modules: [transcriber])
let refiner = try SpeechTimestampRefiner(locale: locale)

Task {
    try await analyzer.start(
        inputSequence: inputs.recordingAudio(for: refiner)
    )
}

for try await result in transcriber.results.refiningTimestamps(with: refiner) {
    // result.text has corrected word-level audioTimeRange attributes
    showCaption(result.text)
}

Analyzer preparation, finalization, cancellation, asset installation, and Speech permission stay in your existing Apple code.

Callback-based audio

If your application builds each AnalyzerInput inside an audio callback, replace:

continuation.yield(AnalyzerInput(buffer: buffer))

with:

continuation.yield(refiner.analyzerInput(buffer))          // optionally: at: startTime

That one line records the PCM and returns the same AnalyzerInput for SpeechAnalyzer.

File input

When using start(inputAudioFile:), initialize the refiner with the same file. It opens a separate handle from audioFile.url, so it does not move the position SpeechAnalyzer reads.

let audioFile = try AVAudioFile(forReading: audioURL)
let refiner = try SpeechTimestampRefiner(locale: locale, audioFile: audioFile)

try await analyzer.start(inputAudioFile: audioFile, finishAfterFile: true)

for try await result in transcriber.results.refiningTimestamps(with: refiner) {
    // result.text has corrected timestamps
}

File audio uses full-file edge handling and is not limited by the streaming rolling-buffer duration.

Result type

The refined stream yields RefinedSpeechResult, which preserves the familiar result surface and adds refinement details:

result.text                 // corrected audioTimeRange attributes
result.range
result.resultsFinalizationTime
result.isFinal

result.words                // [WordTiming]
result.refinedWordCount
result.original             // Apple's SpeechTranscriber.Result, with alternatives and metadata

Volatile results pass through unchanged; finalized results receive corrected timestamps. Apple does not expose an initializer for SpeechTranscriber.Result, so RefinedSpeechResult conforms to SpeechModuleResult, mirrors the commonly used properties, and retains the original value.

Align is deliberately not a SpeechModule. SpeechAnalyzer's modules independently analyze the same audio rather than forming a sequential pipeline, and Apple exposes no public hooks to feed a custom module. The input and result modifiers make the two required data connections using only supported public APIs.

Accuracy

Evaluated on the exact Swift runtime and bundled Core ML models over 223 clean and 210 noisy group-held-out recordings across all nine languages. The table compares Apple's raw word-boundary error against Align's, measured against forced-alignment references.

Condition	Apple raw error	Align error	Reduction	Median	Within 50 ms
Clean	113.5 ms	44.9 ms	60%	28.2 ms	75.1%
Noisy	124.4 ms	50.1 ms	60%	32.0 ms	69.4%

Error is mean absolute distance from the reference boundary. Align roughly halves Apple's typical error and removes most of its large mistakes: on clean audio the median boundary lands within about 28 ms and three quarters land within 50 ms.

References are machine forced-alignment estimates, not human annotations, so these figures show a large, consistent reduction of Apple's timing error rather than sample-accurate ground truth. Words whose correction would be structurally invalid or lack context keep Apple's original timestamp, so refinement does not regress those boundaries.

Languages

English, Spanish, French, Italian, Portuguese, German, Japanese, Korean, and Chinese. A locale outside this set is passed through unchanged, so it is always safe to construct a refiner.

Model and resources

The compiled Core ML stages, the log-mel filter bank, the refiner configuration, and the correction calibrator are bundled in the AlignCoreMLResources target and enabled by the default-on BundledModel trait. Total resource size is about 0.7 MB. Disable the trait to supply your own directory or bundle, as shown in Install.

License

Desert Ant Labs Source-Available License. Free for most apps; a commercial license is required at scale. Full terms are at the link. Licensing: licensing@desertant.com.