Skip to content

feat(recorder): on-device recorder host + whisper transcription + chat/RAG#433

Open
dishit-wednesday wants to merge 91 commits into
devfrom
feat/locket-pro
Open

feat(recorder): on-device recorder host + whisper transcription + chat/RAG#433
dishit-wednesday wants to merge 91 commits into
devfrom
feat/locket-pro

Conversation

@dishit-wednesday

@dishit-wednesday dishit-wednesday commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

TL;DR - Open-core host that wires in the private pro/ recorder submodule and ships
the shared on-device transcription + retrieval engine it relies on. Recorder feature code
stays closed-source in pro/; this PR is the public wiring + shared services. Pairs with
off-grid-ai/mobile-pro#11 (the submodule pointer references a commit on that pro branch, so
they land together).

flowchart TB
  subgraph pro["pro/ submodule (private, autolinked)"]
    rec["Recorder + AudioNormalizer"]
  end
  subgraph core["core (public host)"]
    slot["MemoryTabScreen<br/>paywall + registry slot"]
    ws["whisperService<br/>windowed transcribe + CoreML guard"]
    rag["RAG 'Recordings' KB<br/>metadata column + migration"]
    chat["chat: summarize attachment"]
  end
  slot -. "registered-screen slot" .-> rec
  rec -->|"16k mono WAV"| ws
  ws -->|"transcript + t0/t1 segments"| rag
  ws -->|"transcript"| chat
Loading

What's in this PR

  • Recorder host - Memory tab -> Recorder; Settings moved to Home header; guarded pro
    native autolink (react-native.config.js); iOS background-audio mode.
  • Whisper - windowed / resumable / memory-bounded transcription; CoreML asset guard;
    .en language forcing; iOS streaming-lifetime patch (whisper.rn+0.5.5).
  • Chat / RAG - recordings knowledge base (chunk metadata column + migration);
    summarize oversized transcript/document attachments.
  • Integration - .gitmodules re-pointed to canonical off-grid-ai/mobile-pro;
    submodule pointer bumped to the merged pro commit.
  • Temporary - reset-license testing controls (remove/re-gate before shipping).

How it works (expand for detail)

Transcription pipeline (whisper.cpp via whisper.rn)
  • Windowed, resumable, memory-bounded. transcribeFile drives whisper.rn with an
    {offset, duration} window so long recordings transcribe in chunks and resume from a
    checkpoint (last finished chunk) instead of reloading the whole file. Live cumulative
    segments surface through whisper.rn's onNewSegments (native new_segment_callback),
    wired only when nProcessors == 1 (parallel decode cannot stream in order).
  • iOS streaming lifetime patch (whisper.rn+0.5.5). The file path
    (transcribeFile -> ObjC transcribeData) crashed because new_segment_callback's
    user_data was a stack struct freed before the callback fired. The patch hoists it so it
    outlives the call; compiled into the iOS binary, so live segments work on both platforms.
  • CoreML asset guard. Enabling CoreML without the per-model
    ggml-<model>-encoder.mlmodelc bundle makes whisper.rn fail CoreML init, fall back to
    CPU, then crash at transcribe 0% on some devices (A12 / iPhone XS). loadModel checks the
    encoder asset exists and downgrades useCoreMLIos to false when missing, so the toggle is
    a no-op instead of a crash.
  • English-only language forcing. .en models carry only English tokens; auto-detect
    (returns codes like "tg") or an explicit non-English pick makes whisper unstable on iOS
    (crash at 0%, 762s for 13%, garbled). transcribeFile forces language: 'en' for any
    model whose catalogue lang === 'en', falling back to the .en.bin filename convention.
Retrieval (recordings knowledge base)

Transcripts index into a local RAG "Recordings" KB. rag_chunks gains a metadata TEXT
column (recordingId, startMs, eventTitle, ...) via a guarded ALTER TABLE migration for
older installs; chunk metadata is JSON-serialized on write.

Integration / build
  • The pro native library is autolinked through a guarded react-native.config.js (only
    when the pro android/iOS sources are present), so closed-source code never leaks into the
    public repo but builds when the submodule is checked out.
  • .gitmodules re-pointed from the old wednesday-solutions/offgrid-pro to the canonical
    off-grid-ai/mobile-pro; submodule pointer bumped to the merged pro commit.

Testing

npm run lint (0 errors), tsc --noEmit (clean), and the full jest suite pass; the
pre-push hook ran them, no --no-verify used. Added CoreML-fallback + RAG-metadata coverage.

Follow-ups (not blocking)

  • Slot-key indirection: MemoryTabScreen looks up the recorder by the concrete pro
    screen name; move it behind an abstract slot key so public never names a pro internal.
  • whisperService max-lines: deferred with a TODO; clearing it means moving model
    download/management into its own module (~11 call sites across core + pro).
  • Walkthrough video on the PR.

Reviewer's guide (commits by theme)

The branch is chronological; this groups the commits by area.

  • Recorder host + navigation - rename Memory tab to Recorder; Memory tab (Pro
    recorder/paywall) + move Settings to Home header; autolink pro native + pod install; iOS
    background-audio mode.
  • Whisper transcription - core file-transcription; offset/duration chunking + live
    segments + CoreML; CoreML asset guard + .en forcing + tdrz; iOS streaming lifetime
    patch; memory-snapshot logging; whisper remote-server provider type.
  • Chat / RAG - recordings KB (chunk metadata, indexText, seed project); summarize
    oversized attachments; stream summaries live.
  • Integration (submodule) - 4x bump pro submodule; point submodule at canonical
    off-grid-ai/mobile-pro.

dishit-wednesday and others added 30 commits June 28, 2026 03:00
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…, stop, native log)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…option (core)

whisperService.transcribeFile gains offset/duration so a recording can be
transcribed in chunks, and streams segments live via onNewSegments (cumulative
result + per-segment t0/t1). loadModel takes useGpu/useFlashAttn/useCoreML and
passes them to initWhisper (useCoreMLIos on iOS). Bumps the pro submodule to
the chunked/resumable transcription experience.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…header

Replaces the Settings bottom tab with a Memory tab. MemoryTabScreen renders the
Pro recorder when it is registered (pro.activate) and a paywall otherwise, read
reactively so unlocking Pro swaps it in without a restart - the tab shell stays
in core, the recorder injects from the submodule. Settings becomes a pushed
RootStack screen reached from a gear in the Home header. Updates nav types and
the onboarding step that pointed at the old SettingsTab, and the SettingsScreen
nav-prop type (no longer a tab).

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Adds a map-reduce summarizer (transcriptSummarizer) for text too large for the
context window: split into context-sized chunks, summarize each, then combine
(recursing if needed) - so the whole transcript is read instead of truncated to
the tail like contextCompaction does. A Summarize action on a document/transcript
attachment chip streams progress (part i/N, combining) into an assistant message,
then the final summary. MediaAttachment gains recordingId + transcript range
metadata, and a one-shot chatAttachmentInbox lets the Pro recorder hand a
transcript to the composer across the navigation boundary.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summarizing an oversized transcript used to show a static "part i of N"
counter for minutes and only stream on the final combine pass, and the
Summarize button was clipped invisible inside the 60x60 attachment chip.

- generateWithMaxTokens gains an optional onToken callback.
- transcriptSummarizer streams each map part as it is written (new 'combining'
  phase marks the final pass), so text flows from part 1, not part 3.
- useSummarizeAttachment renders the live work (parts stack while streaming,
  then the combined answer streams over the top) on a 50ms flush so the JS
  thread is not saturated by per-token deep store updates.
- Document/transcript chips get a larger 168x76 layout with a filled, visible
  Summarize button and a spinner+label busy state.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…er.rn 0.5.5)

Hoist the segment-callback user_data to the dispatch-block scope so live segment streaming on the iOS file-transcribe path does not deref a dangling pointer; add an NSLog marker to verify the patch is compiled into a build.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…d project

Thread per-chunk metadata (recordingId/startMs/eventTitle) through chunking, the SQLite store, and retrieval; add indexText() to index a raw transcript; seed a 'Recordings' project via ensureProject so recordings can be asked about.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Move the exact timestamp out of the system prompt (which busted llama.rn's prefix cache every turn) and onto the latest user message; keep only the stable date in the system prompt so the ~800-token system+tools prefix is reused. Add tool-loop diagnostic logging.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… model

Guard CoreML so it falls back to CPU when the per-model encoder asset is absent (was crashing on A12); force 'en' for English-only ggml-*.en models; add the small.en-tdrz model entry; re-enable iOS live segment streaming.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Used by transcribeChunked to log per-chunk memory footprint.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…tream fast

On CPU-only low-RAM devices, reasoning/preamble burned the whole token
budget before any summary streamed. Force the thinking channel off for
internal generations (summarize, tool-selection), instruct the prompts to
emit no preamble, strip any leaked control tokens, and cap each MAP chunk
to ~1500 input tokens so each part prefills in under a minute and starts
streaming sooner. Reduce/combine passes still use the full context budget.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Snapshot RAM before and after loadModel. On 4 GB iOS devices a large
whisper model can push the app past the jetsam limit and the OS kills it
mid-load, which presents as a crash; the before/after pair localizes a
kill to load vs transcription. Fire-and-forget so it adds no await points
to the load critical path.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The tab and paywall are about on-device recording/transcription, so call
it Recorder. Tab label, testID (recorder-tab), and paywall title updated.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
loadModel gained an options param (CoreML/lang config); update the store
tests to match the new call signature.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Turn the bare gating card into a feature list - always-on recording (Android),
on-device transcription, summaries, calendar context - with a privacy line and
the unlock CTA. Copy follows the brand voice (privacy-as-mechanism, no jargon).

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Points the pro submodule at the locket recorder feature set: continuous
recorder, on-device transcription, calendar matching, summaries, knowledge-base
sync, and the recording-detail/list UI polish.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Update tests left stale by intended code changes, and add coverage for new
behavior. No production code changed; each assertion was verified against the
current implementation (the source of truth), not weakened to pass.

- nav/onboarding: Memory tab renamed to Recorder (recorder-tab), and Settings
  moved off the tab bar to a Home-header route, so STEP_TAB_MAP.exploredSettings
  is 'Settings' and the 5th tab label/testID is Recorder.
- whisper: loadModel now forwards {useGpu,useFlashAttn,useCoreMLIos}; .en models
  force language 'en' (plain .bin auto-detects).
- rag: schema gained a metadata column (4th CREATE = ALTER migration; 4th insert
  bind = metadata, null when absent).
- generationToolLoop: the precise time-of-day note now rides on the latest user
  message (keeps the system+tools prefix cacheable), not the system prompt.
- whisperService.branches: reset RNFS exists/stat mocks per test - clearAllMocks
  does not drain the mockResolvedValueOnce queue, so onces leaked between tests.

Added: CoreML-encoder-missing -> CPU fallback guard, and RAG metadata JSON
serialization on the write path.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
RNWhisper.mm's vadDetectSpeech/vadDetectSpeechFile decode the WAV to a malloc'd
float PCM buffer and pass it to detectSpeech synchronously, but never free it.
Over a long recording that leaks ~one slice of PCM per call and OOMs. Add
free(data) after the synchronous detectSpeech consumes it, in both methods.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Add ungated controls to clear the cached/stored Pro license so the
free -> activate flow can be re-tested on a release build (where the __DEV__
toggle is inert). The Keygen machine slot stays claimed (fingerprint persists),
so re-pasting the key re-activates instantly. Marked TEMPORARY - re-gate behind
__DEV__ or remove before shipping.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Point at the pro commits adding the recorder transcription/archive UI polish and
the speech/noise/silence cleanup classifier.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The ChatInput render fn was 358 lines (limit 350) - the #431 lint blocker. Lift
the send/stop/voice action button into a module-scope ActionButton component
(derives colors/styles from the same hooks; behaviour identical to the previous
inline ternary). Render fn now ~343 lines. 91/91 ChatInput tests pass, tsc clean.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The submodule URL still referenced the old wednesday-solutions/offgrid-pro repo.
Point it at the canonical off-grid-ai/mobile-pro so CI and fresh clones fetch the
pro recorder code from the right place.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
dishit-wednesday and others added 3 commits July 13, 2026 00:24
…a, prompt fix, recorder baseline)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…p (ec24cac)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… default

whisperStore deleteModel/deleteModelById now fall back to another on-disk model (mock listDownloadedModels); loadModel auto-enables CoreML when the encoder asset is present.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
dishit-wednesday and others added 24 commits July 13, 2026 02:11
When a dev-grammar completion fails and retries without the grammar, the streaming state carried over from the failed attempt, duplicating/garbling the retry's output (and re-emitting via onStream). Reset fullContent/fullReasoningContent/streamed* in generateResponse, and fullResponse/collectedToolCalls + the ToolCallTokenFilter buffer in generateWithToolsImpl. Matches the existing reset in generateWithMaxTokens. (Gitar code-review finding.)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…-transcribe off + recorder comment)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… delete

deleteModel updated downloadedModelId (fallback) but left the deleted model in presentModelIds, so the models list kept showing a model whose file is gone. Recompute presentModelIds from disk, matching deleteModelById. (Gitar re-review finding.)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ropout)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ndoff doc

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…oisy audio

whisper.cpp -mc 0: don't condition on previously-decoded text, which sent it into a repeat-the-same-token spiral on ambient clips.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…-gated-recorder

# Conflicts:
#	.gitignore
#	android/app/src/main/assets/ggml-hexagon/libggml-htp-v69.so
#	android/app/src/main/assets/ggml-hexagon/libggml-htp-v73.so
#	android/app/src/main/assets/ggml-hexagon/libggml-htp-v75.so
#	android/app/src/main/assets/ggml-hexagon/libggml-htp-v79.so
#	android/app/src/main/assets/ggml-hexagon/libggml-htp-v81.so
#	pro
#	src/services/llmHelpers.ts
#	src/stores/whisperStore.ts
Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summaries prefer a remote provider whenever active (offloads generation off-device to save battery/RAM); does not gate on a local model being loaded. Chat generation keeps its own local-first policy.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Forwards a short proper-noun/jargon list to whisper.cpp as its initial prompt so names are spelled the user's way. Pairs with the pro-side vocabulary setting.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…GC-time UAF

Insight generation churns conversations (reset->close->recreate). Each finished generation leaves fbjni HybridData peers to be reclaimed on the GC 'HybridData Dest' thread; under churn that reclaim can fire mid-decode of the next conversation and dereference freed memory (observed SIGSEGV, poison 0x01). Drain the reference queue at the quiescent teardown point (gc + runFinalization + brief yield) so reclaims do not overlap live native work.

Includes temporary DIAG-UAF lifecycle logging to confirm the fix on-device; strip before merge.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…, permissions, iOS recorder, VAD teardown fix)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Pure selectTextModelToLoad(models, budgetMB, { activeId, footprintMB }): active-if-fits -> largest-that-fits -> smallest. Footprint is injected (caller passes hardwareService.estimateModelRam) so weights + mmproj + overhead count the one canonical way. Unit-tested for every branch.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
loadTextModel({ textOnly: true }) skips loading the mmproj (saving its RAM) and sizes the residency budget on the gguf weights alone. Gated - chat still loads vision by default. Guards the post-load mmproj-clear so a text-only load never erases a model's persisted vision link.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…n prompt)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…s, auto-prune)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
… redesing.md)

android/temp.txt was a 1532-line AI planning transcript; redesing.md a stray design note with broken image refs. Neither belongs in the repo (flagged by review).

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
Summaries prefer a remote provider, so a local-only stopGeneration left a remote completion running after Stop. abort() now also calls the active provider's stopGeneration.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…ot path

Removes the DIAG-UAF create/send/close Log.i lines and the convSeq/currentConvSeq bookkeeping added to diagnose the GC-time HybridData UAF. Keeps the mitigation itself (gc + runFinalization drain at close).

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…prompt)

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…nscription

cleanTranscription only kept text containing [a-z0-9], so a Hindi/Arabic/CJK
transcript (no ASCII letters) was wiped to '' - silent data loss for non-English
users. Match any-script letters/digits (\p{L}\p{N}) instead. Also: the single
shared Whisper context can't run two file transcriptions at once, so a second
overlapping request now throws WhisperBusyError instead of orphaning the first
job's cancel handle, and unloadModel cancels an in-flight file transcribe before
releasing the context (prevents a use-after-free SIGSEGV).

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
…budget

Expose the residency manager (the one owner of the RAM cap + load gate) from the
services barrel so callers that auto-pick a model to load can budget against the
SAME number the loader enforces (modelResidencyManager.getBudgetMB), instead of a
divergent hardcoded fraction. Used by memory-aware auto-load callers.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
The pro recorder/insights feature lives in the private pro/ submodule; its
architecture, product-design handoffs, and transcript-quality R&D notes must not
ship in the public core repo. Remove the 22 locket/recorder/insights/transcript
planning + handoff docs from docs/plans/. Keeps only genuinely-core planning
(backend-per-device, whisper-download-sync). Tree-only removal; a full history
scrub, if needed, is a separate step at the public cutover.

Co-Authored-By: Dishit Karia <hanmadishit74@gmail.com>
feat(core): summarizer streaming/dynamic-context + whisper model fallback + pro bump [WIP/divergent]
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant