Skip to content

feat: run photo analysis and gaps on the device's own AI - #34

Merged
jvsena42 merged 2 commits into
mainfrom
feat/on-device-ai-toggle
Jul 27, 2026
Merged

feat: run photo analysis and gaps on the device's own AI#34
jvsena42 merged 2 commits into
mainfrom
feat/on-device-ai-toggle

Conversation

@jvsena42

Copy link
Copy Markdown
Owner

Summary

AI features currently require a user-supplied Anthropic key, so users without one get nothing, and every clothing photo leaves the device. Both platforms now ship a capable on-device model, so this adds an On-device AI toggle to Settings → AI Features. When it's on, photo analysis and gap recommendations run locally through Gemini Nano (ML Kit GenAI) on Android and Apple Intelligence (FoundationModels) on iOS. The toggle is disabled, with a subtitle explaining why, when the device can't provide a model.

On-device availability now also counts as "AI available" for feature gating, so a user with Apple Intelligence and no Claude key gets those features unlocked.

Two research findings shaped the design:

  • FoundationModels is Swift-only. Kotlin/Native interop reaches C and Objective-C only, so unlike Vision in BackgroundRemover.ios.kt it cannot be called from iosMain. iOS goes through a Swift implementation registered into Kotlin at launch. It needs iOS 26 while the deployment target is 18.2 — handled with #available plus weak linking, no target bump.
  • Android has no system Gemma. Gemma via MediaPipe would mean a 3–4.4 GB app-managed download. Gemini Nano ships with the OS and exposes checkStatus(), which is exactly the signal the "disabled when unavailable" toggle needs. The Android engine is isolated behind one interface, so a Gemma backend can be added later without touching shared code.

Changes

  • Shared AI layer — new data/source/ai package holding AiPrompts, AiResponseModels and AiResponseParser, extracted from ClaudeApiClient so both providers use one JSON contract. The parser gained stripCodeFence(): small models fence their JSON despite being told not to.
  • OnDeviceAiEngine — a plain interface with platform impls bound in Koin (the SecretStore idiom, not BackgroundRemover's expect class), so the shared prompt/parsing logic is testable against a fake. It's a thin text-in/text-out primitive; OnDeviceAiSource owns the prompts and parsing.
  • AndroidAndroidOnDeviceAiEngine on com.google.mlkit:genai-prompt:1.0.0-beta2. Written against the real API surface read out of the AAR with javap: ImagePart takes ByteArray directly (no BitmapFactory), and GenerateContentRequest.Builder.promptPrefix serves as the system-instruction slot.
  • iOSOnDeviceAiService.swift implements the Kotlin OnDeviceAiBridge, registered in iOSApp.init(). Availability crosses the boundary as a Kotlin enum rather than strings, which this repo already proves bridges cleanly. -weak_framework FoundationModels added to Config.xcconfig.
  • RoutingWardrobeRepositoryImpl branches on the preference in analyzeAndTag and getGapRecommendations, per ARCHITECTURE.md. No fallback to Claude on a local failure: opting in has to mean nothing is uploaded. Try-It stays cloud-only — it reasons over the whole wardrobe against a new photo, which a small local model handles poorly.
  • SettingsbooleanPreferencesKey("on_device_ai_enabled") in DataStore, plus getOnDeviceAiAvailability() and a derived isAiAvailable() on SettingsRepository. SettingsViewModel auto-clears the preference if the device loses support (Apple Intelligence switched off, model evicted).
  • GatinghasApiKeyisAiAvailable in WardrobeState and GapsState, and ai_locked_description reworded to mention both options. TryItState.hasApiKey is unchanged.
  • UISettingsToggleCard (Compose, Material3 Switch) and settingsToggleCard (SwiftUI Toggle), listed above the Claude key card since it's the free and private option. Three Compose previews including the disabled state.
  • Strings — seven new keys plus the reworded lock message in all four localization files (en + pt-BR, Android + iOS).
  • Tests — new OnDeviceAiSourceTest, SettingsViewModelTest and FakeOnDeviceAiEngine; routing and no-fallback cases added to WardrobeRepositoryImplTest; WardrobeViewModelTest extended for the new gating.

Test plan

Automated (all green):

  • ./gradlew detekt
  • ./gradlew :shared:allTests — 38 tests across the three touched/new classes, 0 failures
  • ./gradlew :composeApp:assembleDebug
  • ./gradlew :shared:compileKotlinIosSimulatorArm64 — not covered by CI

Manual:

  • Android on a Gemini Nano device (Pixel 8+/Galaxy S24+): toggle is enabled; turn it on, then with no Claude key and airplane mode on, add a photo and run AI analysis — tags should come back. Confirm the Gaps tab loads AI recommendations offline, and that Try-It still shows the locked state.
  • Android on an unsupported device/emulator: toggle greyed out with the unsupported subtitle and cannot be switched on.
  • iOS on an Apple Intelligence device running iOS 26: same walkthrough.
  • iOS 18.x simulator: confirm the app launches (this is what validates the weak link) and the toggle is disabled with the unsupported-OS subtitle.
  • Toggle on, disable Apple Intelligence in iOS Settings, reopen Worn Settings: the toggle should have auto-cleared and be disabled.
  • Switch to pt-BR and check every new string on both platforms.

Important

The Swift has not been compiled. linkDebugFrameworkIosSimulatorArm64 is skipped on Linux — Apple framework linking needs macOS — so OnDeviceAiService.swift and the SwiftUI toggle are unverified. The iOS 26 image-attachment call is a best reading of the API and is the most likely thing to need adjusting:

try await session.respond(to: Prompt { userText; Attachment(ImageAttachmentContent(image)) })

Please check it against the SDK in Xcode. SystemLanguageModel.default.availability and LanguageModelSession(instructions:) are on firmer ground.

Also worth a look before merge: genai-prompt is at beta2, and release builds minify — if R8 strips the ML Kit GenAI classes, keep rules will be needed in composeApp/proguard-rules.pro.

Checklist

  • ./gradlew detekt passes
  • Tested on Android
  • Tested on iOS
  • Updated documentation (if applicable)

🤖 Generated with Claude Code

jvsena42 and others added 2 commits July 27, 2026 20:18
The system prompts, wardrobe summary and JSON-to-domain mapping were private
to ClaudeApiClient. A second AI provider needs all of it, and both must ask
for and parse the exact same JSON contract, so move it to a provider-neutral
data/source/ai package. Claude's public API and wire format are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds an "On-device AI" toggle to Settings, disabled with an explanatory
subtitle when the device cannot provide a local model. When on, photo
analysis and gap recommendations run through Gemini Nano (ML Kit GenAI) on
Android and Apple Intelligence (FoundationModels) on iOS, so users without a
Claude key get AI features and their photos never leave the device.

The engine is an interface with per-platform implementations bound in Koin
rather than an expect class, so the shared prompt and parsing logic stays
testable against a fake. Provider choice lives in WardrobeRepositoryImpl per
ARCHITECTURE.md. There is deliberately no fallback to Claude when a local
call fails: opting in has to mean nothing is uploaded.

FoundationModels is Swift-only and unreachable from Kotlin/Native interop, so
iOS goes through a Swift bridge registered at launch. It needs iOS 26 while
the app deploys to 18.2, hence #available guards plus weak linking.

Try-It still uses Claude — it reasons over the whole wardrobe against a new
photo, which a small local model handles poorly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jvsena42
jvsena42 enabled auto-merge July 27, 2026 23:19
@jvsena42
jvsena42 disabled auto-merge July 27, 2026 23:20
@jvsena42
jvsena42 merged commit 6c0494a into main Jul 27, 2026
1 check passed
@jvsena42
jvsena42 deleted the feat/on-device-ai-toggle branch July 27, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant