feat: run photo analysis and gaps on the device's own AI - #34
Merged
Conversation
The system prompts, wardrobe summary and JSON-to-domain mapping were private to ClaudeApiClient. A second AI provider needs all of it, and both must ask for and parse the exact same JSON contract, so move it to a provider-neutral data/source/ai package. Claude's public API and wire format are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds an "On-device AI" toggle to Settings, disabled with an explanatory subtitle when the device cannot provide a local model. When on, photo analysis and gap recommendations run through Gemini Nano (ML Kit GenAI) on Android and Apple Intelligence (FoundationModels) on iOS, so users without a Claude key get AI features and their photos never leave the device. The engine is an interface with per-platform implementations bound in Koin rather than an expect class, so the shared prompt and parsing logic stays testable against a fake. Provider choice lives in WardrobeRepositoryImpl per ARCHITECTURE.md. There is deliberately no fallback to Claude when a local call fails: opting in has to mean nothing is uploaded. FoundationModels is Swift-only and unreachable from Kotlin/Native interop, so iOS goes through a Swift bridge registered at launch. It needs iOS 26 while the app deploys to 18.2, hence #available guards plus weak linking. Try-It still uses Claude — it reasons over the whole wardrobe against a new photo, which a small local model handles poorly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
AI features currently require a user-supplied Anthropic key, so users without one get nothing, and every clothing photo leaves the device. Both platforms now ship a capable on-device model, so this adds an On-device AI toggle to Settings → AI Features. When it's on, photo analysis and gap recommendations run locally through Gemini Nano (ML Kit GenAI) on Android and Apple Intelligence (
FoundationModels) on iOS. The toggle is disabled, with a subtitle explaining why, when the device can't provide a model.On-device availability now also counts as "AI available" for feature gating, so a user with Apple Intelligence and no Claude key gets those features unlocked.
Two research findings shaped the design:
FoundationModelsis Swift-only. Kotlin/Native interop reaches C and Objective-C only, so unlikeVisioninBackgroundRemover.ios.ktit cannot be called fromiosMain. iOS goes through a Swift implementation registered into Kotlin at launch. It needs iOS 26 while the deployment target is 18.2 — handled with#availableplus weak linking, no target bump.checkStatus(), which is exactly the signal the "disabled when unavailable" toggle needs. The Android engine is isolated behind one interface, so a Gemma backend can be added later without touching shared code.Changes
data/source/aipackage holdingAiPrompts,AiResponseModelsandAiResponseParser, extracted fromClaudeApiClientso both providers use one JSON contract. The parser gainedstripCodeFence(): small models fence their JSON despite being told not to.OnDeviceAiEngine— a plain interface with platform impls bound in Koin (theSecretStoreidiom, notBackgroundRemover'sexpect class), so the shared prompt/parsing logic is testable against a fake. It's a thin text-in/text-out primitive;OnDeviceAiSourceowns the prompts and parsing.AndroidOnDeviceAiEngineoncom.google.mlkit:genai-prompt:1.0.0-beta2. Written against the real API surface read out of the AAR withjavap:ImageParttakesByteArraydirectly (noBitmapFactory), andGenerateContentRequest.Builder.promptPrefixserves as the system-instruction slot.OnDeviceAiService.swiftimplements the KotlinOnDeviceAiBridge, registered iniOSApp.init(). Availability crosses the boundary as a Kotlin enum rather than strings, which this repo already proves bridges cleanly.-weak_framework FoundationModelsadded toConfig.xcconfig.WardrobeRepositoryImplbranches on the preference inanalyzeAndTagandgetGapRecommendations, per ARCHITECTURE.md. No fallback to Claude on a local failure: opting in has to mean nothing is uploaded. Try-It stays cloud-only — it reasons over the whole wardrobe against a new photo, which a small local model handles poorly.booleanPreferencesKey("on_device_ai_enabled")in DataStore, plusgetOnDeviceAiAvailability()and a derivedisAiAvailable()onSettingsRepository.SettingsViewModelauto-clears the preference if the device loses support (Apple Intelligence switched off, model evicted).hasApiKey→isAiAvailableinWardrobeStateandGapsState, andai_locked_descriptionreworded to mention both options.TryItState.hasApiKeyis unchanged.SettingsToggleCard(Compose, Material3Switch) andsettingsToggleCard(SwiftUIToggle), listed above the Claude key card since it's the free and private option. Three Compose previews including the disabled state.OnDeviceAiSourceTest,SettingsViewModelTestandFakeOnDeviceAiEngine; routing and no-fallback cases added toWardrobeRepositoryImplTest;WardrobeViewModelTestextended for the new gating.Test plan
Automated (all green):
./gradlew detekt./gradlew :shared:allTests— 38 tests across the three touched/new classes, 0 failures./gradlew :composeApp:assembleDebug./gradlew :shared:compileKotlinIosSimulatorArm64— not covered by CIManual:
Important
The Swift has not been compiled.
linkDebugFrameworkIosSimulatorArm64is skipped on Linux — Apple framework linking needs macOS — soOnDeviceAiService.swiftand the SwiftUI toggle are unverified. The iOS 26 image-attachment call is a best reading of the API and is the most likely thing to need adjusting:Please check it against the SDK in Xcode.
SystemLanguageModel.default.availabilityandLanguageModelSession(instructions:)are on firmer ground.Also worth a look before merge:
genai-promptis at beta2, and release builds minify — if R8 strips the ML Kit GenAI classes, keep rules will be needed incomposeApp/proguard-rules.pro.Checklist
./gradlew detektpasses🤖 Generated with Claude Code