Bug: Multiple bug behaviors with LLM profiles

[DevinVinson]

This is a bit of a long process to get it to happen but I've been trying to figure out the exact steps. The end result is eventually the API key value gets deleted or corrupted for an LLM profile which causes conversations to start erroring out.

I'm being specific with the models because these are the ones I've definitely seen it happen with. I have a ~5 minute video as well but these are the steps.

1. Add claude-opus-4.5 via OpenHands, check that it works and is your only available profile
2. Add gpt-5.4 via OpenHands
3. See that you have two available profiles. Use claude-opus-4.5 to test conversations work
4. Start a new chat. Switch to gpt-5.4. See when chat starts the profile is back to claude-opus-4.5 and is not using the profile you selected.
5. Go to LLM Profiles and see that gpt-5.4 is now set as Active.
6. Start a new conversation. See that gpt-5.4 is set as the profile. Send a message and see that claude-opus-4.5 is shown as the profile used.
7. Go to LLM Profiles and Delete claude-opus-4.5 profile.
8. Start a new conversation. See that gpt-5.4 is set as the profile. Send a message and see that the message uses the gpt-5.4 profile successfully and shows it in the selected profile in chat.
9. Go to LLM Profiles and add a new profile. Select Claude opus 4.6 via OpenHands. See that the API key is pre-filled with a green checkmark and .
10. Set the new claude-opus-4-6 profile as Active.
11. Start a conversation, see claude-opus-4-6 profile selected. Start the conversation and see the server activity reflect a missing LLM key and no error in the chat session.
12. Go to LLM Profiles. Open the claude-opus-4-6 profile and see there is no API key set.

I think this is 3 specific bugs:

1. The profile shown in the new chat home view isn't always the profile used to start the conversation
2. When adding a new LLM profile the API is sometimes shown with a green checkmark and which doesn't seem to actually save an API key
3. Starting a conversation with an incorrectly configured LLM profile doesn't seem to show an error the user can act on

[VascoSch92]

Bug 1 (dominant): selected profile reverts when starting a conversation

Root cause: a settings-cache invalidation asymmetry across the three profile mutations.

SettingsService keeps an in-memory cache (5-min TTL) that conversation-start reads via getSettingsForConversation() (settings-service.api.ts:269-275, :97-99), cleared only by saveSettings/invalidateCache (:507, :516).

Hook	Used by	Clears settings cache?
useActivateLlmProfile	Profiles page "Set as Active", onboarding	✅ use-activate-llm-profile.ts:17
useDeleteLlmProfile	"Delete profile"	✅ use-delete-llm-profile.ts:18
useSwitchLlmProfile	chat-header / home-page switcher	❌ use-switch-llm-profile.ts:28-37

The home-page switch (conversationId === null) calls activateProfile(), the same endpoint useActivateLlmProfile uses (agent-server-conversation-service.api.ts:610-613), and invalidates the profiles list but not the settings cache. So the server's active profile flips while the stale encrypted LLM config of the previous profile is still what gets sent on conversation start. This matches every step of the repro: the Profiles page correctly shows the new profile (list is invalidated) while chat reverts to the old model; deleting the old profile fixes it (delete does call invalidateCache()); and the 5-min TTL makes it self-heal, hence "hard to reproduce".

Fix, mirror the sibling hooks in useSwitchLlmProfile.onSuccess, on the home-page path only (the per-conversation /switch_llm swap is unaffected):

onSuccess: (_data, { conversationId }) => {
  queryClient.invalidateQueries({ queryKey: LLM_PROFILES_QUERY_KEYS.all });
  if (conversationId) {
    queryClient.invalidateQueries({
      queryKey: ["user", "conversation", conversationId],
    });
  } else {
    SettingsService.invalidateCache();
    queryClient.invalidateQueries({ queryKey: SETTINGS_QUERY_KEYS.personal() });
  }
},

Bug 2: new profile shows a green "key set" check but saves no key

Root cause, the embedded form's API-key placeholder and KeyStatusIcon reflect the global settings?.llm_api_key_set (llm-settings.tsx:172, :176-177), not the profile being created. Create mode seeds the field empty (llm-settings-local-view.tsx:315), and the key-reuse-on-save fallback is edit-mode only (:189-196). So when a global key already exists, a new profile shows a green "set" field that's actually blank, then saves with api_key omitted.

Fix , drive the indicator from the form's own create-vs-edit state (empty in create, encrypted initial value in edit) instead of the global flag; optionally block saving a keyless profile.

Bug 3: no actionable error when starting with a misconfigured profile

A UX/error-surfacing gap, not fully provable from the frontend alone (depends on what the agent server emits). A frontend preflight that warns/blocks starting a conversation with a profile whose api_key_set is false would address it regardless.

Priority

Bug 1 first (dominant, smallest and lowest-risk change), then Bug 2, then Bug 3.

[hieptl]

Hello @DevinVinson, @VascoSch92,

I wanted to check if this issue is still open, or if Vasco’s pull request has addressed all the related points.

Thank you very much! 🙏

[VascoSch92]

Hi @hieptl

I still have to address bug No.3

I will do it today and ask you for a review.

[VascoSch92]

Hey @hieptl

here is the PR: #711

Then I think we can close the issue.