chore(upstream): update repository - #20
Merged
Merged
Conversation
… TTFT Mirror the sagemaker_chat fix on the native sagemaker/ streaming path: the sync and async completion handlers read the invocations-response-stream body with iter_bytes(chunk_size=1024) / aiter_bytes(chunk_size=1024), so httpx withholds bytes until 1024 accumulate and tokens arrive in gap-then-burst waves. Drop the fixed chunk size so each decoded event is forwarded as its bytes arrive. Also add a boundary-agnostic decoder test proving frames reassemble correctly regardless of where transport reads split the stream. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…_sync_call Extract the inline sync streaming post/decode into make_sync_call so it can be exercised with an injected client, mirroring make_async_call, and add a sync regression test that each token is forwarded after exactly one pulled frame. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): give each Models + Endpoints tab its own path * refactor(ui): decompose Models + Endpoints into per-tab pages with URL-driven detail Dissolve the 488-line ModelsAndEndpointsView monolith into one page per tab under the models-and-endpoints route, with a persistent layout owning the header, cost banner, tab bar and refresh. Each tab page owns only its own state; shared lists come from a small useModelDashboardData hook. Replace the stateful model/team drill-in (setSelectedModelId/setSelectedTeamId full-page takeover) with real URL navigation: ?model=<id> and ?team=<id> render ModelInfoView/TeamInfoView from the layout, so a model or team detail view is now shareable, bookmarkable and back-button friendly. Removes the empty placeholder pages from the first commit. Swap the tab bar off phased-out tremor onto antd Tabs. * fix(ui): render model tab panels standalone instead of Tremor TabPanel AllModelsTab, ModelRetrySettingsTab and PriceDataManagementTab rooted their render in a Tremor <TabPanel>, which only renders inside a Tremor <TabGroup>. After the decomposition these panels live under antd Tabs / as route pages with no such ancestor, so All Models (and the other two) rendered blank. Root them in a plain container instead. The existing component tests mocked @tremor/react (stubbing TabPanel to render children), which hid this; add a regression test that renders with real Tremor and asserts the content is visible standalone. Also type visibleSlugs/TAB_LABELS with the canonical ModelTabSlug so a tab added without a matching label is a compile error. * fix(ui): make model/team drill-in navigation work under the /ui static mount The drill-in close (Back to Models) and open were no-ops: the dashboard is a static export served under /ui, a prefix the Next router (basePath "") does not know, so a router.push to the current pathname with only the query changed is deduped and never re-renders. Drive the ?model=/?team= overlay via real browser navigation (window.location) so open and close reliably work; verified live. Also address review feedback: gate the tab-permission redirect on teams/uiSettings having loaded so a team admin hard-loading /add is not bounced to the base before their membership resolves, and memoize getProviderFromModel on modelCostMapData so the health tab's provider labels refresh when the cost map loads. * fix(ui): use window.location.replace for the tab-permission redirect router.replace is unreliable under the /ui static mount (same class of Next-router issue that broke the drill-in back button), so the forbidden-tab redirect could fail to fire. Use window.location.replace, which keeps the no-history semantics of a permission redirect and is deterministic. Redirect stays gated on teams/uiSettings having loaded. * fix(ui): drive model/team drill-in with history.pushState for client-side nav Switch the ?model=/?team= overlay navigation from window.location.assign to window.history.pushState, which Next's App Router observes. This keeps navigation client-side (no full page reload, React Query cache preserved) while still working for the same-path query-only change that router.push cannot do under the /ui static mount. Open, close (Back to Models) and browser Back are all verified in the built UI. Adds unit coverage for the open/close/read behavior.
…id} (BerriAI#32350) * fix(organization): persist cleared fields on /organization/update Clearing an org field (the Metadata box or a TPM/RPM/max_budget limit) via PATCH /organization/update looked like it saved but reverted on refresh; the partial-update merge could not tell a cleared field from an untouched one and dropped every clear The endpoint now decides SET vs CLEAR vs UNTOUCHED purely from which keys the raw request body carried, via a pure build_organization_update_plan. Budget nulls flow to update_budget (null clears via exclude_unset), metadata is replace-when-sent (written as {} for the non-nullable Json column), and a budget write on an org with no budget_id creates and links a budget row. This removes the exclude_none dump, both "if v is not None" filters, and the additive _update_dictionary merge Resolves LIT-3664 * feat(organization): add RESTful PATCH /v2/organization/{organization_id} Adds a v2 organization-update endpoint with a deterministic partial-update contract, and reverts the v1 /organization/update changes so its public behavior stays untouched On v2 a field present in the request body is written (null/[]/{} clears, a value sets) and an omitted field is left untouched; presence is read from model_fields_set. Clearing a TPM/RPM/max_budget limit or the metadata now persists instead of being dropped as if it were never sent. Metadata is replace-when-sent and written as {} when cleared, since the org metadata Json column is non-nullable. Budget nulls flow to update_budget, and an org with no budget row gets one created and linked. The endpoint is hidden from the public Swagger docs via include_in_schema=False, and stays typed in the generated dashboard schema Resolves LIT-3664 * test(organization): cover v2 auth guard, negative budget, and object_permission Adds v2 endpoint tests that were missing: the real _verify_org_access path rejects a non-admin caller with 403 and writes nothing, a negative max_budget is rejected with 400 before any DB access, and a sent object_permission is passed to the upsert helper with its id linked onto the org write Refs LIT-3664 * fix(organization): 400 on null-clear of required org fields; drop dead budget upsert organization_alias and models are non-nullable columns, so a v2 request clearing them with null hit a 500 (NOT NULL violation) and could partially apply the budget half of the request first; the endpoint now returns a 400 with a clear message. Also removes the unreachable "create a budget when the org has none" branch from _apply_organization_budget_updates, since budget_id is a non-nullable FK and every org already has one, so the endpoint no longer needs to link a newly-created budget id Refs LIT-3664 * fix(organization): let v2 clear object permissions when sent as null Sending object_permission: null now detaches the org's permission by setting the nullable object_permission_id to null, instead of being a silent no-op, so the endpoint honors its documented "null clears" contract and an admin can actually revoke vector-store/MCP access. Sending a value still merges as before Refs LIT-3664 * fix(organization): make v2 PATCH atomic, strict, and 422-consistent Tighten the PATCH /v2/organization/{id} endpoint against standard HTTP PATCH (RFC 5789 / RFC 7396 JSON Merge Patch) semantics: - Apply the budget-row and org-row writes in one prisma transaction so a failure between them can no longer half-apply the patch (RFC 5789 requires a PATCH to apply atomically). The budget write is inlined as a tx-aware call mirroring the team-member budget path rather than the standalone update_budget route handler - Set extra="forbid" on OrganizationUpdateRequestV2 so an unknown or misspelled key is a 422 instead of a silently dropped no-op; the contract is presence-driven, so swallowing unknown keys is unsafe - Return 422 (not 400) for the hand-rolled field validations (negative budgets, null-clear of required organization_alias/models, invalid model_max_budget) so every validation failure matches the 422 that pydantic already returns for bad values - Document the per-field clear tokens accurately: null clears budget limits and metadata, [] clears models, and organization_alias cannot be cleared Tests cover the single-transaction write path, unknown-field rejection, the 422 status changes, and the budget_reset_at recompute. * fix(organization): reject empty object_permission on v2 PATCH instead of silently keeping grants object_permission is a nested merge field on PATCH /v2/organization/{id}: a sent object merges into the existing permission row (updating one grant list without touching the others), and null detaches it. An empty {} therefore merged nothing and left every existing vector-store/MCP grant in place, so an admin who sent {"object_permission": {}} to strip access silently kept it. Reject a present-but-empty object_permission with a 422 that points the caller at null, mirroring how the endpoint already rejects a null clear of the required organization_alias/models. This keeps merge semantics for non-empty payloads and does not affect the Admin UI, which only ever sends a fully populated object or omits the field. * fix(organization): JSON-serialize model_max_budget on the v2 budget write model_max_budget is a Json column on the budget table. Route the budget-row write through jsonify_object so a dict value is serialized the same way new_budget and the org-row metadata write already do it, keeping every Json column on this endpoint written consistently. Raw dicts already round-trip (update_budget writes them unserialized), so this is not a correctness fix so much as making the one Json column on the budget path follow the same serialization as the rest of the file. Added a test that a patched model_max_budget reaches the budget write JSON-serialized. * refactor(organization): trim v2 docstrings and consolidate planner tests Trim the verbose docstrings on the v2 endpoint, request model, and the two pure helpers to the essential contract, and drop a stale line that still referenced update_budget's exclude_unset (the budget write is inlined now). Collapse the nine per-case planner tests into one parametrized test asserting exact budget/org split per body, and fold the two model-validation rejection cases into one parametrized test. Same 36 test cases run; the planner assertions get stronger (exact-equality instead of presence/absence) and the test additions shrink by ~85 lines. * refactor(organization): inline the v2 update planner into the endpoint Fold the OrganizationUpdatePlan dataclass and build_organization_update_plan helper into update_organization_v2. The budget-vs-org split is a few dict comprehensions built in one shot, so the extra type plus builder was more ceremony than the job needed. Drops the now-unused dataclass/AbstractSet imports and the isolated planner unit tests; the split is exercised end-to-end by the endpoint tests. * fix(organization): run v2 object permission upsert inside the update transaction prepare_object_permission_upsert splits the shared helper's read-and-merge step from its write so the v2 endpoint can upsert the permission row on the same prisma transaction as the budget and org writes. Previously the upsert ran before the transaction, so a rolled-back org write left merged grants live on the permission row the org still pointed at. The upsert record now pins object_permission_id, since the column's @default(uuid()) would otherwise mint a fresh-create id different from the one linked on the org. v1 and the team/key callers of handle_update_object_permission_common keep their existing behavior * fix(lint): keep the v2 org PR within the strict-rule budget The strict gate flagged the PR's new code after the base merge: 11 UP045 Optional fields and a typing.List on OrganizationUpdateRequestV2, Dict annotations in the new upsert helper and the TypeAdapter, and a B008 from the v2 endpoint's Depends default. The model and helper now use pipe unions and builtin generics, and the endpoint takes its auth dependency via Annotated, which avoids the call-in-default pattern B008 targets * fix(routes): expose /v2/organization on the backend component allowlist The component-split coverage test requires every app route on a component; the new v2 org PATCH belongs with the other management endpoints on the backend, alongside the existing /v2/key and /v2/team prefixes * fix(organization): clear budget_reset_at when budget_duration is cleared via v2 PATCH
* test(ui): pin search-tools info view behavior before shadcn migration Rewrite the markup-coupled copy-button assertions in SearchToolView to role queries plus lucide icon-state, and add a role/text characterization suite for SearchConnectionTest, which had none. Both are green against the current antd/Tremor components so they can act as an unedited regression net across the migration. * refactor(ui): migrate search-tools info view to shadcn Port the search-tools detail view and its two helpers off antd and Tremor onto the installed shadcn primitives plus token utilities: - SearchToolView (the info page reached by clicking a tool) now uses ui/button, ui/card and a plain CSS-grid header instead of Tremor Card/Grid/Title/Text and antd Button - SearchToolTester swaps antd Input/Button/Spin and Tremor Card/Title for ui/input, ui/button and UiLoadingSpinner, with no inline styles - SearchConnectionTest swaps antd Button/Divider/Typography and the inline keyframe spinner for ui/button, ui/separator and UiLoadingSpinner Markup only; no behavior change. The list page (SearchTools) and its create and edit forms stay on antd because they are Form-bearing and blocked until the forms migration. Icons move from antd and heroicons to lucide. The retired antd no-restricted-imports suppressions are pruned from the baseline. * fix(ui): drop redundant vertical padding in SearchToolTester card The shadcn Card already applies py-6 and gap-6 to its flex children, so the pt-6/pb-6/mb-6 added during the migration stacked on top of it and roughly doubled the vertical whitespace. Keep only px-6 (Card has no horizontal padding) and let the Card own the vertical rhythm, which restores the original 24px spacing. * style(ui): format SearchConnectionTest test file with prettier
…rating Adds role/text-based coverage for MemoryDetailDrawer (which had none) and extends MemoryView's test past the mocked table to the header, the create modal trigger and the detail drawer round trip. Both are green against the current antd components, so they act as an unedited regression net for the shadcn migration that follows.
Replaces the antd Drawer with ui/sheet and the antd Button, Typography and Space usage with ui/button plus token utilities, and swaps the @ant-design PlusOutlined icon for lucide's Plus. Toasts now go through the shared MessageManager so the route no longer imports antd directly. The route's tests were written against the antd components in the previous commit and are unchanged here, so they pass on both implementations. MemoryEditModal is left alone because it is built on antd Form; the table already sits on the shared DataTable.
…itellm_/migrate-page-memory-9b2c09 # Conflicts: # ui/litellm-dashboard/eslint-suppressions.json
…s run offline for any uid (BerriAI#34325) * fix(docker): bake non_root prisma engines at /opt/prisma so migrations run offline for any uid The non_root image baked the prisma CLI and engines under /app/.cache and used the CLI's default (library) engine mode. Prisma stopped baking the library engine, so `prisma migrate deploy` fell back to downloading it at startup, which needs network egress and a writable cache. Under an arbitrary non-root uid (OpenShift restricted-v2), an air-gapped network, or a readOnlyRootFilesystem, that download fails and the proxy starts on an empty schema while every DB endpoint returns 500. The migration entrypoint exits 0 on that failure, so a default-uid `docker run` with network never surfaced it Bake to /opt/prisma, a fixed world-readable path no cache mount shadows, and pin PRISMA_CLI_PATH plus PRISMA_CLI_QUERY_ENGINE_TYPE=binary so the baked binary engine is used directly, matching Dockerfile and Dockerfile.database. A build-time guard asserts the binary query engine is present, so a future prisma change that stops baking it fails the image build instead of silently degrading migrations Adds docker/test_offline_migration.sh, run from image-scan, which migrates a fresh Postgres with no egress as a non-root uid and asserts the schema was created, the case a default-uid `docker run` with network cannot catch * test(docker): move the offline migration check into a gated pytest and stop pinning XDG_CACHE_HOME at the read-only bake The offline migration check lived in docker/ as a shell script. It now lives in tests/proxy_migration_tests/ as a pytest gated on LITELLM_IMAGE, matching the sibling schema-migration test gated on DATABASE_URL, and image-scan invokes it with pytest instead of bash. It also asserts the migration entrypoint's exit code alongside the table count, so a crash or a container-startup failure fails loudly rather than only surfacing as a low table count Runtime XDG_CACHE_HOME pointed at /opt/prisma/.cache, which is baked a+rX with no write, so any XDG-aware library writing a cache at runtime would be denied for every uid. Leave it unset so it falls back to $HOME/.cache (/app/.cache, created here and owned by the runtime uid), matching Dockerfile and Dockerfile.database which never pin XDG at runtime. A second test guards against a future edit pointing a cache or home var back at the read-only bake
…er both ingress headers
…t_format_remaining_keywords fix(anthropic): strip all remaining output_format schema keywords rejected by Anthropic
…creens The migrated sheet asked for a flat 720px width while its only max-width came from the primitive's sm:-scoped rule, so below the sm breakpoint no cap applied at all: on a 375px viewport the sheet rendered 720px wide with its left edge at -305px, and because it is position:fixed there was no scroll to reach the hidden content. The primitive's own w-3/4 default did not have this problem; the fixed pixel width is what removed the guard. Caps the width to the viewport at every breakpoint and only asks for 720px from sm up. Verified in a browser at 375px, 700px and 1280px: the sheet is now 375, 700 and 720 wide respectively, always at left 0.
…nt (LIT-3637) Admits a keyless SSO user (no virtual key) at the aggregate /mcp endpoint from a gateway DCR session bearer, resolving team/org/SCIM/budget authorization fresh on every call. - Aggregate DCR front door: stateless /register (sealed llm_dcrc_ client ids), SSO-backed /authorize + /authorize/complete, and /token minting identity-only session tokens with PKCE, single-use codes/flows, and rotating refresh tokens. - Admission: a session-shaped Authorization at the aggregate scope opens via _admit_gateway_session, reloads the live user, and runs the centralized policy gate; failures return the RFC 9728 invalid_token challenge. Gated on the un-forgeable, server-only mcp_admitted_user_subject marker, so virtual-key and JWT auth are unchanged. - Authorization model: an admitted subject is resolved as one plain UserAPIKeyAuth per grant source (its own grants, plus each team it is a live roster member of), each answered by the SAME resolver virtual keys use, then unioned. That branch is the FIRST statement of BOTH public resolvers, so no single-credential prelude runs for it and a fault in a lookup it never uses cannot deny its grants. A source team counts only while it is a live grantor: roster membership, not blocked, and neither the team nor its owning org over budget (enforced through the SAME _team_max_budget_check / _organization_max_budget_check owners common_checks uses for keys). Each team source carries that team's own org, so the existing org ceiling caps it; for a keyless source the org list only ever intersects (a ceiling must not become a grant) and an unresolvable ceiling denies rather than silently uncapping, on both the server and tool axes. _roster_team_object is the single owner of "which teams count": a team whose roster no longer lists the user neither grants servers nor throttles, in one place. - Rate limits: the subject is bounded by its user rpm/tpm AND by the per-server mcp_rpm_limit of the team a call is ATTRIBUTED to — the same single source billing charges, from the same owner. A key charges its one pinned team's bucket; a keyless subject has no team_id, so admission stamps each granting team's limit map onto the auth (server-only field, stripped from validated input like the marker) and the limiter emits that team's mcp_per_team descriptor. Charging every granting team instead would let one cross-team user drain several teams' SHARED buckets on a single call and block their other members; and a server the user's OWN grant reaches charges no team bucket at all, because no team provided it. Per-KEY MCP limits do not apply because there is no key. - Wrapper channels: the manager-level union treats the admitted subject by the same grant model. The admin-role short-circuit and the absolute no_mcp_servers early-return are key-credential rules and never apply to it (a session bearer is a third-party client credential, not the dashboard, and the subject's opt-out silences only its own source). Operator-open channels (allow_all_keys, the user's own BYOM submissions) are owned by one operator_open_server_ids helper that BOTH the server union and the admitted tool resolution consult (suppress-BYOM-when- explicitly-scoped is a key-credential rule and never applies to the subject, whose user row carries the DB-default empty mcp_servers), so an open-channel server is default-open for tools instead of listable but uninvokable. - Redirect URIs: one owner, validate_redirect_uri_shape, decides redirect-URI hygiene (bad scheme, fragment, missing host, userinfo, backslash host) and resolves allowlisted native callbacks, shared by DCR registration and the OAuth endpoints. Registration keeps a deliberately wider trust policy than validate_trusted_redirect_uri: public dynamic registration accepts any https client, and its controls are mandatory S256 PKCE plus the consent screen. - Egress leak-defense: a gateway admission credential (session bearer / bridge envelope) is scrubbed from EVERY egress header context, anchored to the credential shape, so it can never be forwarded upstream and replayed. - Single-use guard: auth-code, refresh and connect-flow claims resolve the proxy's cross-worker redis cache themselves rather than trusting the cache passed in, and fail CLOSED on a Redis fault instead of falling back to a per-worker count that a captured id could replay through another worker. - Sign-in return_to: one shared, never-raising helper persists a safe return_to for every sign-in branch (SSO/Okta/generic and username/password), and every branch RESUMES through the same _sso_return_to_redirect the SSO callback uses, so however a deployment signs in the stored value is honored identically (same-origin path directly; control_plane_url via the one-time login-code handoff). A stale cookie is ignored rather than failing a completed sign-in. - Budgets, both halves: ENFORCEMENT (an already over-budget team or its owning org stops being a grantor, in the source gate) and ACCOUNTING (a team-derived tool call is billed to the granting team and ITS org, so that budget accumulates and the right organization is charged). A server the user's own grant reaches bills the user; when several teams grant one server the pick is the lowest team_id, stable and auditable. Billing rides a COPY, so authorization still sees the full union, and it is inert when the target server cannot be resolved from the tool name. Deferred (tracked): client-selected server scoping of the session token (LIT-4680). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Compress the security rationale in the gateway-session admission path of user_api_key_auth_mcp.py, keeping the load-bearing "why" and dropping the restatement, and remove a garbled dead comment in get_allowed_tools_for_server In the tests, hoist the duplicated _team / _admitted_subject fixtures to module-level factories and parametrize the four fail-closed session-bearer variants into one case. No behavior change; the 294 tests in the file still pass
…ker_chat_streaming_ttft fix(sagemaker): forward stream events as they arrive to cut TTFT
…_admission feat(mcp): admit gateway DCR session bearers at the aggregate /mcp scope
…iscovery_logging fix(mcp): log actionable OAuth discovery failures for misconfigured server urls
isAdminRole compares against a list that mixes raw and formatted role strings: it holds raw org_admin but not the "Org Admin" that formatUserRole produces, and AuthContext stores the formatted form. A keyless org admin therefore read as a non-admin and was redirected to the connect page. Gate positively on internalUserRoles instead, which carries both representations, so the redirect targets the persona it is meant for and any role that is not unambiguously an internal user is left on the dashboard. The shared admin list is left alone: completing it would change org-admin access across every isAdminRole caller, which is a roles-policy decision of its own.
* test(ui): characterise the workflow runs detail drawer before migrating it Pins the drawer's behaviour against the current antd implementation: the metadata fields it surfaces, the timeline ordered by sequence number, the empty-events copy, the messages section staying collapsed until opened, the in-drawer refresh refetching, and the close control dismissing it. Every assertion is role/text based so the same file can stay green once the component moves off antd, without being edited. * refactor(ui): migrate workflow runs to shadcn Replaces the antd Drawer, Collapse, Button, Spin, Tooltip and Empty on the Workflow Runs page with the installed Base UI primitives (Sheet, Collapsible, Button, UiLoadingSpinner, Tooltip) and lucide icons, and moves the page's hardcoded hex colours, fonts and geometry onto design tokens and utility classes so the page can be themed. The only inline styles left are the gantt bars' computed left/width, which are runtime values. Behaviour is unchanged: the drawer's characterisation tests were written against the antd version in the previous commit and pass here without being edited. Retires the file's now-unused antd no-restricted-imports suppression.
…Table (BerriAI#34363) * refactor(ui): migrate models and endpoints table onto the shared DataTable Rebuilds the All Models table on the shared DataTable, following the 2a treatment from the Models + Endpoints design: one card holding search, the Team and View selectors, refresh, columns and filters, with the active filters on a chip row and the pagination footer at the bottom. Retires the last hand-rolled tremor renderer (all_models_table.tsx) and the antd/tremor column defs in molecules/models/columns.tsx, replacing them with a thin AllModelsTable consumer plus AllModelsTableColumns built from the shared cell library. Behavior is preserved end to end. The server sort field mapping now lives next to the column ids so the two cannot drift. Status keeps its column and its sort, hidden by default behind the Columns menu because the design shows nine columns. Access groups collapse into a "+N more" tooltip instead of a per-row expand toggle, and the full reset moves into the filter drawer footer where the design puts it. Adds the shadcn hover-card primitive (Base UI PreviewCard in the base-vega style) for the model information hover, which needs an interactive surface a tooltip cannot provide. * fix(ui): stop the models tab re-querying on mount The mount-time effect fires the debounced search with the initial empty value, and its callback rebuilt the pagination object unconditionally. That produced a second render (and a second query) roughly 300ms after mount with no user input, which on a slow CI machine swapped the table's row nodes mid-interaction and made a click land on a detached node. resetToFirstPage now returns the existing state when already on the first page, so React bails out instead of re-rendering. Pinned with a test that asserts no additional query after the debounce settles; it fails without the fix.
…tion_linear_oauth test(e2e): drive a real Linear OAuth MCP through chat completions under both ingress headers
…erriAI#34343) * refactor(ui): migrate request logs table onto the shared DataTable Moves the Request Logs tab off the local view_logs/table.tsx clone and onto the shared DataTable in server sort, pagination, and filter mode. The container is split into RequestLogsPanel (data owner: the spend-logs query, the session dedup and composition pipeline, and the detail drawer), a thin RequestLogsTable, and RequestLogsTableColumns. The clone itself stays for now because TopModelView and TopKeyView still consume it The advanced filter bar moves into the shared DataTableFilterDrawer, so filters commit on Apply and render as removable chips. That makes the per-keystroke debounce in the query hook redundant, and the hook now takes ColumnFiltersState, PaginationState, and SortingState directly instead of carrying its own filter shape. Reset still restores the default 24 hour window alongside the filters Adds shared/PaginatedSearchSelect, a Base UI combobox with server-side search and infinite scroll, and uses it for the Key Alias and Model filters. That retires the three logs-only antd pickers (PaginatedKeyAliasSelect, PaginatedModelSelect, FilterTeamDropdown) and the FilterComponent molecule they plugged into. The shared TeamDropdown is deliberately untouched: six other surfaces still render it, five of them as a bare child of an antd Form.Item that injects value/onChange implicitly * test(ui): pin team-scoped key alias filtering in the logs filter drawer The Key Alias filter narrows its options to the team selected in the same drawer, a cross-filter dependency carried over from the antd picker it replaced. Nothing covered it: the live QA pass explicitly did not exercise it either, so it was the one behaviour in this migration that could regress silently Asserts the selected team id reaches useInfiniteKeyAliases, that the lookup stays unscoped when no team is picked, and that the scope does not leak into the Model lookup, which shares the same combobox but takes no team
…on v2 resolver The v2 credential resolver owns oauth2_token_exchange end to end: any server with a token-exchange config maps to a non-None TokenExchangeConfig spec, and that config is in _create_mcp_client's override-exclusion set, so a caller x-mcp-* override cannot force it back to v1 either. The v1 handler resolve_mcp_auth reached at spec is None was therefore dead for OBO, including its warn-then-proceed-unauthenticated fall-through. Delete auth/token_exchange.py and the exchange branch, dropping the subject_token parameter that only fed it. Separately, the REST listing and call paths still ran the v1 per-user OAuth lookup for servers the v2 resolver owns. Unlike the two protocol-path call sites they gated on auth_type == oauth2 only, with no to_server_spec check, so a migrated authorization_code server did a DB round-trip whose Authorization header _resolve_v2_auth then discards. Add the same guard via _is_v1_resolved_oauth2_server, shared by the per-server lookup and the prefetch preflight. Also collapses MCPOAuth2TokenCache.async_get_token's now single-caller require_client_credentials_flow kwarg and removes the dead _get_bulk_user_oauth_headers helper (zero callers).
… a dirty-field PATCH (BerriAI#34324) * feat(ui): rebuild Organization Settings on react-hook-form + zod with a dirty-field PATCH Replaces the antd Settings form in organization_view.tsx with OrgSettingsForm, the first consumer of the shared RHF + zod form kit. The form derives a minimal payload from RHF dirty tracking via pickDirty and sends it to the typed PATCH /v2/organization/{organization_id}, so untouched fields are omitted, emptied widgets clear with null ([] for lists), and the old full-send builder with its length > 0 clear-dropping guards is deleted. Adds src/lib/forms/useZodForm.ts so every form gets the z.input/z.output generics and zodResolver wiring from one place, and forwardRefs ui/textarea so RHF can register it under React 18 * fix(ui): forwardRef InputGroupTextarea to match the forwardRef'd Textarea * test(ui): pin that an mcp server edit preserves existing org toolsets * docs(ui): explain the useZodForm generics * chore(ui): re-prune eslint suppressions after rebase onto staging
Adds GET /v1/tool/spend returning per-tool and daily tool spend with a deduplicated request total, and a cache leakage breakdown on the Prompt Caching tab of the Cost Optimization page. Tool-spend rows are validated at the boundary with pydantic, the endpoint is scoped to proxy admins, date params are cast to timestamptz for real-Postgres query_raw, and the leakage math treats litellm-normalized prompt_tokens as cache-inclusive (uncached = max(0, prompt - cache_read - cache_creation)).
… redirect AuthContext sets token and clears authLoading in one effect, then a second token-keyed effect populates userRole, so there is a render where the user is signed in but userRole is still the initial empty string. The positive internalUserRoles check reads that interim role as non-internal, which let the api-keys dashboard paint for a frame before the role arrived and the keyless redirect ran. Treat "signed in on the post-login landing with an unhydrated role" as a resolving state that holds the loading screen, so the dashboard never flashes. Every login=success token carries a required user_role claim, so the role always hydrates within a tick and this cannot hang; it is scoped to the landing, so ordinary dashboard visits are unaffected.
…BerriAI#34413) require_env hard-failed a test (and, for the shared litellm-ops secret, drove piling every provider credential into one blob) whenever an optional cred was absent. Most call sites either read a value the test actually uses or just gated on the runner's env for a key the gateway consumes. Read os.environ directly where the test uses the value; drop the presence-only gates so those cases run against the proxy instead of pre-failing on the runner's environment. Removes the require_env helper from e2e_config.
…lect fix(install): pass an explicit Python version request to uv tool install
…point-standards-b1cd57 refactor(management): move the logs end-user filter onto /management/v1
…d writer Resolves LIT-4823. An adversarial review reproduced against real Postgres that a batched increment upsert stalling past the prisma engine timeout leaves its transaction open on the pooled connection; the retry draws the same connection, its statements stack into the still-open transaction, and one commit applies both increment sets while the writer reports success. httpx.ReadTimeout is exactly that post-send case and every spend writer retried it. DB_RETRY_SAFE_ERROR_TYPES (ConnectError only, the failure that proves the statements never reached the database) is now the single owner of what a non-idempotent writer may retry. All seven entity and daily spend writer retry arms and the tool usage flush consume it. DB_CONNECTION_ERROR_TYPES is unchanged for the idempotent spend-log writer, whose create_many with skip_duplicates may safely retry the full tuple. The corruption was reproduced on update_daily_user_spend (seeded 10|100|1, expected 11|110|2, observed 12|120|3); the new policy tests pin that a ReadTimeout drops the batch loudly on the first attempt and a ConnectError still retries.
…t block a built-in logger (BerriAI#34804)
…usage (BerriAI#34803) litellm_provider_cache_creation_input_tokens_metric only read the Anthropic-style top-level usage.cache_creation_input_tokens and had no prompt_tokens_details fallback, unlike its cache-read twin. OpenAI models that bill prompt cache writes report them only in prompt_tokens_details.cache_write_tokens, so the counter never fired for them. Resolve provider cache read/write tokens through a shared helper that falls back to prompt_tokens_details.cache_write_tokens (canonical) then cache_creation_tokens when the explicit top-level field is absent, and give litellm_input_cache_creation_tokens_metric the same fallback for raw usage dicts that only carry cache_write_tokens
The backfill compares the naive start_time column against a timestamptz cutover, and that coercion follows the session time zone, so a non-UTC session shifts the cutover boundary by the offset. Pinning the session makes the whole script timezone-independent. The date bucketing itself was already safe: to_char on a timestamp without time zone ignores the session time zone and the stored values are UTC
basedpyright's inference load now exceeds node's ~4GB default heap cap on ubuntu-latest once the Any hotspots carry real types; the node process died with a JS heap OOM, emitted nothing, and the gate refused the vacuous run. 12GB leaves headroom on the 16GB runner.
Exercises the streaming field-fill heuristics, model_construct fallbacks, and the get/cancel/delete/list request and response transforms that had no tests.
…able chore(typing): clear 2.7k basedpyright Any errors across 15 hotspot files
…drail and passthrough suites (BerriAI#34833) * test(e2e): wait for MCP tool discovery instead of racing it /v1/mcp/server returns as soon as the DB row is written, but the gateway runs the initialize + tools/list handshake against the upstream lazily, on the first request that needs it. Every MCP test read tools/list immediately after registering, so it raced that handshake. The gateway reports a server it has not discovered yet exactly like a dead one: it catches the per-server handshake exception and returns an empty tool list. The tests asserted on a single read, so the race surfaced as "granted key never saw search_datadog_logs; tools=frozenset()" while a sibling test against the same upstream in the same run passed. Add McpClient.await_tool, which polls tools/list to the suite's existing poll_timeout and returns the qualified tool name, and route the four discovery sites through it. An unreachable upstream or an unapplied grant still fails, and the failure now names the last tools/list result. Refs LIT-4821 * test(e2e): wait for a2a agents to reach the data plane after registration POST /v1/agents is a control-plane write; the /a2a/{agent_id} routes that serve the card and run message/send are data plane and only see the agent after the next DB reload. Every test registered an agent and immediately read its card or sent it a message, so the first data-plane touch could 404 on the agent it had just created. register_agent now waits for the card to become servable before returning, the same way ProxyClient.create_model waits for a new model, so callers do not each have to poll. Registration failures skip the wait, leaving the two rejection tests unchanged. A genuine propagation failure now fails naming the agent id and the last card read rather than as a bare 404 on whichever /a2a call ran first. Refs LIT-4821 * test(e2e): wait for presidio guardrails to sync before asserting masking Registering a guardrail is a control-plane write; the data-plane worker that serves /chat/completions only picks it up on its next periodic DB sync (~30s), so the first call after the create ran against a worker with no guardrail and passed the raw email straight through. The tests asserted on that first call, so they read in-flight propagation as a PII leak. Confirmed directly against a live proxy: the same call is unmasked at t=0s and masked at t=8s, and the presidio analyzer itself correctly returns EMAIL_ADDRESS with score 1.0 the whole time. The MCP guardrail suite already documents and waits out this exact sync delay; presidio never got the same treatment. Poll the call until the placeholder replaces the PII, so the assertions judge the synced state. A guardrail that never masks still fails, on the last unmasked content. pre_call and post_call now pass repeatably. Refs LIT-4821 * test(e2e): drop the presidio logging_only check pending LIT-4841 pre_call and post_call masking both pass once the guardrail-sync wait is in place, but logging_only left the raw email in the OTEL span's gen_ai.input.messages on every attempt across a full poll deadline. Keeping an assertion against known-failing behavior just turns every run red, so the cell is tracked in LIT-4841 instead. The registry row stays, so guardrail.presidio.logging_only.masks now reports as an uncovered gap rather than silently disappearing. Refs LIT-4821, LIT-4841 * test(e2e): wait for guardrail sync in bedrock, moderation and block-code checks All three asserted on the first call after registering a guardrail, so they were served by a data-plane worker that had not synced it yet (~30s DB poll) and read in-flight propagation as a guardrail that failed to block. Verified directly: the openai_moderation guardrail lets a flagged prompt through at t=0s and returns "Violated OpenAI moderation policy" at t=8s. The reasoning-only responses noted in triage (content=None with reasoning_tokens set) were a symptom of the same thing, not the cause; these are pre_call guardrails, so a synced guardrail rejects the request before the model runs. Add poll_until_blocked to guardrails_client for the two that surface a non-success status, and poll on the block marker in the block_code_execution check, which replaces the reply rather than erroring. All eight guardrail tests now pass. Refs LIT-4821 * test(e2e): drop the openai prompt-cache check pending LIT-4841 Prompt caching never engages through the proxy: cached_tokens is 0 on every repeat, while the identical payload sent straight to OpenAI reports 3615 cached tokens on the second call. Pinning prompt_cache_key on the proxy request restores caching (3328 tokens), so something varying per request is defeating OpenAI's automatic prefix cache. That is a product bug with a direct billing cost, tracked in LIT-4841. The registry row stays, so llm.chat_completions.openai.prompt_cache_5m.nonstream.works now reports as an uncovered gap instead of failing every run. Refs LIT-4821, LIT-4841 * test(e2e): drop the responses metadata redis-ttl check It failed on a Redis read timeout against the stage serverless cache (berrie-litellm-stage-ieib2i.serverless.use1.cache.amazonaws.com:6379), a reachability problem this suite has hit before rather than a proxy defect the assertion can pin down. The file held only this test. Its other cell, llm.responses.openai.basic.nonstream.works, is still covered by test_responses_e2e.py; other.config.responses.metadata_redis_ttl_bounded becomes an uncovered registry row, taking headline coverage 314/431 -> 312/431. Refs LIT-4821 * test(e2e): fix passthrough header propagation and openai body, drop the cost check Three separate problems behind the two passthrough failures. The header test 404'd because POST /config/pass_through_endpoint is a control-plane write and the worker serving the route only registers it on its next config reload; measured at ~18s on a live proxy. Wait for the route to stop 404ing before calling it. The readiness probe reuses the master key and omits anthropic-version so polling does not bill a completion per attempt. The openai passthrough body sent max_tokens, which the gpt-5 family rejects outright ("Unsupported parameter: 'max_tokens' is not supported with this model"). Confirmed against OpenAI directly: max_tokens 400s, max_completion_tokens 200s. Passthrough forwards the body untouched by design, so the body was simply wrong. test_openai_passthrough_nonstreaming_logs_cost still finds no SpendLogs row for its call_id after the fix, so it is removed rather than left red; the gemini and anthropic passthrough cost checks still cover that path. Passthrough suite is 8/8 green. Refs LIT-4821
…itellm_fix_responses_bridge_streaming_contract
…is replaced or deleted Auto-router-family deployments live in two structures: the model_list, and a pre-routing strategy registry keyed by (model_name, tags). Removing a deployment dropped it from the model_list without releasing its registry slot, so the re-add that follows hit the "already exists" guard in _register_pre_routing_strategy and ignore_invalid_deployments swallowed it. The deployment came out and never went back, while the DB row and the endpoint response both looked fine. Only a restart healed it, and under multiple replicas each pod diverged into holding a different subset of routers. Removal now releases the (model_name, tags) slot from every strategy registry, in both upsert_deployment and delete_deployment, guarded on the auto_router/ prefix so removing a regular deployment cannot evict a router that merely shares its model_name. Releasing from every registry rather than the first match is what makes this correct for hybrids: registration is one-to-many, since a complexity router configured with adaptive is also registered in adaptive_routers under the same key by the deferred finalize pass. Releasing only the first match left that adaptive strategy live, so a deleted or replaced alias stayed routable through it. Adaptive post-call hooks are rebuilt whenever the adaptive registry changes, not only at the end of set_model_list. The hook set is defined as exactly one hook per registered adaptive router, so a released router stops recording turns instead of holding a hook bound to a strategy nothing points at any more. The swallowed upsert failure is logged at warning instead of debug, which is below the default log level and left this failure with no observable signal anywhere. delete_deployment resolves the outgoing deployment before popping it, and a resolution failure no longer aborts the removal; previously an entry that failed validation would have been left in the model_list permanently. delete_model drops its blanket pop across all four registries. That predates this change and over-evicts: it removes every tag variant registered under the name while only one is being deleted, and nothing reloads on that path to restore the survivors. delete_deployment now handles it correctly and tag-scoped, so the endpoint-level eviction and its helper are removed rather than left to mask it.
…response stream helper
…ty router participates in adaptive routing The finalize re-run in upsert_deployment keyed off the auto_router/adaptive_router prefix only, so editing a complexity router with adaptive enabled released its adaptive_routers entry (and post-call hook) without rebuilding it: complexity routing kept serving while bandit recording, DB persistence and /adaptive_router/state went silently dark until the next full reload. Gate the re-run on a participation predicate that mirrors both arms of the finalize pass, drop the import that pass no longer uses, and pin the registry helpers with direct contract tests
…AI#34815) * fix(ui): validate default team values in Default User Settings The Default User Settings form accepted any free-text team id, and the proxy persisted it without checking the team exists. New users were then silently never added to the default team because the consume-time 404 from team_member_add was swallowed at debug level. Backend: PATCH /update/internal_user_settings now rejects unknown and duplicate team ids with a 400 naming them, before any persistence or team budget side effects. Team-add failures in _add_user_to_team now log at ERROR with user and team ids. UI: DefaultUserSettings rewritten as a shadcn + react-hook-form + zod form following the org-settings pattern. The team id free-text input is replaced with a searchable server-backed team picker, so only existing teams can be selected; zod blocks empty and duplicate rows. The shared deriveErrorMessage helper now unwraps the HTTPException detail.error shape so backend validation errors surface readably in toasts. * fix(ui): restore read-only view with Edit Settings toggle on default user settings Parity with the pre-migration form: the tab renders a read-only summary of the saved defaults, Edit Settings opens the RHF form, Cancel discards pending edits and returns to the summary, and a successful save returns to the summary showing the new values. Model sentinel labels in the summary are derived from ModelSelect's now-exported special values instead of duplicating the strings. * refactor(ui): rename MODEL_SELECT_SPECIAL_VALUES_ARRAY to MODEL_SENTINEL_OPTIONS * fix(ui): move Edit Settings into the card header action slot
…th-api-key-c92cc3 fix(proxy): sanitize per-key callback config out of logged metadata
fix(proxy): roll up tool spend daily instead of scanning SpendLogs
… delete delete_deployment resolved the outgoing deployment through get_deployment before popping it, and ran the strategy release before repairing the index maps. Both halves of that ordering could leave the router inconsistent. A resolution failure meant the entry left the model_list with its registry slots still held, so the alias stayed routable and the name could not be reused; a failure inside the release meant the outer handler returned None with the entry already popped and model_id_to_deployment_index_map never repaired, breaking every later lookup and delete until a restart. upsert_deployment already had this right: it pops, repairs the caches and indices, and only then releases the slot. delete_deployment now follows the same sequence and resolves the deployment from the item it just popped rather than through a lookup that can fail. Releasing the slot is secondary to structural integrity, so it runs last and a failure there is logged instead of abandoning a removal that has already happened.
…ridge_streaming_contract fix(responses_bridge): keep one chat completion id per stream and always stream completed responses
…m_anthropic fix(guardrails): compress content-parts messages in headroom guardrail (Anthropic traffic)
…stry_leak_on_edit fix(router): release the pre-routing strategy slot when a deployment is replaced or deleted
…ntervened (BerriAI#33821) * fix(guardrails): classify all 4xx HTTPException guardrail blocks as intervened * fix(guardrails): narrow HTTPException block classification to 400/403/422 --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…e test (BerriAI#34868) * test(reasoning-effort-grid): bump cell-count assertion for claude-opus-5 The claude-opus-5 grid entry added in ae81625 raised the Anthropic direct route to 31 model combos, but test_grid_cell_count still expected 30, so the suite went red on the tripwire rather than on any behavior change. * test(openai): swap the retired deep-research model out of the bridge test OpenAI shut down o3-deep-research and o4-mini-deep-research on 2026-07-23, so the live call in this test now comes back as a 400 'Model not found'. The test was never about deep research specifically; the bridge fires on any model whose cost-map mode is "responses", so it now uses gpt-5.5-pro, the newest responses-only OpenAI model, and is renamed to say that. gpt-5.5-pro was confirmed present on the CI account with an authenticated GET /v1/models before being picked.
…ding (BerriAI#34870) Both specs assert against UI that has since moved, so they fail on selectors rather than on behavior. The MCP discovery modal became a shadcn/Base UI dialog when mcp-servers migrated off antd, so `.ant-modal` no longer matches it; locate it by its dialog role instead. The create form below it is still an antd Modal and keeps its existing locator. The no-team internal user has no keys, and a keyless non-admin is now sent to /ui/connect on the post-login landing, which has no sidebar. Wait for that redirect to settle, then navigate to the keys page explicitly; the redirect is gated on the ?login=success marker that the fresh navigation drops, so the dashboard sticks and the rest of the test is unchanged.
…two known product bugs (BerriAI#34853) * test(e2e): let the ui suite run from a read-only cwd The playwright suite never executed on stage. It died in globalSetup before a single test ran, and the reported error was a red herring. /app/e2e/ui is a read-only filesystem in the packaged e2e image (the image runner already redirects playwright's own artifacts to TMPDIR for this reason), but the suite wrote three things relative to cwd: the per-role storageState files, the failure-screenshot directory, and the html report. Reproduced in the pod: storageState raises EROFS, mkdir test-results raises ENOENT. Worse, the catch block that exists to capture a screenshot threw its own ENOENT while handling a failure, so the real login error was replaced by a filesystem error. That is why the run looked like a missing directory rather than whatever actually went wrong. Route every artifact through ARTIFACT_DIR (E2E_UI_ARTIFACT_DIR, default "." to keep run_e2e.sh behavior unchanged), make the diagnostic screenshot best-effort so it can never mask the underlying failure, and point playwright's reporter and outputDir at the same place so a bare `npx playwright test` works there too. fixtures/users.ts had its own copy of the five storageState filenames; it now re-exports the ones from constants so the paths have a single definition. Verified in the read-only pod: both writes fail before, both succeed after. 85 tests enumerate and tsc --noEmit is clean. Refs LIT-4821 * fix(e2e): create the ui artifact root before writing into it storageState() does not create missing parents, and nothing created ARTIFACT_DIR itself. Pointing E2E_UI_ARTIFACT_DIR at a writable path that did not exist yet therefore failed with ENOENT on the very first role's snapshot, before any UI test ran; the same class of failure the artifact-dir change was meant to remove, just moved one level up. Reproduced: writing admin.storageState.json into a missing directory raises ENOENT. My earlier pod verification masked this because the probe called mkdirSync itself, which the real code path never did. mkdir the root once at the top of globalSetup, before the login loop. recursive makes it idempotent, handles nested paths, and keeps the default "." a no-op. Playwright creates its own outputDir lazily, so globalSetup is the only place that needs this, and migration.serverRootPath.globalSetup delegates here so it is covered too. * test(e2e): skip the mid-conversation cache checks pending LIT-4873 A mid-conversation role="system" reminder invalidates the prompt cache on the vertex_ai, azure_ai and bedrock_invoke Messages paths. Measured on the reminder turn, same conversation shape throughout: direct to api.anthropic.com 7013 read cache preserved litellm -> anthropic/claude-opus-4-8 7013 read cache preserved litellm -> vertex_ai/claude-opus-4-8 0 read cache destroyed and the Vertex control with the same added assistant/user turns but no reminder reads 7013, so it is the reminder on the non-first-party paths and not the extra turns. Anthropic keeping the cache rules out provider behavior; litellm's first-party anthropic path keeping it rules out the shared Messages transform. That makes these assertions correct and the failure a real billing bug, so the tests are skipped rather than weakened; the bodies stay intact and must be restored unchanged with the fix. Registry rows are left in place, so the three mid_conversation_system.nonstream.cache_hit cells report as uncovered gaps. Skips are decorators rather than a pytest.skip() inside the shared helper: a mid-function skip fires only after setup has already registered a real deployment via /model/new and left the rest of the body unreachable. Only Vertex was measured end to end. Azure Foundry and Bedrock Invoke are inferred from matching nightly failures and should be confirmed with the fix. Refs LIT-4821, LIT-4873
…c lag (BerriAI#34854) * test(e2e): harden harness and tests against data-plane pod churn A stage autoscaler scale-down produced a 2s window of ALB 502s that killed six budget tests on their first management call, and a freshly scaled-up pod that had not run its 30s DB object sync yet failed two MCP tests and one prometheus cardinality test. Retry transient gateway errors (502/503/504, connection errors) once at the shared e2e_http dispatch seam, poll MCP server registration to the poll deadline instead of asserting a single-shot listing, anchor the MCP guardrail full-sync wait to the later of the guardrail and server writes, and turn the prometheus alias poll into a drive-and-scrape convergence loop that re-sends traffic for missing aliases and unions results across scrapes * test(e2e): drain request body in retry stub handler so keep-alive reuse cannot misparse leftovers as requests * revert(e2e): drop the transient-502 retry seam A raw 502 during a pod scale-down is what a real client sees, so the suite retrying past it hides an availability gap instead of flagging it. The gateway-side fix is graceful drain on the deployment; until then the failures are signal * test(e2e): cap per-alias driver re-drives in the prometheus cardinality poll Bounds worst-case provider spend to 4 completions per alias while scrapes keep polling to the deadline; counters persist on whichever pod served them, so the cap costs no convergence unless that pod dies * test(e2e): drop driver re-drives from the prometheus cardinality poll The per-key cardinality contract is process-local and counters persist on whichever pod served the driver call, so unioning aliases across free scrape polls converges without re-sending billable traffic. The residual gap, a pod dying inside the poll window, is deferred to direct per-pod scraping
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Upstream update