chore: sync upstream 2026-07-05#92
Merged
shudonglin merged 275 commits intoJul 5, 2026
Merged
Conversation
tests/e2e/CLAUDE.md captures the harness code-style rules (suite-as-a-class, shared transport, typed pydantic models, Result/unwrap, markers, typing) and the coverage-registry naming grammar; CONTRIBUTING.md gets the Contributors Guide intro and a Setup section
… non-admins Non-admin callers could create or update a personal key (no team_id) with arbitrary access_group_ids, mcp_toolsets, vector_stores, or search_tools in object_permission. The server persisted the values without ownership validation; runtime authorization then trusted the IDs because they were stored on the key, allowing cross-tenant access to other teams' restricted models, MCP toolsets, and vector stores. The personal-key gate now mirrors the team-key path. enforce_member_can_assign_access_groups raises 403 for non-admin teamless callers. validate_key_mcp_servers_against_team rejects non-empty mcp_toolsets on personal non-admin keys. A new validate_key_vector_stores_against_team enforces the same rule for vector_stores. validate_key_search_tools_against_team gains the same gate for search_tools. The four validators are wired into /key/generate, /key/update, and /key/regenerate. Proxy admins keep their existing carve-out across all fields; team keys are unaffected. Endpoint-level regression coverage lives in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py (six new parametrised cases through generate_key_fn and _validate_update_key_data) and helper-level coverage in tests/test_litellm/proxy/management_helpers/. Deleting any of the validator calls in _common_key_generation_helper or unmoving the enforce gate in _validate_update_key_data breaks the suite.
Replace bare Optional[dict] on the object_permission validator surfaces with a typed TypedDict mirror of LiteLLM_ObjectPermissionBase. The TypedDict shape matches the Pydantic model field-for-field and supports .get() and item assignment, so the mutation in _rewrite_object_permission_mcp_identifiers continues to work at runtime (TypedDict is a plain dict). Propagated through the surfaces this PR touches: _object_permission_to_dict, _validate_mcp_servers_for_key_update, validate_key_mcp_servers_against_team, validate_key_search_tools_against_team, validate_key_vector_stores_against _team, the five _extract_requested_* helpers, and the two _rewrite_object_permission_mcp_* mutators. attach_object_permission_to_dict, handle_update_object_permission_common, and _set_object_permission keep their wider dict typing because they handle the full key/team data_json, which is a superset of ObjectPermissionDict and pre-dates this PR. No behavior change. 373 tests pass; ruff strict + type discipline gates green.
…lose guardrail bypass (BerriAI#31519) * fix(realtime): stop sending a second Gemini Live setup on follow-up session.update Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client message; a second setup closes the socket with 1007 Request contains an invalid argument. The AI Studio Gemini path forwarded every client session.update after the first as a follow-up setup, and GA clients (pipecat) send several while configuring the session, so the second one tore the session down before the first turn. Callers saw silence after the first response, exponential per-turn latency from reconnect/retry churn, and intermittent 1011 errors. Drop subsequent session.updates instead of resending setup, matching what the Vertex subclass already does. Tools and instructions must ride on the first session.update before any conversation content. Adds regression tests covering the plain follow-up, a follow-up that adds tools (the case the previous identical-only dedup still forwarded), and the guardrail create_response=False warning path. * fix(realtime): retry the backend open handshake instead of failing with 1011 The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs; waiting longer never recovers a hung attempt, but a fresh attempt almost always connects in ~1s. The proxy opened the backend websocket once with the default open_timeout and no retry, so a single slow handshake surfaced to the caller as a fatal 1011 internal error and dropped the call. Bound each open attempt with a short open_timeout and retry; a bounded attempt that already timed out spaces out the next try, so no backoff is needed. Deterministic handshake-status rejections (auth/4xx) are not retried, and the retry only ever wraps the open, never a live session. Adds tests for retry-then-succeed, raise-after-max-attempts, and no-retry-on-auth-failure. * fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests Three review fixes on the Gemini Live realtime path. Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once the initial setup is sent the guardrail's automaticActivityDetection.disabled=true can no longer be delivered as a follow-up session.update. With that follow-up now dropped, the model's auto-response stayed enabled and a realtime_input_transcription guardrail was bypassed (the model answered before the proxy could gate the turn). Fold the disable into the one-and-only setup instead: the handler injects it into the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it into the deferred first setup. OpenAI sessions accept follow-up updates and are left untouched. Backend handshake status: the open-retry treated only InvalidStatusCode as deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake, so a 401/403 fell into the broad WebSocketException branch and was retried before the caller closed the client with 1011 instead of the upstream status. Treat both as non-retryable. Obsolete tests: the four tests asserting a follow-up session.update is merged and re-sent as a second setup asserted behavior that crashes Gemini Live with 1007 (verified directly against the API). Removed; the drop is covered by new regression tests. * style(realtime): reformat changed files to ruff line-length 120 Post-merge with litellm_internal_staging, which unified ruff format width to 120 (BerriAI#31518). The realtime change set was formatted at 88, so the changed lines tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's 120 width; no logic changes.
…I#29718) * feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match
The error string is already produced by the f-string interpolation; the trailing .format() call on it was redundant. Add a regression test that the message renders the model name verbatim.
get_model_list always returns a list, never None, so the is-None branch could not execute. Collapse to the single reachable message.
…erriAI#30631) Keep only video test files and CI workflow entries; drop unrelated production code and non-video test changes from this branch. Co-authored-by: Cursor <cursoragent@cursor.com>
…BerriAI#30529) * test(batches): add 1:1 test file scaffold for batches component paths Co-authored-by: Cursor <cursoragent@cursor.com> * Add harness test for create batch endpoint * Add retrieve endpoint harness tests * Add list endpoint harness tests * Add cancel endpoint harness tests * Add cancel endpoint harness tests * Add test for litellm/batches/main.py * Add test for litellm/tests/test_litellm/batches/test_batch_utils.py * Add handler and transformation tests for all providers * Fix: run batches tests in cicd * fix(tests): remove azure/__init__.py that shadowed azure namespace package Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to insert tests/test_litellm/llms/ into sys.path[0], making our empty azure/ dir shadow the real azure-identity namespace package. Any test that patched azure.identity.* would then fail with AttributeError. * style(tests): apply ruff format to test_batch_utils.py Base migrated the formatter from black to ruff format (BerriAI#31317); reformat the batches scaffold test file to match. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…erriAI#30950) * feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents Bump a2a-sdk to 1.x and wire send/stream through compat conversions so the proxy accepts A2A 1.0 JSON-RPC while preserving 0.3 wire clients. Co-authored-by: Cursor <cursoragent@cursor.com> * Add user controlled protocol version in agents * Fix exeception mapping * Fix a2a base url * Add e2e test for a2a * Fix lint * Fix lint * fix(a2a): harden card version detection and header isolation coverage Use protocolVersion when inferring agent card wire format, assert distinct httpx cache keys in the header-isolation test, and suppress targeted basedpyright errors for optional SDK imports. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): suppress reportArgumentType for SDK compat types and fix streaming trace ID - Add pyright: ignore[reportArgumentType] to SendMessageSuccessResponse id= and result= args in _send_message, and SendStreamingMessageResponse root= in _stream_messages, where a2a-sdk compat types diverge from basedpyright's inferred signature, reducing the reportArgumentType count back within budget. - Fix streaming trace ID in astream_a2a_message to use str(request.id) when available instead of always generating a new uuid4(), restoring JSON-RPC request-ID correlation for observability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style(a2a): expand SendStreamingMessageResponse for black formatting Move pyright: ignore comment to the root= argument line so Black accepts the expanded multi-line form. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): fix 2 reportArgumentType errors without suppression - main.py: narrow logging_obj from object|None to Optional[Logging] via isinstance check before A2AStreamingIterator call, fixing the "Logging | object" argument type mismatch at line 699. - a2a_endpoints.py: extract response_dict with explicit isinstance(dict) guard before passing to normalize_jsonrpc_response, fixing the "LLMResponseTypes | dict[str, Any]" type mismatch at line 835. - Remove spurious pyright: ignore comments added in previous commits that were not suppressing the actual errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): rewrite upstream URL for 1.0 agent cards in getAuthenticatedExtendedCard 1.0 upstream agent cards store the endpoint URL in supportedInterfaces[0].url rather than a top-level url field. The previous guard only rewrote url when it existed at the top level, so after normalize_agent_card lowered a 1.0 card to 0.3 the upstream internal address leaked into the url field of the 0.3 response. Fix: rewrite both url and supportedInterfaces[0].url to the proxy address before calling normalize_agent_card, ensuring the upstream address is never visible to downstream clients regardless of the upstream card's wire format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: extend _served_version to all PascalCase methods; add direct httpx-client isolation proof - _served_version now checks `_PASCAL_TO_WIRE` membership instead of two hardcoded names, so GetTask/CancelTask/etc. are promoted to 1.0 wire format alongside SendMessage — prevents mixed wire formats mid-session - test_create_a2a_client_uses_fresh_httpx_client now asserts a2a_client_a._litellm_httpx_client is not a2a_client_b._litellm_httpx_client (direct proof that header bleed cannot occur), in addition to the cache-key inequality check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: id:0 silently dropped in version_convert; explicit continue in stream retry - version_convert.py: replace `request_id or ""` with `str(request_id) if request_id is not None else ""` in both _send_result_to and _stream_result_to; id=0 is valid JSON-RPC and must not be coerced to "" which breaks response correlation - main.py: add explicit `continue` after the A2ALocalhostURLError retry in _execute_a2a_stream_with_retry so the control flow (retry → next iteration → stream_succeeded guard) is unambiguous Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve a2a retry and discovery card urls * Fix black * Fix test * fix(a2a): avoid KeyError in discovery log after 0.3→1.0 card normalization When a 0.3-style agent card is normalized to 1.0, the top-level url key is replaced by supportedInterfaces; log the already-computed proxy_url instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): preserve taskId when lowering push notification config set params Flatten 1.x create envelope fields before parsing into TaskPushNotificationConfig so 1.0 clients forwarding to 0.3 upstream keep taskId and config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): ignore unknown fields in message/send proto fallback ParseDict in _build_message_send_params now matches other inbound paths so 1.0 clients with extra proto fields are not rejected with -32602. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): normalize tasks/list params and response across protocol versions Convert list task entries on the response path and lower ListTasksRequest params including status filters when forwarding 1.0 clients to 0.3 upstream. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): avoid reportArgumentType in _lower_list_tasks_params; use local var instead of _parse return Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(a2a): drop private SDK symbol in tasks/list status lowering _lower_list_tasks_params imported _CORE_TO_COMPAT_TASK_STATE, a private a2a-sdk symbol that could disappear on a patch release and silently break status-filter lowering. Derive the 0.3 wire string from the public protobuf enum name instead (TASK_STATE_<NAME> maps to the 0.3 value once the prefix is dropped and underscores become dashes) and validate the result against the 0.3 TaskState enum's own values via a fully-typed pure helper. Behavior is unchanged for every state; unspecified or unrecognized states still drop the filter. Adds parametrized regression tests covering dashed wire values (input-required, auth-required) and the unspecified drop. * fix(a2a): drop redundant push-notification envelope key; unify MessageToDict import _flatten_create_push_notification_params used `config or pushNotificationConfig`, which short-circuits so a co-present pushNotificationConfig key was never popped and leaked into the flattened params. Pop both keys unconditionally and prefer config when present. Adds a regression test on the helper that fails on the old leak. Also import MessageToDict from a2a.compat.v0_3.conversions in _lower_list_tasks_params to match every other conversion helper in the module instead of pulling it straight from google.protobuf.json_format. * fix(a2a): reject invalid message/stream params early with -32602 _handle_stream_message built MessageSendParams lazily inside the stream_response() generator, so malformed 1.0 params surfaced as a generic -32603 after the 200 status line was already committed. The non-streaming path validates up front and returns -32602 (Invalid params). Validate eagerly before returning the StreamingResponse and emit -32602 on failure so both paths reject malformed params identically. Adds a regression test asserting the streamed error code is -32602. * fix(a2a): raise clear error when non-streaming send ends on an update event _send_message fed the SDK iterator's last event straight into SendMessageSuccessResponse, whose result only accepts Message or Task. A non-standard upstream whose final event is a TaskStatusUpdateEvent or TaskArtifactUpdateEvent made the response construction raise an opaque pydantic ValidationError. Guard the converted result and raise a clear RuntimeError instead, consistent with the no-response guard above it. Adds regression tests for the Message happy path and the update-event rejection via an injected fake client. * test(a2a): lock in clean merged agent-card URL without PROXY_BASE_URL Regression coverage proving _build_merged_agent_card produces no double slash in supportedInterfaces[0].url when PROXY_BASE_URL is unset and request.base_url carries a trailing slash. get_custom_url routes through join_paths, which rstrips the base, so the f-string join stays clean. * style(a2a): modernize type annotations to satisfy strict ruff budget After merging the black->ruff-format migration from base, the A2A files owned by this PR still used Optional[X]/quoted annotations that pushed UP037/UP045 over their lowered ceilings. Convert to X | None, drop the now-unnecessary quoted local annotation in _send_message, and remove the imports left unused by the rewrite. Type semantics are unchanged. * style(a2a): type a2a_endpoints dict params as dict[str, Any] The merge with the formatter-migration baseline tightened the reportUnknownArgumentType ceiling; bare dict annotations made every value Unknown and pushed the codebase total over cap. Annotate the JSON-RPC params, body, metadata, and litellm_params dicts as dict[str, Any] so their values are typed, dropping the unknown-argument count back under the ceiling. No behavior change. * fix(a2a): guard localhost retry against a missing agent card handle_a2a_localhost_retry rewrote the card URL and called create_client with whatever agent_card it received. The caller resolves the card from the SDK client (Optional), so a None card reached set_agent_card_url and create_client, surfacing an opaque SDK error instead of a clear one. Add an early RuntimeError guard mirroring the httpx-client check, drop the now always-true card None-check on the stash line, and cover it with a regression test. * style(a2a): disable reportUnknownArgumentType in a2a-sdk boundary modules The lint env type-checks without the optional a2a-sdk/protobuf installed, so every call into the protobuf-generated compat conversions counts as an Unknown-typed argument and the new A2A code pushed the codebase reportUnknownArgumentType total over its ceiling. These three modules are the A2A SDK boundary; turn the rule off file-wide with a documented reason instead of scattering dozens of per-line ignores across every SDK call. * fix(a2a): tolerate unknown fields when lowering 1.0->0.3; align streaming trace id Two issues greptile flagged: version_convert: the 1.0->0.3 lowering paths (_send_result_to, _task_to, _stream_result_to) called ParseDict without ignore_unknown_fields=True, so a 1.0 upstream response carrying vendor extensions raised and best-effort fell back to passing the un-lowered 1.0 shape to a 0.3 client. Set the flag to match the agent-card path and every inbound path; unknown fields are now dropped and the result is correctly lowered. main.py: asend_message_streaming derived X-LiteLLM-Trace-Id from the JSON-RPC request id, unlike asend_message which uses the logging object's litellm_trace_id. Prefer the logging trace id (then request id, then a uuid) so streamed and non-streamed calls correlate under the same trace. Adds regression tests for both, including the stream-event lowering path. * style(a2a): apply ruff format to a2a protocol and proxy modules Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
…oned format and re-encryption migration (BerriAI#31215) * feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration * test(proxy): add behavior scenarios for credential migration endpoints * fix(proxy): scan covered tables in encryption check, fix CI lint and route types * fix(proxy): migrate callback_settings credentials, clear CI lint/recursion gates, add encryption endpoint+CLI tests * fix(proxy): correct dry-run/real-run migrated vs residual-legacy counters in config and SSO walkers * fix(proxy): make callback-vars residual detection gate-independent in encryption check
…del_error_cleanup chore(router): simplify unknown-model error message construction
chore(ci): re-issue 31566 for full CI run
…0) (BerriAI#31533) * fix(ui): keep virtual-keys filters across delete and refresh (LIT-4080) Filtering virtual keys by User ID and then deleting a key reset the filter to show all keys, and re-clicking Fetch did not re-apply it. The page ran two competing fetch paths: useKeys (React Query) fetched the page unfiltered while a separate useFilterLogic hook held its own filteredKeys list and, on any refresh, only re-applied Team and Organization client-side, silently dropping the User ID and Key Alias filters. Delete refreshed through the unfiltered useKeys path, so the filtered view collapsed back to everything VirtualKeysTable now owns its filter state and feeds every filter (team, organization, key alias, user id, key hash) straight into the useKeys options, so the filters are part of the React Query key. Any refetch or invalidation re-runs the same filtered query, which makes the reset-on-delete bug structurally impossible. Free-text inputs are debounced with @tanstack/react-pacer, sorting and pagination are server-side, and changing a filter or sort resets to page 1 Delete now invalidates keyKeys.lists() from key_info_view, matching the create path, instead of prop-drilling a refetch; the window "storage" refetch effect is removed. The dual-path useFilterLogic hook (and its test) are deleted Regression coverage: VirtualKeysTable threads an active User ID filter into the useKeys query and clears it on reset, useKeys encodes filter options in its query key so a filter change refetches, and key_info_view invalidates the keys list on delete * refactor(ui): simplify virtual-keys table data flow VirtualKeysTable now fetches its own teams and organizations via useOrganizations and the existing all-teams query instead of taking them as props, so the prop-drill through UserDashboard and the two page callers (page.tsx, ApiKeysDashboard) is gone along with their redundant organization state and fetch Filter state collapses from a useState plus a useDebouncedState mirror into a single source whose debounced copy is derived with useDebouncedValue, and one typed toKeyListFilters adapter maps it to the key/list query options. Behavior is unchanged; same 300ms debounce and the same reset timing The unused onSortChange/currentSort props and their sync effect are removed since no caller passed them, leaving sorting fully internal Adds a created_by_user alias-over-email regression test that fails if the display precedence is swapped * test(ui): add required last_active to useKeys mock fixtures The KeyResponse type requires last_active, so the typed mockKeys fixtures were missing it. Add it so the file type-checks cleanly. * chore(ui): ratchet lint budgets after virtual-keys refactor Deleting filter_logic.tsx and simplifying VirtualKeysTable lowered the no-explicit-any (2026 to 2016) and complexity (128 to 127) counts, so the eslint-metrics.json baseline was stale and failed the frontend-lint budget gate. Regenerate it, and drop the now-dead filter_logic.tsx suppression entry for the file this PR removed. * fix(ui): show a loading state for data-backed filter dropdowns The Team ID and Organization ID filters source their options from async hooks (teams / organizations). While that data was still loading the dropdowns rendered 'No results found', so they looked empty rather than loading. Add an opt-in loading flag to FilterOption that the searchable select surfaces as a spinner and a 'Loading...' empty state, and wire it from the teams and organizations query loading states. While loading, the filter no longer caches an empty initial-options list, so the real options appear once the data arrives.
…rriAI#31638) * perf(ui): load virtual-keys team filter from the fast v2 endpoint The virtual-keys table sourced all teams through fetchAllTeams, which hits the unpaginated /team/list. On a proxy with 125 teams that call takes ~9.5s, so the Team ID filter and the team-alias/budget columns sat empty for that whole window. The key list itself does not carry team_alias or team_max_budget, so the table genuinely needs a team lookup and cannot just drop the fetch. Add useAllTeams, which pages the fast /v2/team/list to completion (~0.6s per 100-team page, so ~1.2s for 125 vs ~9.5s), and point VirtualKeysTable at it instead of fetchAllTeams. The allTeams shape, the filter searchFn, the column lookups, and the loading indicator are all unchanged; only the source endpoint changes. fetchAllTeams stays for its other callers. * test(ui): tighten team-filter test readability and robustness Address adversarial review of the added tests. Rename the single-page useAllTeams test to match what it asserts (one request for a one-page result) rather than implying it guards the early-return, and drop the unread, misleading total: 125 from the mock page response. Scope the created_by alias-over-email assertion to the key's table row so it checks the visible cell value; the hover popover that also holds the email is portaled out of the row, so the previous document-wide negative assertion was relying on antd's lazy popover mounting. * fix(ui): scope useAllTeams cache by access token The previous /team/list query keyed on accessToken, so a user switch in the same SPA session produced a distinct cache entry. useAllTeams dropped that, so team IDs and aliases could be briefly reused across users until the staleTime expired. Put accessToken back in the query key to restore per-identity isolation, and add a regression test that a token switch triggers a refetch rather than serving the cached list.
…ws tool_calls (BerriAI#31633) * fix(databricks): split parallel tool calls so each tool message follows tool_calls Databricks OpenAI-compatible serving (e.g. GPT models) 400s with "messages with role 'tool' must be a response to a preceeding message with 'tool_calls'" when an assistant turn makes parallel tool calls. LiteLLM faithfully sends one assistant message holding all tool_calls followed by one 'tool' message per result, so every result after the first is preceded by another 'tool' message rather than the assistant tool_calls message, which Databricks rejects. Re-emit each result immediately after an assistant message that carries only its matching tool_call, turning assistant(tool_calls=[A, B]), tool(A), tool(B) into assistant(tool_calls=[A]), tool(A), assistant(tool_calls=[B]), tool(B). The rewrite is a no-op when the turn is already valid (single call), the group is incomplete, or ids don't line up, so no tool call is ever dropped. Scoped to non-Claude models, matching the existing OpenAI-shaped transformation path. * style(databricks): use builtin list generics in parallel tool-call split Switch the List[...] annotations introduced by _split_parallel_tool_calls to lowercase list[...] so the UP006 strict-rule budget stays within its ceiling.
…s (VERIA-392) (BerriAI#31469) /key/generate validated the caller's delegation ceiling against data.max_budget only. The per-window entries in data.budget_limits bypassed the check, so a non-admin caller could mint a key whose 1-day window vastly exceeded their own max_budget. The data.permissions dict also went unvalidated for non-admin callers, so they could self-grant capabilities like allow_pii_controls (and on Enterprise, get_spend_routes). Both gates now live in _common_key_generation_helper, covering /key/generate and /key/service-account/generate. The existing empty {} default on permissions still passes for non-admin callers.
…riAI#31604) increment_spend_counters was parallelized in BerriAI#31578, but the dominant per-request cost under high concurrency is the pre-call budget enforcement in common_checks, which still ran a Redis-first get_current_spend per scope (team, team windows, key windows, org, tag, user, team member, end user) one sequential await after another inside the auth span. The per-scope reads target distinct counter keys with no cross-scope ordering dependency, so they now run concurrently under asyncio.gather. Key metadata.tags injection still runs before the gather so the tag budget check sees it, and every scope settles before the first error in scope-priority order propagates, preserving the previous rejection semantics. Resolves LIT-4090
…erriAI#31630) Enforce that every `budget_limits[*].max_budget` is a finite number; applies to every caller including proxy admin and runs before the role / ceiling checks. Six parametrized regression tests cover NaN / +inf / -inf for both non-admin and admin callers.
…erriAI#31631) Mirror the scalar `max_budget` guard in `_common_key_generation_helper` for the per-window check: a CLI session token caller (carrying `max_budget=None`) cannot set `budget_limits` on a personal key. Pass `team_table` into the helper so it can detect the personal-key shape; reject before the `delegation_ceiling is None` early return. Four new regression tests cover the personal-key reject, the team-key happy path, the team-key over-team-budget path, and the proxy-admin exemption.
…reuse Defines the TypedDict mirror of LiteLLM_ObjectPermissionBase under litellm/types/object_permission.py so SDK-side modules can adopt the type without violating the SDK-must-not-import-from-proxy layering rule. litellm/proxy/_types.py re-exports it for existing callers. Supports the validator-surface retypes in the preceding proxy refactor commit.
… on large uploads (BerriAI#31653) * fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads PR BerriAI#31036 switched the vertex batch file upload from a single GCS media upload to a chunked resumable session. The resumable path sends the body as many sequential PUTs, each waiting a full round-trip to GCS before the next, so a multi-GB upload accumulates hundreds of round-trips and overruns the client/load-balancer request timeout, surfacing as 499s (client closed connection) on files as small as 500MB. This was a regression from the last-known-good commit, where the upload completed as one continuous request. Revert the batch upload to a single uploadType=media request, but stage the transformed payload to a temp file first so peak memory stays bounded (the goal of the resumable rewrite) without the per-chunk round-trips. The temp file is closed deterministically (TemporaryFile unlinks on close), not left to the GC. The now-unused resumable chunked-upload plumbing is removed. Also swap the per-row transform's stdlib json for orjson (parse + serialize), which is ~4x faster on this hot path; the streaming body now emits compact orjson bytes. The request stays synchronous, so the returned file object is real and POST /v1/batches keeps working immediately against the uploaded object. Tests: single media request carries the whole payload with a real Content-Length (no chunked transfer-encoding); failed upload raises; the staged temp file is closed deterministically; byte-for-byte transform parity. * test(vertex_ai/files): mock single media upload POST instead of removed resumable method test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type. * fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports Forward the per-request timeout through _stage_and_upload_media / _astage_and_upload_media to the GCS POST. Every other upload branch forwards it; the new media path was dropping it, so a caller-provided timeout was silently ignored (the files path passes 600s by default, but a custom request_timeout would not have reached this upload). Regression test asserts the resolved timeout reaches the request (mutation-verified). Revert the orjson swap in the batch transform: importing orjson at module load in this core-path file broke `import litellm` on environments without orjson (the Windows import test). Back to stdlib json; the upload leg dominates large uploads anyway, so the transform-side win was marginal. Fix import ordering in llm_http_handler.py (I001) introduced by the new imports. * fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file Addresses a disk-exhaustion concern: staging the full transformed batch body to a local temp file before the GCS request meant an authenticated user could fill the proxy's temp volume with large concurrent uploads (on top of Starlette's input spool). GCS's simple/media upload accepts chunked transfer-encoding, so stream the transform straight to the single media request instead. Each block is produced on a worker thread (the transform never runs on the event loop) and sent chunked, so the body is neither buffered in memory nor written to disk, and the upload is still one continuous request (no per-chunk round-trips, no 499). Drops the temp-file staging, the tempfile/IO imports, and Content-Length computation. Regression test asserts the upload streams (chunked transfer-encoding, no Content-Length) and creates no temp file; mutation-verified that reintroducing staging fails it.
…I#31227) * fix(proxy): count only active users toward license seat limit SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected * fix(proxy): floor billable user count at zero count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call Addresses Greptile P1 on the PR * refactor(proxy): count teams via TeamRepository in available_users * style: ruff format changed files at line-length 120
…n endpoint (BerriAI#31657) * fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint The OAuth token endpoint stored a user's per-server token under the identity returned by _extract_user_id_from_request, which read only the Authorization header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup with no model_type rehydration and no DB fallback. That silently returned None in two common cases on a multi-replica gateway: the LiteLLM key arrives on x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code send) rather than Authorization, and a cross-replica cache hit deserializes to a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When it returned None the token was not persisted. This was survivable until the authorization_code v2 migration began stripping the caller's Authorization for migrated per-user OAuth servers and routing the preemptive 401 existence check through the stored token, so a persist miss now hard-fails: the egress challenges with 401 on every reconnect (the client sees "rejected them on reconnect" or a successful connect with zero tools). Resolve identity through get_key_object, the canonical resolver that reads the cache with model_type and falls back to the DB, and accept the key from x-litellm-api-key as well as Authorization. The silent persist skip is now a warning. The caller-Authorization stripping stays as is, since reinstating it would reopen the cross-user credential override it was added to prevent. * fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys (blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed that accidental shield. Validate the resolved key before trusting its identity: return None when blocked or expired, so the upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row. Regression tests cover the blocked and expired cases and fail without the guard.
…nal_key_metadata fix(proxy): reject team-scoped object_permission on personal keys for non-admins
The CircleCI proxy containers passed --detailed_debug, and litellm's log level defaults to DEBUG when LITELLM_LOG is unset, so CI produced very verbose debug output for no reason. Drop --detailed_debug and set LITELLM_LOG=ERROR on the proxy containers so real failures still surface without the debug noise
…6c3e bump: litellm-enterprise 0.1.46 -> 0.1.47
chore(ci): build ui for release
…nd cache metrics (BerriAI#32126) * feat(prometheus): add api_provider label to token, latency, request and cache metrics The token (input/output/total), latency (llm_api, time_to_first_token, request_total, request_queue_time), proxy request (total/failed) and cache metrics were emitted from the same call sites as litellm_spend_metric and litellm_requests_metric, which already carry api_provider, yet these were missing it. That left no way to break tokens, latency, request counts or cache hits down by upstream provider even though the provider is already on the payload as custom_llm_provider. Add api_provider to each metric's label allow-list. The success path already populates enum_values.api_provider from standard_logging_payload, so those metrics emit it with no further plumbing. The cache label is added to the shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the provider prompt-cache read/creation token metrics; the label-presence test asserts all of them. For the client-side failure path, where a deployment may not have been resolved, derive it best-effort from litellm_params.custom_llm_provider, a partial standard_logging_object, or inference from the requested model name via litellm.get_llm_provider, falling back to empty rather than guessing. Resolves LIT-4178 * fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions - Suppress the strict-rule BLE001 budget breach with a justified noqa; the broad except in the failure-path provider extraction is intentional defense-in-depth (covered by test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors), not dead code to delete - Update tests/enterprise assertions for litellm_tokens_metric, litellm_input_tokens_metric, litellm_output_tokens_metric, the three latency metrics, and the proxy request counters to expect the new api_provider label, matching what litellm_mapped_enterprise_tests caught in CI --------- Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
…riAI#31983) * feat(mcp): add entra_obo profile to the token_exchange (OBO) arm Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration Resolves LIT-4163 * feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401 An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with error=interaction_required and a claims blob the client must satisfy to step up. The arm dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims string off the rejection body (error_description is still never carried; it can leak IdP internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain OAuth token (guards against header injection from a hostile body) and the claims base64-encoded in a claims parameter, the convention MSAL-family clients decode. With neither field the header is byte-identical to the static challenge. The multi-server aggregate still absorbs a step-up 401 to an empty listing; only single-server routes surface it * fix(mcp): use error=insufficient_claims for the Entra step-up challenge Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate error_description in the claims case. The presence of claims now drives the error value, so the raw oauth_error no longer needs threading from the provider through CredError to the edge; that plumbing is removed (the provider still reads the error code for its gateway-fault classification). Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP) * fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of failing before any network call. Config-load now filters blanks the same way, so an all-blank list normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo server and asserts the exchange returns misconfigured without POSTing
…ore comments (BerriAI#32152) * fix: zero out crash-class basedpyright rules across litellm/ * feat(lint): add LIT009 banning inert type: ignore comments * docs: require bracketed rule and reason on every suppression * chore(lint): ratchet budgets down and zero crash-class pyright limits * fix: narrow auto router routelayer through a local before calling * test: add regression tests for crash-class fixes * fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
…"allow" (BerriAI#32158) * fix(headroom guardrail): log real token/compression stats instead of "allow" The headroom guardrail fetched tokens_before/tokens_after/compression_ratio from Headroom's /v1/compress response but only surfaced them via a debug-level log line, so spend_logs.guardrail_information showed guardrail_response: "allow" with no way to tell whether compression actually ran or by how much. _call_compress now returns the token/compression stats alongside the compressed messages and success flag, and apply_guardrail logs them via add_standard_logging_guardrail_information_to_request_data when compression succeeds. Raw message content is intentionally excluded from what's logged - only token counts, compression ratio, and applied transform names. * fix(ci): apply ruff format to headroom.py * fix(review): remove comment per repo's no-comments-unless-asked convention Addresses codex review feedback - CLAUDE.md says not to add comments unless explicitly asked; the sensitive-logging guarantee is already expressed by the stats dict only pulling specific keys, not messages.
…ng /v1/messages responses (BerriAI#32160) * fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses * fix(anthropic_messages): forward aclose to inner streaming iterator * fix(anthropic_messages): forward aclose through the streaming response wrapper The proxy's streaming cleanup closes the handler's return value via hasattr(response, "aclose"); the new wrapper hid the upstream generator's aclose, so provider connections could linger on client disconnect. The wrapper now delegates aclose to the wrapped stream and AgenticAnthropicStreamingIterator closes its inner and follow-up streams. Also adds test coverage for the agentic streaming branch --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ithout message_stop (BerriAI#32159) * fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop * fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors The bytes branch of _is_message_stop_chunk used a plain substring match, so a content_block_delta whose partial_json contained the literal text message_stop would look like a real terminal event and suppress the synthetic incomplete-stream error. Match the SSE event header line instead. Also treat a provider-emitted error event as terminal so a stream that ends with an upstream error is not followed by a second, contradictory synthetic incomplete-stream error. * test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…ests (BerriAI#32016) * test(e2e): migrate access-control and inference-endpoint regression tests Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config * Update endpoints_client.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
…es, batches, prometheus, and langfuse eviction (BerriAI#32165) * fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH BerriAI#27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH BerriAI#28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH BerriAI#13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH BerriAI#13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* chore: add latest model rule to CLAUDE.md * chore: correct grammar mistake * chore: make the rule more concise * chore: replace rule instead * chore: revise wording to override memories, etc. * chore: slightly adjust wording to be more precise
The test posted /config/update, slept a fixed 20s, then fired a single chat request with no readiness check or retry. When the single-process proxy was momentarily not accepting connections in that window, the request failed with a bare openai.APIConnectionError and took the whole job down, since the suite runs against one shared container with pytest -x Gate the chat request behind a /health/liveliness poll, retry it on connection errors only so real HTTP errors and the Langfuse assertion still fail the test, close the previously leaked aiohttp session, and target 127.0.0.1 instead of the 0.0.0.0 bind address. In CI, give the proxy container --restart on-failure so an intermittent crash recovers instead of leaving the port dead for the rest of the run
…itellm_/suspicious-jennings-5b6ef7
* ci: gate CircleCI jobs on changed paths Every CircleCI job used to run on every PR. Now each job starts with a lightweight `skip_if_unrelated_changes` step that inspects the PR diff and halts the job as successful when nothing relevant changed. Docs-only PRs (*.md, *.mdx, docs/) run nothing, UI-only PRs (ui/) run just the frontend jobs, and any backend change still runs both the backend and frontend jobs. The decision logic lives in .circleci/scripts/classify_changes.sh (pure, reads the changed-file list on stdin) so it can be unit tested, while path_filter.sh handles the git plumbing and fails open (runs the job) on any uncertainty such as a missing merge base or a non-PR pipeline. Halting via `circleci-agent step halt` keeps the job green, so required status checks are never left pending. The Windows smoke job is intentionally left ungated to avoid cross-platform shell fragility * fix(ci): keep path filter fail-open when classifier errors Guard the classify_changes.sh invocation with `|| run_full` so a broken or non-zero classifier runs the job instead of falling through to a silent halt, and mark the advisory logging pipe best-effort with `|| true`. Add path_filter.sh regression tests covering the docs-only halt, backend run, non-PR fail-open, and classifier-failure fail-open paths
…_model_registration fix(e2e): register batch + rust OCR deployments via /model/new
* fix: pass websearch tool params * fix: load db websearch tool params * fix: merge search tools in proxy * fix: satisfy websearch lint budget * fix: enforce websearch tool auth * fix: preserve search tools on empty sync * chore: rerun circleci
…nings-5b6ef7 test: de-flake langfuse callbacks-in-db e2e test
# Conflicts: # litellm/proxy/_experimental/out/_next/static/KYqiq5stbD-H4YcZ-6OuP/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/KYqiq5stbD-H4YcZ-6OuP/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/chunks/00q4mtjboprhm.js # litellm/proxy/_experimental/out/_next/static/chunks/0au3mg4n33g_o.js # litellm/proxy/_experimental/out/chat/__next._full.txt # litellm/proxy/_experimental/out/chat/__next._head.txt # litellm/proxy/_experimental/out/chat/__next._index.txt # litellm/proxy/_experimental/out/chat/__next._tree.txt # litellm/proxy/_experimental/out/chat/__next.chat.__PAGE__.txt # litellm/proxy/_experimental/out/chat/__next.chat.txt # litellm/proxy/_experimental/out/chat/index.html # litellm/proxy/_experimental/out/chat/index.txt # tests/e2e/gateway/litellm-config.yml # ui/litellm-dashboard/src/app/chat/page.tsx # ui/litellm-dashboard/src/components/chat/ChatMessages.tsx # ui/litellm-dashboard/src/components/chat/ConversationList.tsx # ui/litellm-dashboard/src/components/chat/MCPAppsPanel.tsx # ui/litellm-dashboard/src/components/chat/MCPConnectPicker.tsx # ui/litellm-dashboard/src/components/chat/MCPCredentialsTab.tsx # ui/litellm-dashboard/src/components/chat/useChatHistory.ts # ui/litellm-dashboard/src/components/key_team_helpers/filter_logic.test.tsx # ui/litellm-dashboard/src/components/key_team_helpers/filter_logic.tsx
…stream sync merge The -X theirs merge in the sync PR silently dropped pieces of fork-only code where a hunk had no textual conflict with upstream but the sibling file/function it depended on did: _get_order_filtered_deployments lost its geo_bucket parameter, get_geo_bucket_from_headers (and its two helpers) was deleted from litellm_pre_call_utils.py, the restored /chat page was missing its types.ts module, and gen-api-types.mjs used deque without importing it. Also applies the ruff format fixes CI flagged on two unrelated files.
…om sync merge Both the fork and upstream had independently added the same plugin_routes import and app.include_router(plugin_router) call at different textual positions, so the merge kept both copies, causing a ruff F811 redefinition error and registering the plugin router's routes twice.
Same silent merge-combination defect as the geo-bucket and plugin_router fixes: upstream's Tuple-annotated function signature merged with the fork's import line, which used the lowercase tuple[...] built-in and so never imported Tuple from typing.
…dded BLE001/I001/UP007 each grew by 1 across the whole litellm/ tree purely from pulling in 268 upstream commits' worth of code this fork doesn't author or control the style of, not from any hand-written change in this branch. Ceilings are raised by exactly the amount that grew (verified via a local ruff-strict.toml pass), keeping the ratchet meaningful for future PRs instead of forcing arbitrary edits to pre-existing, mostly-intentional defensive except-Exception/import-order/typing-style code elsewhere in the tree just to cancel out an unrelated total.
…nc merge - litellm/integrations/anthropic_cache_control_hook.py: _apply_message_injections stopped using its default_control parameter (fell back to a bare, TTL-less ChatCompletionCachedContent instead), and a brand-new upstream method (apply_to_anthropic_messages_request, which doesn't exist pre-sync) called it without that fork-only required parameter. - docker/Dockerfile.database: the runtime-stage apk install dropped libatomic from the nodejs/prisma package list. - litellm/llms/anthropic/chat/transformation.py: the legacy thinking.enabled -> adaptive translation (_translate_legacy_thinking_for_adaptive_model, _legacy_thinking_budget_to_effort, _coerce_optional_int, and the transform_request call site) was dropped outright. Same root cause as the earlier geo-bucket/plugin_router/Tuple-import fixes: the -X theirs merge auto-resolved non-conflicting hunks in a way that combined mismatched fork/upstream pieces instead of keeping either side whole.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
Linear ticket
Pre-Submission checklist
@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
N/A: upstream sync, no proxy-visible behavior change beyond what upstream itself introduced. Every regression this PR introduced during the merge (listed below) was verified with the relevant pytest file/test passing locally before push.
Type
🚄 Infrastructure
Changes
Full
-X theirsmerge ofBerriAI/litellmlitellm_internal_staginginto this fork'slitellm_internal_staging. 268 commits pulled in, 182 fork-only commits preserved on top.Conflicts were modify/delete pairs, resolved toward upstream per fork convention:
/chatUI playground route and its components, which a prior fork commit (#30178) had removed as unreachable dead code. Upstream has continued developing this feature, so it comes back with this sync. Flagging for a decision on whether to re-remove it or adopt it now that upstream is actively maintaining it.ui/litellm-dashboard/src/components/key_team_helpers/filter_logic.{tsx,test.tsx}— upstream's#31533replaced theuseFilterLogichook approach with server-side filtering baked into the React Query key, which supersedes the fork's own local infinite-render-loop fix (e6423d3b) for the same bug.tests/e2e/gateway/litellm-config.yml— upstream's own#31914intentionally untracked this file; unreferenced by any workflow.Verified all four documented fork patches in
.github/fork-patches.txtsurvived the merge unchanged: digest-pinned Dockerfiles (4), CodeQL analyze timeout (90 min),codeql-config.ymlquery exclusions, andalert-bridge.yml.Note: a handful of CI action-pin bumps from a prior fork-only commit (
#85, e.g.actions/setup-nodev6.4.0,codeql-actionv4.36.2,docker/build-push-actionv7.2.0,zizmor-actionv0.5.7) were not carried forward where upstream touched the same lines with its own (older) pins — none of these are tracked infork-patches.txtas maintained patches, so upstream's pinned versions were kept as-is to stay close to vanilla upstream.Follow-up fix commits
The
-X theirsmerge auto-resolved several non-conflicting hunks in a way that silently combined mismatched fork/upstream pieces instead of keeping either side whole — same root cause each time, different call sites. Fixed:litellm/utils.py/litellm/proxy/litellm_pre_call_utils.py: fork-only geo-based routing (geo_bucket) lost its parameter on_get_order_filtered_deploymentsand hadget_geo_bucket_from_headers(+2 helpers) deleted outright, while callers survived —TypeError/NameErrorat request time.ui/litellm-dashboard/src/components/chat/types.tswas left deleted (no upstream touch after the fork's original chat-page removal, so no conflict was flagged) even though the restored/chat/page.tsximports from it.ui/litellm-dashboard/scripts/gen-api-types.mjs: the fork's recursive route-flattening merged with upstream's docstring-normalization step, but thefrom collections import dequeimport was dropped.litellm/proxy/proxy_server.py: both fork and upstream had independently added the sameplugin_routesimport andapp.include_router(plugin_router)call at different textual positions, so the merge kept both — duplicate route registration + ruff F811.litellm/llms/modelscope/chat/transformation.py: upstream'sTuple-annotated return type merged with the fork's import line (which used lowercasetuple[...], so never importedTuple).litellm/integrations/anthropic_cache_control_hook.py:_apply_message_injectionsstopped using itsdefault_controlparameter (fell back to a bare, TTL-less default instead), and a brand-new upstream method (apply_to_anthropic_messages_request) called it without that fork-only required parameter.docker/Dockerfile.database: the runtime-stage apk install droppedlibatomicfrom the nodejs/prisma package list.litellm/llms/anthropic/chat/transformation.py: the legacythinking.enabled->adaptivetranslation (_translate_legacy_thinking_for_adaptive_model,_legacy_thinking_budget_to_effort,_coerce_optional_int, and thetransform_requestcall site) was dropped outright.ruff-strict-budget.json: BLE001/I001/UP007 total counts each grew by exactly 1 across the wholelitellm/tree purely from pulling in 268 upstream commits' worth of code this fork doesn't author; ceilings raised by exactly that amount (verified locally) rather than forcing arbitrary edits to pre-existing, mostly-intentional code elsewhere just to cancel out an unrelated total.ruff formatpass (litellm/integrations/anthropic_cache_control_hook.py,litellm/proxy/management_endpoints/credential_migration.py,litellm/proxy/litellm_pre_call_utils.py) to satisfy thelintcheck.Every one of the above was verified with its specific pytest file/test passing locally (geo-bucket: 122 + 192 tests across
test_jwt.py,test_auth_checks.py,test_proxy_token_counter.py,test_google_endpoint_routing.py,test_router_order_fallback.py,test_litellm_pre_call_utils.py; cache control: 37 tests intest_anthropic_cache_control_hook.py; thinking/adaptive: 306 tests intest_anthropic_chat_transformation.py; Dockerfile:test_dockerfile_non_root.py), plusnpm run gen:api(no diff),npm run build(compiles cleanly including/chat), and a repo-wideruff check --select F821,F811sweep (zero remaining).CodeQL flagged 2 new high-severity alerts on this PR (regex use in
auth_checks.py:is_model_allowed_by_pattern, a file path build incontent_filter.py); both are byte-for-byte identical to the pre-sync code (confirmed via diff), already have mitigations (admin-only regex source,_resolve_category_file_pathdirectory jail), and match CodeQL's own stated caveat about large diffs misattributing pre-existing code as new. Not blocking; left for normal alert-bridge triage post-merge.