Skip to content

Add Cache-Aware "Token + Cache" Usage Limit Type - #3543

Draft
Kaveh-Vakili wants to merge 1 commit into
devfrom
cache-token-weighted-limits
Draft

Add Cache-Aware "Token + Cache" Usage Limit Type#3543
Kaveh-Vakili wants to merge 1 commit into
devfrom
cache-token-weighted-limits

Conversation

@Kaveh-Vakili

Copy link
Copy Markdown
Contributor

Description

Adds a "Token + Cache" usage limit type so admins can restrict a member's model usage by a token budget that also counts cache read/creation tokens, weighted at a per-model percentage, instead of only plain prompt+completion tokens. Includes a new Cache read/write weight (%) config in Model Settings, backed by the backend's new CACHE_READ_WEIGHT/CACHE_WRITE_WEIGHT model metadata fields (see companion backend PR).

Changes Made

  1. engine-metadata-display.tsx: add cacheReadWeight/cacheWriteWeight to ModelMetadata/ModelSettingsValues, normalize on fetch, and render read-only in ModelMetadataFields

  2. engine-model-settings.tsx: editable Cache read weight / Cache write weight inputs (0–1000%, blank = 100% default) next to Context window/Max output tokens, validated and included in the UpdateModelMetadata save payload

  3. members-table.tsx (Edit Member modal), add-members.tsx / model-restriction-fields.tsx (Add Members), member-profile-form.tsx, user-add-overlay.tsx: add "Token + Cache" to the Usage Limit Type selector and extend the Max Tokens field's visibility/payload conditionals to include it (previously only "token" showed/sent Max Tokens)

  4. members-list.tsx: recognize token_cache in the restriction label and Limit Value column (previously fell through to "—"), and carry max_tokens through on inline permission-only edits for this mode
    libs/i18n/.../en/shared/members.json: add the tokenCache label ("Token + Cache")

How to Test

  1. Open a model's Settings page, set Cache read weight / Cache write weight (e.g. 10 / 200), Save, reload — values persist. Try a value over 1000 — should be rejected.
  2. In that model's Access Control, open Edit Member for a user and confirm "Token + Cache" now appears in the Usage Limit Type dropdown, and selecting it reveals the Max Tokens field (same as plain "Token").
  3. Set it to Token + Cache with a Max Tokens value, Save — reopen the modal and confirm it re-loads correctly, and check the members table shows "Token + Cache" in the restriction column with the correct Limit Value.
  4. Repeat step 2–3 via the Add Members overlay's Model Limit Restrictions section, and via the org-level member profile form / Add User overlay — all four surfaces should offer and persist the same option consistently.
  5. Confirm existing plain Token and Compute time flows are unaffected (regression check).

Notes

  1. This is UI/config plumbing only — actual enforcement (throwing when a member exceeds a Token + Cache limit) lives in the backend PR. This PR alone lets you configure and persist the setting; pair it with the backend PR to see it enforced live.

  2. The Usage Limit Type dropdown is duplicated across 4 components (members-table.tsx, add-members.tsx/model-restriction-fields.tsx, member-profile-form.tsx, user-add-overlay.tsx) with independently-maintained label maps — already a pre-existing drift risk (e.g. label capitalization already differed slightly between two of them before this PR). Worth consolidating into one shared constant/component in a follow-up rather than continuing to add new options in four places.

Adds a Cache read/write weight (%) pair to the Model Settings card,
backed by the new CACHE_READ_WEIGHT/CACHE_WRITE_WEIGHT model metadata
fields, and a new "Token + Cache" option to every surface that offers
the Usage Limit Type selector so members can be restricted with a
cache-aware token budget instead of the plain Token limit.

- engine-metadata-display.tsx: add cacheReadWeight/cacheWriteWeight to
  ModelMetadata/ModelSettingsValues, normalize, and render read-only
- engine-model-settings.tsx: editable Cache read/write weight inputs
  (0-1000%, blank = 100% default) alongside Context window/Max output
  tokens, wired into the UpdateModelMetadata save payload
- members-table.tsx, add-members.tsx, model-restriction-fields.tsx,
  member-profile-form.tsx, user-add-overlay.tsx: add the "Token +
  Cache" option to the Usage Limit Type selector and extend the
  Max Tokens field's visibility/payload conditionals to include it
- members-list.tsx: recognize token_cache in the restriction label
  and Limit Value column, and carry max_tokens through on inline
  permission-only edits
- members.json (en): add the tokenCache i18n label
@snyk-io

snyk-io Bot commented Aug 12, 2026

Copy link
Copy Markdown

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant