From 7fa55b2531576990e4ce4f5ee8822d4207ab0e77 Mon Sep 17 00:00:00 2001 From: M N V Sai Krishna Date: Wed, 20 May 2026 09:22:16 -0700 Subject: [PATCH] Expand TrueFoundry skill workflows --- skills/_shared/references/api-endpoints.md | 5 + skills/gateway/SKILL.md | 11 +- skills/gateway/references/monitoring.md | 107 ++++++++++++++++-- skills/platform/SKILL.md | 25 +++- skills/platform/references/access-control.md | 61 ++++++++-- .../platform/references/secrets-and-tokens.md | 54 ++++++++- skills/prompts/SKILL.md | 37 ++++++ skills/skills-registry/SKILL.md | 22 ++++ 8 files changed, 301 insertions(+), 21 deletions(-) diff --git a/skills/_shared/references/api-endpoints.md b/skills/_shared/references/api-endpoints.md index fba7ad3..639c4a0 100644 --- a/skills/_shared/references/api-endpoints.md +++ b/skills/_shared/references/api-endpoints.md @@ -349,6 +349,7 @@ $TFY_API_SH GET /api/svc/v1/workspaces | GET | `/api/svc/v1/teams` | List teams | | POST | `/api/svc/v1/teams` | Create a team | | GET | `/api/svc/v1/teams/{id}` | Get team | +| POST | `/api/svc/v1/teams/{id}/members` | Add member to team | | DELETE | `/api/svc/v1/teams/{id}` | Delete team | --- @@ -361,6 +362,8 @@ $TFY_API_SH GET /api/svc/v1/workspaces | GET | `/api/svc/v1/users/{id}` | Get user | | POST | `/api/svc/v1/users/invite` | Invite user | +Invite and access grants are separate: invite with `/users/invite`, then add collaborators to resources after confirming role/resource. + --- ## Personal Access Tokens @@ -384,6 +387,8 @@ $TFY_API_SH GET /api/svc/v1/workspaces | POST | `/api/svc/v1/virtual-accounts/{id}/regenerate-token` | Regenerate token | | DELETE | `/api/svc/v1/virtual-accounts/{id}` | Delete virtual account | +Service accounts are valid collaborator subjects as `serviceaccount:name`, but this reference does not currently include a verified service-account creation endpoint. + --- ## Agents diff --git a/skills/gateway/SKILL.md b/skills/gateway/SKILL.md index 4a422e3..e38286e 100644 --- a/skills/gateway/SKILL.md +++ b/skills/gateway/SKILL.md @@ -164,7 +164,7 @@ Supported providers reference: [references/guardrail-providers.md](references/gu ## AI Monitoring -Query gateway request traces via the spans API. Requires either `tracingProjectFqn` or `dataRoutingDestination` (suggest `"default"` as starting point). +Query gateway request traces via the spans API and aggregate usage via the metrics API. Requires either `tracingProjectFqn` or `dataRoutingDestination` for trace queries; suggest `"default"` as a starting point when the user does not know the destination. ### Recent Requests @@ -183,6 +183,13 @@ For all monitoring use cases (cost analysis, errors, model usage, user filtering ### Aggregated Metrics +Use this path for aggregate questions such as: + +- "Show cost incurred for the last 3 months." +- "Break cost down by model, user, team, or virtual account." +- "Show total tokens and latency by model." +- "Which virtual account generated the most cost?" + ```bash $TFY_API_SH POST /api/svc/v1/llm-gateway/metrics/query '{ "startTs": "...", "endTs": "...", @@ -193,6 +200,8 @@ $TFY_API_SH POST /api/svc/v1/llm-gateway/metrics/query '{ }' ``` +When answering a time-range question, calculate exact `startTs` and `endTs`, state the range used, and present totals in a compact table. If the user asks for monthly breakdowns, run one query per month unless the API exposes a time-bucket field. + --- ## Generating Manifests diff --git a/skills/gateway/references/monitoring.md b/skills/gateway/references/monitoring.md index 2f3c124..1867b89 100644 --- a/skills/gateway/references/monitoring.md +++ b/skills/gateway/references/monitoring.md @@ -1,6 +1,6 @@ -# AI Monitoring & Traces +# AI Monitoring & Metrics -Query gateway request traces, costs, latency, errors, and token usage via the spans query API. +Query gateway request traces, costs, latency, errors, and token usage. Use the spans API for request-level investigations and the metrics API for aggregate questions such as "show cost for the last 3 months." ### Required Parameter @@ -28,6 +28,38 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ }' ``` +### Metrics API + +**Endpoint:** `POST /api/svc/v1/llm-gateway/metrics/query` + +Use this API for aggregate gateway questions. Calculate `startTs` and `endTs` from the user's requested time range. If the user says "last 3 months", use the exact three-month window ending now unless they ask for calendar months. + +```bash +$TFY_API_SH POST '/api/svc/v1/llm-gateway/metrics/query' '{ + "startTs": "2026-02-20T00:00:00.000Z", + "endTs": "2026-05-20T00:00:00.000Z", + "datasource": "modelMetrics", + "type": "distribution", + "aggregations": [ + {"type": "count", "column": "costInUSD"}, + {"type": "sum", "column": "costInUSD"}, + {"type": "sum", "column": "inputTokens"}, + {"type": "sum", "column": "outputTokens"}, + {"type": "p50", "column": "latencyMs"}, + {"type": "p90", "column": "latencyMs"} + ], + "groupBy": ["modelName"] +}' +``` + +Available aggregation columns: `costInUSD`, `inputTokens`, `outputTokens`, `latencyMs`, `interTokenLatencyMs`, `timeToFirstTokenMs`, `timePerOutputTokenLatencyMs` + +Aggregation types: `count`, `sum`, `p50`, `p75`, `p90`, `p99` + +Group-by dimensions: `modelName`, `userEmail`, `virtualaccount`, `team`, `virtualModel`, `errorCode`, `requestType`, `providerAccountType`, `providerModelName`, `metadata.` + +For calendar-month cost breakdowns, run one metrics query per month. Do not invent a month group-by unless the API response or product docs expose one. + ### Common Monitoring Use Cases #### 1. Show Recent Requests @@ -41,7 +73,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ }' ``` -#### 2. Cost Analysis (LLM Spans) +#### 2. Cost Analysis (Request-Level LLM Spans) Filter for LLM spans and extract cost attributes: @@ -50,7 +82,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "dataRoutingDestination": "default", "filters": [ - {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "LLM"} + {"spanAttributeKey": "tfy.span_type", "operator": "EQUAL", "value": "LLM"} ], "limit": 200, "sortDirection": "desc" @@ -69,7 +101,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "dataRoutingDestination": "default", "filters": [ - {"spanFieldName": "statusCode", "operator": "eq", "value": "ERROR"} + {"spanFieldName": "statusCode", "operator": "EQUAL", "value": "ERROR"} ], "limit": 50, "sortDirection": "desc" @@ -85,7 +117,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "dataRoutingDestination": "default", "filters": [ - {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "LLM"} + {"spanAttributeKey": "tfy.span_type", "operator": "EQUAL", "value": "LLM"} ], "limit": 200, "sortDirection": "desc" @@ -125,7 +157,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "dataRoutingDestination": "default", "filters": [ - {"spanAttributeKey": "tfy.span_type", "operator": "eq", "value": "MCP"} + {"spanAttributeKey": "tfy.span_type", "operator": "EQUAL", "value": "MCP"} ], "limit": 50, "sortDirection": "desc" @@ -153,7 +185,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "dataRoutingDestination": "default", "filters": [ - {"spanFieldName": "spanName", "operator": "contains", "value": "completions"} + {"spanFieldName": "spanName", "operator": "STRING_CONTAINS", "value": "completions"} ], "limit": 50, "sortDirection": "desc" @@ -167,7 +199,7 @@ $TFY_API_SH POST '/api/svc/v1/spans/query' '{ "startTime": "2026-03-26T00:00:00.000Z", "dataRoutingDestination": "default", "filters": [ - {"gatewayRequestMetadataKey": "tfy_gateway_region", "operator": "eq", "value": "US"} + {"gatewayRequestMetadataKey": "tfy_gateway_region", "operator": "EQUAL", "value": "US"} ], "limit": 50, "sortDirection": "desc" @@ -222,7 +254,62 @@ Custom metadata keys set via `X-TFY-LOGGING-CONFIG` headers. #### Filter Operators -`eq`, `neq`, `contains`, `not_contains`, `starts_with`, `ends_with` +`EQUAL`, `IN`, `NOT_IN`, `STRING_CONTAINS`, `STRING_STARTS_WITH`, `STRING_ENDS_WITH`, `GREATER_THAN`, `LESS_THAN` + +### Aggregate Cost Recipes + +#### Total Cost for a Time Range + +```bash +$TFY_API_SH POST '/api/svc/v1/llm-gateway/metrics/query' '{ + "startTs": "2026-02-20T00:00:00.000Z", + "endTs": "2026-05-20T00:00:00.000Z", + "datasource": "modelMetrics", + "type": "distribution", + "aggregations": [ + {"type": "count", "column": "costInUSD"}, + {"type": "sum", "column": "costInUSD"}, + {"type": "sum", "column": "inputTokens"}, + {"type": "sum", "column": "outputTokens"} + ] +}' +``` + +#### Cost by Model + +Use the same body and add: + +```json +"groupBy": ["modelName"] +``` + +#### Cost by User, Team, or Virtual Account + +Use one of these groupings: + +```json +"groupBy": ["userEmail"] +"groupBy": ["team"] +"groupBy": ["virtualaccount"] +``` + +#### Error or Latency Summary + +```bash +$TFY_API_SH POST '/api/svc/v1/llm-gateway/metrics/query' '{ + "startTs": "2026-02-20T00:00:00.000Z", + "endTs": "2026-05-20T00:00:00.000Z", + "datasource": "modelMetrics", + "type": "distribution", + "aggregations": [ + {"type": "count", "column": "costInUSD"}, + {"type": "p50", "column": "latencyMs"}, + {"type": "p90", "column": "latencyMs"}, + {"type": "p99", "column": "latencyMs"} + ], + "groupBy": ["errorCode"] +}' +``` ### Response Structure diff --git a/skills/platform/SKILL.md b/skills/platform/SKILL.md index 05aa6ef..276690b 100644 --- a/skills/platform/SKILL.md +++ b/skills/platform/SKILL.md @@ -19,9 +19,10 @@ Platform setup and access management: verify credentials, discover workspaces an - Verify TrueFoundry credentials and connectivity (preflight check) - List clusters, workspaces, GPU types, or base domains - Find workspace FQNs for deployment targets +- Invite users by email - List or create roles, teams, or collaborators - Manage secret groups and secret references (`tfy-secret://`) -- List and create personal access tokens (PATs) +- List and create personal access tokens (PATs) and virtual accounts (VATs) ## When NOT to Use @@ -96,6 +97,7 @@ Manage roles, teams, and collaborators. For full API calls, tool call syntax, pr | Action | API Call | |--------|---------| +| Invite user | `$TFY_API_SH POST /api/svc/v1/users/invite '{...}'` | | List roles | `$TFY_API_SH GET /api/svc/v1/roles` | | List teams | `$TFY_API_SH GET /api/svc/v1/teams` | | List collaborators | `$TFY_API_SH GET '/api/svc/v1/collaborators?resourceType=TYPE&resourceId=ID'` | @@ -109,6 +111,19 @@ Subject format: `user:email`, `team:slug`, `serviceaccount:name`, `virtualaccoun Destructive operations (delete roles, teams, collaborators): direct to dashboard. +### Invite Users + +Use this for "invite new users by email" requests. + +1. Collect email addresses. +2. Confirm target tenant. +3. Ask whether to only invite or also grant access to a resource. +4. If granting access, list roles/resources first and ask for explicit confirmation. + +```bash +$TFY_API_SH POST /api/svc/v1/users/invite '{"emails":["alice@example.com"]}' +``` + ## Secrets Manage secret groups and `tfy-secret://` references. Never ask user to paste secret values in chat. @@ -129,12 +144,16 @@ For full create/update flows, API patterns, and security policies, see [referenc ## Access Tokens -List and create PATs. Token values are shown only once at creation. +List and create PATs and manage virtual accounts/VATs. Token values are shown only once at creation or retrieval/regeneration time. | Action | API Call | |--------|---------| | List PATs | `$TFY_API_SH GET /api/svc/v1/personal-access-tokens` | | Create PAT | `$TFY_API_SH POST /api/svc/v1/personal-access-tokens '{"name":"..."}'` | +| List virtual accounts | `$TFY_API_SH GET /api/svc/v1/virtual-accounts` | +| Create/update virtual account | `$TFY_API_SH POST /api/svc/v1/virtual-accounts '{...}'` | +| Get VAT token | `$TFY_API_SH GET /api/svc/v1/virtual-accounts/ID/token` | +| Regenerate VAT token | `$TFY_API_SH POST /api/svc/v1/virtual-accounts/ID/regenerate-token` | > **Security:** Never repeat, store, or log token values. Show masked preview by default; full value only on explicit confirmation. @@ -142,6 +161,8 @@ For full token display policy and security rules, see [references/secrets-and-to Deletion: direct to dashboard. +Service accounts: this skill can grant roles to existing service account subjects using `serviceaccount:name`. Do not claim service-account creation is supported until the create endpoint or dashboard flow is verified. + diff --git a/skills/platform/references/access-control.md b/skills/platform/references/access-control.md index 464b36b..9326162 100644 --- a/skills/platform/references/access-control.md +++ b/skills/platform/references/access-control.md @@ -4,6 +4,33 @@ Manage roles, teams, and collaborators. Roles define permission sets, teams grou When using direct API, set `TFY_API_SH` to the full path of this skill's `scripts/tfy-api.sh`. See `references/tfy-api-setup.md` for paths per agent. +### Users + +Use user workflows for listing existing users and inviting new users by email. Inviting a user and granting access are separate steps. + +#### List Users + +```bash +$TFY_API_SH GET /api/svc/v1/users +``` + +#### Invite Users by Email + +Before inviting, collect the email addresses and confirm the target tenant. If the user also wants workspace or resource access, complete the invite first, then use the collaborator workflow after confirming the role and resource. + +```bash +$TFY_API_SH POST /api/svc/v1/users/invite '{"emails":["alice@example.com","bob@example.com"]}' +``` + +Present: + +```text +Invites: +| Email | Status | +|-------|--------| +| alice@example.com | invited | +``` + ### Roles Roles are named permission sets scoped to a resource type. Built-in roles vary by resource type (for example, `workspace-admin`, `workspace-member`). @@ -217,22 +244,29 @@ To remove this collaborator, open the TrueFoundry dashboard, go to the resource #### Grant a User Access to a Workspace -1. List roles to find the appropriate role ID (e.g., `workspace-admin` or `workspace-member`) -2. Add the user as a collaborator on the workspace with that role +1. Invite the user first if they do not already exist in the tenant. +2. List roles to find the appropriate role ID (e.g., `workspace-admin` or `workspace-member`). +3. Confirm the email, workspace, and role with the user. +4. Add the user as a collaborator on the workspace with that role. ```bash +# Optional: invite the user +$TFY_API_SH POST /api/svc/v1/users/invite '{"emails":["alice@example.com"]}' + # 1. Find the role ID $TFY_API_SH GET /api/svc/v1/roles -# 2. Add collaborator +# 2. Add collaborator after confirmation $TFY_API_SH POST /api/svc/v1/collaborators '{"resourceType":"workspace","resourceId":"WORKSPACE_ID","subject":"user:alice@company.com","roleId":"ROLE_ID"}' ``` #### Create a Team and Grant Access -1. Create the team -2. Add members to the team -3. Add the team as a collaborator on the target resource +1. Create the team. +2. Add members to the team. +3. List roles and select the role ID. +4. Confirm the team, resource, and role with the user. +5. Add the team as a collaborator on the target resource. ```bash # 1. Create team @@ -241,10 +275,23 @@ $TFY_API_SH POST /api/svc/v1/teams '{"name":"ml-engineers","description":"ML eng # 2. Add members (use team ID from response) $TFY_API_SH POST /api/svc/v1/teams/TEAM_ID/members '{"subject":"user:alice@company.com","role":"member"}' -# 3. Grant team access to a workspace +# 3. Grant team access to a workspace after confirmation $TFY_API_SH POST /api/svc/v1/collaborators '{"resourceType":"workspace","resourceId":"WORKSPACE_ID","subject":"team:ml-engineers","roleId":"ROLE_ID"}' ``` +#### Create a Custom Role and Grant It to a Team + +1. Define the role name, resource type, and permission list. +2. Create the role. +3. Create or select the team. +4. Confirm the final subject/resource/role binding. +5. Add the team as a collaborator. + +```bash +$TFY_API_SH POST /api/svc/v1/roles '{"name":"custom-deployer","displayName":"Custom Deployer","description":"Can deploy apps","resourceType":"workspace","permissions":["deploy:create","deploy:read"]}' +$TFY_API_SH POST /api/svc/v1/collaborators '{"resourceType":"workspace","resourceId":"WORKSPACE_ID","subject":"team:ml-engineers","roleId":"ROLE_ID_FROM_RESPONSE"}' +``` + #### Audit Access on a Resource List all collaborators to see who has access and with what role: diff --git a/skills/platform/references/secrets-and-tokens.md b/skills/platform/references/secrets-and-tokens.md index 270161b..7b39219 100644 --- a/skills/platform/references/secrets-and-tokens.md +++ b/skills/platform/references/secrets-and-tokens.md @@ -1,6 +1,6 @@ # Secrets & Access Tokens -Manage secret groups, secret references (tfy-secret://), and personal access tokens (PATs). +Manage secret groups, secret references (tfy-secret://), personal access tokens (PATs), and virtual accounts/VATs. ## Secrets @@ -164,3 +164,55 @@ Never commit tokens to Git or share them in plain text. ### Delete Access Token Do not delete PATs from the agent. If deletion is requested, direct the user to the TrueFoundry dashboard. + +## Virtual Accounts and VATs + +Use virtual accounts when the user needs a machine or application identity with controlled model access. VAT token values are credentials and follow the same one-time display policy as PATs. + +### List Virtual Accounts + +```bash +$TFY_API_SH GET /api/svc/v1/virtual-accounts +``` + +Present: + +```text +Virtual Accounts: +| Name | ID | Models/Scopes | Created At | +|------|----|---------------|------------| +``` + +### Create or Update Virtual Account + +Before creating or updating, collect: + +- Name +- Intended owner/application +- Allowed models or resources, if the API/dashboard requires them +- Expiration or rotation expectations, if applicable + +Show the final payload without token values and ask for explicit confirmation. + +```bash +$TFY_API_SH POST /api/svc/v1/virtual-accounts '{"name":"ci-gateway-client"}' +``` + +### Retrieve or Regenerate VAT Token + +Only retrieve or regenerate when the user explicitly asks and is ready to handle the credential. + +```bash +$TFY_API_SH GET /api/svc/v1/virtual-accounts/VIRTUAL_ACCOUNT_ID/token +$TFY_API_SH POST /api/svc/v1/virtual-accounts/VIRTUAL_ACCOUNT_ID/regenerate-token +``` + +Show only a masked preview by default. Reveal the full token once only after explicit confirmation. + +### Delete Virtual Account + +Do not delete virtual accounts from the agent. If deletion is requested, direct the user to the dashboard. + +## Service Accounts + +Existing service accounts can be used as collaborator subjects with `serviceaccount:name`. This skill does not currently document a verified service-account creation endpoint. Do not claim service-account creation is supported until the dashboard flow or API endpoint is verified. diff --git a/skills/prompts/SKILL.md b/skills/prompts/SKILL.md index 4028f3d..c2fd026 100644 --- a/skills/prompts/SKILL.md +++ b/skills/prompts/SKILL.md @@ -45,6 +45,17 @@ TrueFoundry Dashboard -> AI Gateway -> Prompt Management / Prompt Registry Use the UI when the user wants to visually create, compare, or test prompts. +## Capability Matrix + +| User intent | Supported path | Notes | +|-------------|----------------|-------| +| List prompts | API or UI | Present prompt name, ID, latest version, and tags. | +| Inspect prompt versions | API, SDK, or UI | Include version ID/FQN and tags. | +| Create a prompt | SDK or UI | Review messages/settings before creation. | +| Update a prompt | SDK or UI | Treat as a new version; do not overwrite silently. | +| Tag a version | SDK or UI | Confirm before moving stable tags like `production`. | +| Get prompt FQN | SDK or UI | Use for Gateway and Agent references. | + ## List Prompts API fallback: @@ -76,6 +87,30 @@ Collect: Show the final prompt content and settings before creating or updating. +Review format before create/update: + +```text +Prompt Change +| Field | Value | +|-------|-------| +| Name | my-prompt | +| Repository | ml-repo-fqn | +| Change | create prompt / create new version | +| Variables | user_input | +| Model | model-catalog:openai:gpt-4 | +| Tags | none / production | +``` + +Then show the messages in order: + +```text +System: +... + +User: +... +``` + SDK shape: ```python @@ -101,6 +136,8 @@ client.prompts.create_or_update( Prompt edits create versions. Use tags such as `production` or `staging` to provide stable references. +Before applying a tag, show the current tag target if known and ask for explicit confirmation when the tag is stable or production-facing. + SDK shape: ```python diff --git a/skills/skills-registry/SKILL.md b/skills/skills-registry/SKILL.md index 35c1146..b30fe0b 100644 --- a/skills/skills-registry/SKILL.md +++ b/skills/skills-registry/SKILL.md @@ -35,6 +35,17 @@ Use this skill when the user wants to: - `tfy apply`: declarative/GitOps flow when an `agent-skill` manifest is available. - Multi-file bundles: use the dashboard/registry workflow or exact product-provided command only after verifying it exists in `tfy --help`. +## Capability Matrix + +| User intent | Supported path | Notes | +|-------------|----------------|-------| +| Create a single-file skill | UI | Use the dashboard editor for `SKILL.md`. | +| Publish a reviewed `SKILL.md` | UI or `tfy apply` | Use `tfy apply` only when a valid `agent-skill` manifest is available. | +| Upload a multi-file bundle | Dashboard/registry workflow | Verify any generated command before running it. | +| Download a skill | Skill detail usage/export flow | Inspect downloaded files before editing. | +| Create a new version | Skill detail UI or apply workflow | Show changed content before publishing. | +| Attach a skill to an agent | Agent Playground | Use `truefoundry-agents` for agent-side changes. | + ## UI Path Use this path for single-file skills: @@ -78,6 +89,15 @@ Show the user the file list and ask for confirmation before publishing. Do not run `tfy upload skill`; it is not present in `tfy 0.13.12`. If the product UI shows a generated command in the future, verify it with `tfy --help` before using it. +Before publishing a multi-file bundle, review: + +- `SKILL.md` frontmatter has a focused `name`, action-oriented `description`, compatibility, and minimal `allowed-tools`. +- Instructions are sequential and specific. +- Large reference material lives in `references/` and is linked from `SKILL.md`. +- Scripts live in `scripts/` and are preferred over long pasted command blocks. +- Secrets, tenant URLs, and raw tokens are not embedded in examples. +- Destructive actions are dashboard-only or require explicit user confirmation. + ## Declarative Apply Use `tfy apply` for GitOps-style skill publishing. Before final apply, show: @@ -89,6 +109,8 @@ Use `tfy apply` for GitOps-style skill publishing. Before final apply, show: Then ask for explicit confirmation. +If the manifest schema is unknown, stop at a reviewed draft and ask the user to publish through the UI or provide the product-generated manifest. Do not invent resource fields. + ## Download Existing Skill Use the Usage tab in the Skill Detail page or the dashboard export/download flow.