Skip to content

feat: LLM-based dynamic skill selection#114

Closed
kulvirgit wants to merge 1 commit intomainfrom
feat/llm-skill-selector
Closed

feat: LLM-based dynamic skill selection#114
kulvirgit wants to merge 1 commit intomainfrom
feat/llm-skill-selector

Conversation

@kulvirgit
Copy link
Collaborator

Summary

  • Replace brittle tag-based skill filtering (partitionByFingerprint + rescueByMessage) with a single Haiku LLM call that semantically selects relevant skills based on user message and project fingerprint
  • Remove tags field from Skill.Info schema and all 51 SKILL.md frontmatter files — no longer needed
  • Add 50+ skills (airflow, dbt, databricks, snowflake, etc.) and 2 new dbt-labs migration skills
  • Add MessageContext side-channel and Fingerprint detection for environment-aware context
  • Gated behind experimental.dynamic_skills config flag — disabled by default

How it works

selectSkillsWithLLM() in skill-selector.ts:

  1. Gets the small model via Provider.getSmallModel() (Haiku 4.5 priority)
  2. Sends skill names + descriptions + user message to generateObject() with zod schema
  3. Filters skills by the returned names
  4. 3s timeout via Promise.race — never blocks the main model
  5. Returns ALL skills on any failure (timeout, error, no model, zero selection)

Test plan

  • bun test test/altimate/skill-filtering.test.ts — 20 tests covering all fallback paths, model verification, prompt content
  • bun test test/tool/skill.test.ts — existing tests pass
  • bunx tsc --noEmit — no type errors
  • Live-tested with real Haiku API: "dbt model" → selects dbt-modeling + sql-optimization, "deploy to k8s" → selects kubernetes-deploy, unrelated message → fallback to all skills

…tion

- Add `skill-selector.ts` with `selectSkillsWithLLM()` — uses small model
  (Haiku 4.5 via `Provider.getSmallModel`) + `generateObject` to semantically
  select relevant skills based on user message and project fingerprint
- Remove `partitionByFingerprint()` and `rescueByMessage()` from `tool/skill.ts`
- Remove `tags` field from `Skill.Info` schema and all 51 SKILL.md files
- Add 2 new dbt-labs skills: `migrating-dbt-core-to-fusion`, `migrating-dbt-project-across-platforms`
- Rewrite tests with `SkillSelectorDeps` DI interface for model + generate mocking
- Graceful fallback: returns all skills on timeout (3s), error, no model, or zero selection
@github-actions
Copy link

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

cwd: string
}

let cached: Result | undefined
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: A race condition exists in the Fingerprint module's module-level cache, allowing concurrent sessions with different directories to overwrite and read each other's cached data.
Severity: MEDIUM

Suggested Fix

Refactor the Fingerprint cache to be session-aware or directory-aware instead of using a global singleton. This could be achieved by using a Map keyed by the directory cwd or by storing the fingerprint data within the session's context, similar to how other session-specific data is managed.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: packages/opencode/src/altimate/fingerprint/index.ts#L15

Potential issue: The `Fingerprint` module uses a single module-level variable `cached`
to store fingerprint data. In a concurrent environment with multiple sessions operating
on different directories (`cwd`), a race condition can occur. If Session A detects a
fingerprint for `/project-a` and then yields, Session B can overwrite the cache with
data for `/project-b`. When Session A resumes and calls `Fingerprint.get()`, it will
incorrectly receive the fingerprint data from Session B. While the impact is currently
limited to incorrect skill selection, which has a graceful fallback, this represents a
state corruption issue across sessions.

Did we get this right? 👍 / 👎 to inform future reviews.

@github-actions
Copy link

This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window.

Feel free to open a new pull request that follows our guidelines.

@github-actions github-actions bot closed this Mar 13, 2026
@anandgupta42 anandgupta42 deleted the feat/llm-skill-selector branch March 17, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant