Add source-diverse AI rerank candidate pool

[Mbeaulne]

Description

The AI rerank candidate pool now uses a source-diverse selection strategy instead of relying purely on lexical hits. Previously, the candidate pool was capped at 50 results drawn entirely from lexical matches, falling back to a plain alphabetical slice when no matches were found. This meant components from underrepresented sources (e.g. user uploads) could be crowded out entirely when a query produced many strong lexical hits from a single source.

The new approach builds the candidate pool in three layers:

1. Up to 60 of the strongest lexical hits for the query
2. An evenly-sampled set of up to 8 candidates per source (source-diverse browse)
3. An evenly-sampled alphabetical slice of the full index to fill remaining slots up to the new cap of 80

The rerank base is now always set to aiCandidateMatches rather than switching between lexical and AI candidate lists depending on whether lexical results existed.

Related Issue and Pull requests

Type of Change

• Bug fix
• New feature
• Improvement
• Cleanup/Refactor
• Breaking change
• Documentation update

Checklist

• I have tested this does not break current pipelines / runs functionality
• I have tested the changes on staging

Screenshots (if applicable)

Test Instructions

1. Open the component search panel in the editor.
2. Enter a query that matches many components from a single source (e.g. a library with 100+ entries).
3. Click the AI rerank button and verify that components from other sources (e.g. user-uploaded files) still appear in the ranked results.
4. Verify that the total candidate count sent to the reranker does not exceed 80.
5. Run the unit tests in componentSearchV2Logic.test.ts to confirm the new source-diversity test passes.

Additional Comments

The new sampleEvenly helper picks items at uniform intervals so that the browse sample is representative of the full sorted list rather than just the top entries.

[github-actions]

🎩 Preview

A preview build has been created at: 06-18-build_broader_ai_candidate_pools_for_component_search/88f3546

[Mbeaulne]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• Debounce component search input #2433
• add client-side embeddings cached in IndexedDB #2432
• Add search quality expectation tests for lexical search #2431
• Add deep AI search to rerank all components in selected sources #2430
• Add source-diverse AI rerank candidate pool #2429 👈 (View in Graphite)
• Add negative constraint parsing to lexical search #2428
• Add fuzzy/typo-tolerant matching for component name and I/O fields #2427
• Improve component search scoring relevance #2426
• Add synonym expansion to component lexical search #2425
• Normalize component search tokens for better matching #2424
• Expand component search indexing fields #2423
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Mbeaulne]

🤖 This is an AI-generated code review comment.

[MEDIUM] The source-diversity layer (buildSourceDiverseBrowseMatches) and this alphabetical-fill layer run unconditionally, padding the AI candidate pool toward the cap (AI_CANDIDATE_LIMIT = 80) with alphabetically-early, lexically-irrelevant components even when lexical search already returned a strong source-spanning set.

This is not a correctness bug — lexical hits are preserved first (appended before the fill layers), and RERANK_EXCLUSION_THRESHOLD keeps junk from being badged. But it sends more low-signal candidates to the billed reranker on every rerank (cost/latency), and can surface irrelevant items in the unbadged tail.

Optional fix: only run the fill layer when the pool is under a smaller floor, or skip the alphabetical fill when lexical + diversity already produced a source-diverse set. Worth confirming reranker cost at 80 vs 50 candidates is acceptable.