diff --git a/CHANGELOG.md b/CHANGELOG.md index 283a4c2..14d6617 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,11 @@ version tags such as `v0.3.0`. ## Unreleased -- No unreleased changes. +- Add one-fetch GitHub repository collection preview/apply with explicit child + selection, stale ref/tree validation, and per-skill managed provenance. This + Phase C change targets v0.9.0 and remains unreleased pending v0.9.0 release qualification; collection-level update/rollback remains planned. +- Update collection roadmap/status wording to distinguish the shipped v0.8.0 + local A+B work from the v0.9.0-targeted, unreleased Phase C implementation. ## 0.8.0 diff --git a/Cargo.lock b/Cargo.lock index 5654740..dad2dc1 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3217,6 +3217,7 @@ dependencies = [ "serde", "serde_json", "skillbox-core", + "skillbox-git", "skillbox-github", ] diff --git a/README.md b/README.md index 11b28b2..e78d55b 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ English | [简体中文](README.zh-CN.md) SkillBox is a local-first macOS desktop app with a Rust core and CLI for managing `SKILL.md`-based skill and capability packages without treating any supported agent runtime as the source of truth. -Current release: `v0.8.0`. SkillBox is useful today for local skill management, but it is still early software. Keep backups of important skills, and review each filesystem change before applying it. +Current release: `v0.8.0`. SkillBox is useful today for local skill management, but it is still early software. Keep backups of important skills, and review each filesystem change before applying it. GitHub multi-skill collection preview/apply is implemented for the upcoming v0.9.0 qualification and remains unreleased. ## Promo Video @@ -32,7 +32,7 @@ A 30-second overview of SkillBox: runtime-aware workspaces, review-before-write - **Reviewed Git changes in both directions.** Inspect local user-skill diffs before commit/push. The shipped v0.7 line adds an explicit Check remote -> Review incoming changes -> Apply fast-forward flow for safe inbound updates; diverged history remains a normal Git conflict to resolve outside SkillBox. - **Evidence-aware Calls, references, and operation history.** Count locally confirmed executions plus defensible structured invocations as Calls, keep lower-signal history references separate, and explain coverage without storing full chat transcripts. - **Safe storage and deployment defaults.** Use ordered SQLite migrations, recovery backups, integrity checks, and ownership-checked symlinks instead of silently overwriting runtime content. -- **Git-backed local collections.** Import Review groups skills from the same local Git worktree into one repository card while keeping each child independently selectable, deployable, and usage-tracked. GitHub multi-skill fetch and collection-level update/rollback remain future work. +- **Git-backed local collections.** Import Review groups skills from the same local Git worktree into one repository card while keeping each child independently selectable, deployable, and usage-tracked. GitHub multi-skill preview/apply is implemented for v0.9.0 and remains unreleased pending v0.9.0 release qualification; collection-level update/rollback remains future work. - **Installed-source provenance.** Copied skills with valid v3 installer lockfile entries can appear under one normalized GitHub source collection even without a local Git worktree. This is display-only provenance: it does not invent a branch/HEAD or enable updates, and each child keeps the normal reviewed import path. - **Compatibility before deployment.** Rust-owned runtime profiles identify each workspace and report preserved frontmatter warnings or hard blockers before a confirmed symlink deployment. - **Signed macOS distribution.** Install a notarized DMG or Homebrew cask and apply signed app updates only after confirmation. diff --git a/README.zh-CN.md b/README.zh-CN.md index 6b0bfee..df89e87 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -16,7 +16,7 @@ SkillBox 是一个 local-first 的 macOS 桌面应用,带 Rust core/CLI,用来管理基于 `SKILL.md` 的 skill 与能力包,同时避免把任一受支持的 agent runtime 当作唯一真相源。 -当前版本:`v0.8.0`。SkillBox 现在已经可以用于本地 skill 管理,但仍是早期软件。重要 skills 请保留备份,并在应用每一次文件系统变更前先 review。 +当前版本:`v0.8.0`。SkillBox 现在已经可以用于本地 skill 管理,但仍是早期软件。重要 skills 请保留备份,并在应用每一次文件系统变更前先 review。GitHub 多 skill collection preview/apply 已按 v0.9.0 实现并等待 release qualification,尚未发布。 ## 宣传视频 @@ -32,7 +32,7 @@ SkillBox 是一个 local-first 的 macOS 桌面应用,带 Rust core/CLI,用 - **双向 Git 变更都先审查。** 本地 user-skill diff 会在 commit/push 前 review。已发布的 v0.7 增加显式的 Check remote -> Review incoming changes -> Apply fast-forward 入站流程;远端历史分叉仍在 SkillBox 外按正常 Git 冲突处理。 - **按证据分类的 Calls、引用与操作历史。** Calls 只统计本机 confirmed execution 与可辩护的 structured invocation,低信号 history references 单独展示,并且不保存完整聊天 transcript。 - **安全的存储与部署默认值。** 使用顺序 SQLite migrations、恢复备份、完整性检查和 ownership-checked symlink,不静默覆盖 runtime 内容。 -- **Git-backed 本地 Skill Collections。** Import Review 会把同一 Git worktree 中的 skills 聚合为一个仓库卡片,同时保留每个子 skill 的独立选择、部署和 usage。GitHub 多 skill fetch 与 collection 级更新/回滚仍是后续工作。 +- **Git-backed 本地 Skill Collections。** Import Review 会把同一 Git worktree 中的 skills 聚合为一个仓库卡片,同时保留每个子 skill 的独立选择、部署和 usage。GitHub 多 skill preview/apply 已按 v0.9.0 实现,尚未发布,待 v0.9.0 release qualification;collection 级更新/回滚仍是后续工作。 - **已安装来源 provenance。** 没有本地 Git worktree、但拥有有效 v3 installer lockfile 条目的复制 skill,也可以按规范化 GitHub source 聚合展示。这只是来源展示,不会伪造 branch/HEAD 或开放更新;每个 child 仍走原有的逐 skill review/import 流程。 - **部署前检查 compatibility。** Rust-owned runtime profiles 标识 workspace,并在确认 symlink 部署前报告会原样保留的 frontmatter warnings 或 hard blockers。 - **签名的 macOS 分发。** 可安装已公证 DMG 或 Homebrew cask,app 更新也只在用户确认后应用。 diff --git a/apps/desktop/src-tauri/src/lib.rs b/apps/desktop/src-tauri/src/lib.rs index 73d6dae..a7d383f 100644 --- a/apps/desktop/src-tauri/src/lib.rs +++ b/apps/desktop/src-tauri/src/lib.rs @@ -416,6 +416,36 @@ async fn apply_import_collection( .map_err(|error| format!("Collection import task failed: {error}"))? } +#[tauri::command] +async fn preview_github_skill_collection( + request: skillbox_core::PreviewGithubSkillCollectionRequest, +) -> Result { + tauri::async_runtime::spawn_blocking(move || { + let result = skillbox_core::preview_github_skill_collection_result( + request, + skillbox_core::default_managed_root(), + )?; + serde_json::to_value(result).map_err(|error| error.to_string()) + }) + .await + .map_err(|error| format!("GitHub collection preview task failed: {error}"))? +} + +#[tauri::command] +async fn apply_github_skill_collection( + request: skillbox_core::GithubSkillCollectionApplyRequest, +) -> Result { + tauri::async_runtime::spawn_blocking(move || { + let result = skillbox_core::apply_github_skill_collection( + request, + skillbox_core::default_managed_root(), + )?; + serde_json::to_value(result).map_err(|error| error.to_string()) + }) + .await + .map_err(|error| format!("GitHub collection apply task failed: {error}"))? +} + #[tauri::command] async fn import_candidates(items: Vec) -> Result { tauri::async_runtime::spawn_blocking(move || { @@ -1157,6 +1187,8 @@ pub fn run() { scan_workspace_import_candidates, list_skill_collections, apply_import_collection, + preview_github_skill_collection, + apply_github_skill_collection, import_candidates, list_import_records, revert_import, diff --git a/apps/desktop/src/App.import-candidates.test.js b/apps/desktop/src/App.import-candidates.test.js index cb73cae..37902cb 100644 --- a/apps/desktop/src/App.import-candidates.test.js +++ b/apps/desktop/src/App.import-candidates.test.js @@ -9,6 +9,7 @@ import { collectionSkillCountLabel, importCandidateGroupLocationCount, importCandidateGroupTabs, + normalizeGithubSkillCollectionPreviewResult, normalizeImportCollections, normalizeImportCandidateGroup, normalizeImportCandidateGroups, @@ -67,6 +68,7 @@ import { import { browserImportScanOptions, createImportScanRequestController, + createRemoteImportRequestController, importScanCommandArgs, importScanProgressDetail, importScanProgressLabel, @@ -135,6 +137,22 @@ test('closing and reopening Import Review isolates late scan A from active scan assert.equal(controller.begin(), 4); }); +test('remote preview controller ignores duplicate clicks and late close/reopen results', () => { + const controller = createRemoteImportRequestController(); + const first = controller.begin(); + assert.equal(first, 1); + assert.equal(controller.begin(), null); + controller.invalidate(); + const second = controller.begin(); + assert.equal(second, 3); + assert.equal(controller.isCurrent(first), false); + assert.equal(controller.isCurrent(second), true); + controller.finish(second); + assert.equal(controller.isCurrent(second), true); + controller.invalidate(); + assert.equal(controller.isCurrent(second), false); +}); + test('normalizes backend is_selected false without selecting importable candidate', () => { const candidate = normalizeImportCandidate({ name: 'system', @@ -415,6 +433,8 @@ test('normalizes Git-backed collection children and submits one selected child r const requests = selectedImportCollectionRequests(groups, collections); assert.deepEqual(requests, [{ collectionId: 'collection-demo', + sourceKind: 'git_worktree', + sourceUrl: '', worktreeRoot: '/Users/example/skills-repo', previewId: 'preview-demo', selections: [{ @@ -426,6 +446,83 @@ test('normalizes Git-backed collection children and submits one selected child r }]); }); +test('normalizes a GitHub collection without inventing a local worktree root', () => { + const collections = normalizeImportCollections([{ + id: 'github-collection-demo', + source_kind: 'github_remote', + source_url: 'https://github.com/acme/skills', + requested_reference: 'main', + reviewed_head_sha: '1234567890abcdef', + children: [{ + id: 'child-demo', + group_id: 'skill-demo', + variant_id: 'variant-demo', + name: 'demo', + relative_path: 'skills/demo', + import_status: 'importable', + selected_type: 'remote', + is_selected: true, + locations: [] + }] + }]); + const groups = normalizeImportCandidateGroups([{ + id: 'skill-demo', + name: 'demo', + selected_variant_id: 'variant-demo', + variants: [{ + id: 'variant-demo', + candidate: { name: 'demo', import_status: 'importable', is_selected: true }, + selected_type: 'remote', + locations: [] + }] + }]); + + assert.equal(collections[0].sourceKind, 'github_remote'); + assert.equal(collections[0].canonicalWorktreeRoot, ''); + assert.deepEqual(selectedImportCollectionRequests(groups, collections), [{ + collectionId: 'github-collection-demo', + sourceKind: 'github_remote', + sourceUrl: 'https://github.com/acme/skills', + worktreeRoot: '', + previewId: '', + selections: [{ + relativePath: 'skills/demo', + groupId: 'skill-demo', + variantId: 'variant-demo', + skillType: 'remote' + }] + }]); +}); + +test('routes structured GitHub collection preview outcomes without parsing human errors', () => { + const collection = { id: 'collection-demo' }; + assert.deepEqual( + normalizeGithubSkillCollectionPreviewResult({ + kind: 'collection', + preview: collection + }), + { kind: 'collection', preview: collection } + ); + assert.deepEqual( + normalizeGithubSkillCollectionPreviewResult({ + kind: 'single_skill', + message: 'Use the single-skill preview.' + }), + { kind: 'single_skill', message: 'Use the single-skill preview.' } + ); + assert.deepEqual( + normalizeGithubSkillCollectionPreviewResult({ + kind: 'explicit_reference_required', + message: 'Use /tree/.' + }), + { kind: 'explicit_reference_required', message: 'Use /tree/.' } + ); + assert.throws( + () => normalizeGithubSkillCollectionPreviewResult({ error: 'points to one skill' }), + /invalid result/ + ); +}); + test('installed-source collections stay on per-skill apply and suppress imported type review', () => { const collections = normalizeImportCollections([{ id: 'installed-source-dbs', diff --git a/apps/desktop/src/App.jsx b/apps/desktop/src/App.jsx index 55764bb..9f09db7 100644 --- a/apps/desktop/src/App.jsx +++ b/apps/desktop/src/App.jsx @@ -50,6 +50,7 @@ import { import { normalizeImportCandidateGroups, normalizeImportCollections, + normalizeGithubSkillCollectionPreviewResult, normalizeImportCandidate, selectedImportCollectionRequests, selectedImportCandidates, @@ -61,6 +62,7 @@ import { import { browserImportScanOptions, createImportScanRequestController, + createRemoteImportRequestController, importScanCommandArgs, waitForImportScanDelay } from './importScanProgress.js'; @@ -474,6 +476,7 @@ export default function App() { const historyRequestRef = useRef(0); const importScanControllerRef = useRef(null); const importScanTimingRef = useRef(null); + const remoteImportRequestControllerRef = useRef(null); const authoritativeGenerationRef = useRef(0); const pageRef = useRef(page); const dismissNotice = () => setNotice(''); @@ -1212,6 +1215,7 @@ export default function App() { } function openRemoteImport() { + remoteImportRequestControllerRef.current?.invalidate(); setError(''); setNotice(''); setImportReview((current) => ({ ...current, open: false })); @@ -1224,6 +1228,7 @@ export default function App() { } function closeRemoteImport() { + remoteImportRequestControllerRef.current?.invalidate(); setRemoteImport((current) => ({ ...current, open: false, error: '' })); } @@ -1233,6 +1238,7 @@ export default function App() { async function submitRemoteImport(event) { event.preventDefault(); + const mode = remoteImport.mode; const value = remoteImport.value.trim(); if (!value) { @@ -1240,18 +1246,27 @@ export default function App() { return; } - if (remoteImport.mode === 'url' && !isHttpUrl(value)) { + if (mode === 'url' && !isHttpUrl(value)) { setRemoteImport((current) => ({ ...current, error: 'Enter a full http(s) skill URL.' })); return; } - if (remoteImport.mode === 'markdown' && !value.toLowerCase().endsWith('.md')) { + if (mode === 'markdown' && !value.toLowerCase().endsWith('.md')) { setRemoteImport((current) => ({ ...current, error: 'Enter a local Markdown file path ending in .md.' })); return; } + if (!remoteImportRequestControllerRef.current) { + remoteImportRequestControllerRef.current = createRemoteImportRequestController(); + } + const requestController = remoteImportRequestControllerRef.current; + const requestId = requestController.begin(); + if (requestId == null) { + return; + } + if (!window.__TAURI_INTERNALS__) { - if (remoteImport.mode === 'url') { + if (mode === 'url') { const preview = normalizeRemoteInstallPreview({ preview_id: 'browser-preview', skill_name: remoteImportCandidate(remoteImport.mode, value).name || 'remote-skill', @@ -1297,6 +1312,7 @@ export default function App() { setRemoteImport((current) => ({ ...current, open: false, value: '', error: '' })); setNotice('Browser preview is using a provided remote source.'); setStatus('prototype'); + requestController.finish(requestId); return; } setImportReview({ @@ -1310,6 +1326,7 @@ export default function App() { setRemoteImport((current) => ({ ...current, open: false, value: '', error: '' })); setNotice('Browser preview is using a provided remote source.'); setStatus('prototype'); + requestController.finish(requestId); return; } @@ -1331,49 +1348,102 @@ export default function App() { error: '' }); await waitForNextPaint(); - const result = await invoke('preview_github_remote_skill_install', { - request: { - source_url: value, - target_root: null + if (!requestController.isCurrent(requestId)) { + return; + } + const routedResult = normalizeGithubSkillCollectionPreviewResult( + await invoke('preview_github_skill_collection', { + request: { source_url: value } + }) + ); + if (!requestController.isCurrent(requestId)) { + return; + } + if (routedResult.kind === 'single_skill') { + const result = await invoke('preview_github_remote_skill_install', { + request: { + source_url: value, + target_root: null + } + }); + if (!requestController.isCurrent(requestId)) { + return; } - }); - const preview = normalizeRemoteInstallPreview(result); - setRemoteInstallDialog({ - open: true, - loading: false, - applying: false, - preview, - activePath: preview.activePath, - confirmWarnings: false, - title: `Install ${preview.skillName}`, - subtitle: 'Review the GitHub skill before SkillBox copies it into the managed store.', - applyLabel: 'Install from GitHub', - applyingLabel: 'Installing...', - error: '' - }); - setRemoteImport((current) => ({ ...current, value: '', error: '' })); - setStatus('ready'); - return; + const preview = normalizeRemoteInstallPreview(result); + setRemoteInstallDialog({ + open: true, + loading: false, + applying: false, + preview, + activePath: preview.activePath, + confirmWarnings: false, + title: `Install ${preview.skillName}`, + subtitle: 'Review the GitHub skill before SkillBox copies it into the managed store.', + applyLabel: 'Install from GitHub', + applyingLabel: 'Installing...', + error: '' + }); + setRemoteImport((current) => ({ ...current, value: '', error: '' })); + setStatus('ready'); + requestController.finish(requestId); + return; + } + if (routedResult.kind === 'explicit_reference_required') { + throw new Error(routedResult.message); + } + { + const collectionResult = routedResult.preview; + const collections = normalizeImportCollections([collectionResult.collection]); + const candidates = normalizeImportCandidateGroups( + collectionResult.groups || [], + [] + ); + setImportReview({ + open: true, + loading: false, + candidates, + collections, + errors: collectionResult.errors || collectionResult.collection?.errors || [], + scanError: '', + scanProgress: null, + diagnostics: collectionResult.diagnostics || null, + title: 'GitHub Collection Review', + subtitle: 'Review selected skills from one repository snapshot before SkillBox writes managed state.', + noticePrefix: '', + remoteRequestId: requestId + }); + setRemoteImport((current) => ({ ...current, value: '', error: '' })); + setRemoteInstallDialog((current) => ({ ...current, open: false, loading: false })); + setStatus('ready'); + requestController.finish(requestId); + return; + } } else { setNotice('Markdown file import is not wired yet.'); } } catch (submitError) { + if (!requestController.isCurrent(requestId)) { + return; + } setRemoteImport((current) => ({ ...current, - open: remoteImport.mode !== 'url', + open: mode !== 'url', error: submitError.message || String(submitError) || 'Unable to prepare this import.' })); setRemoteInstallDialog((current) => ({ ...current, loading: false, error: submitError.message || String(submitError) })); setStatus('ready'); + requestController.finish(requestId); return; } setRemoteImport((current) => ({ ...current, open: false, value: '', error: '' })); setStatus('ready'); + requestController.finish(requestId); } function closeImportReview() { importScanControllerRef.current?.invalidate(); + remoteImportRequestControllerRef.current?.invalidate(); setImportReview((current) => ({ ...current, open: false, @@ -1426,6 +1496,12 @@ export default function App() { } async function runCandidateImport(selected, noticePrefix = '', collectionRequests = []) { + const remoteRequestId = collectionRequests.length > 0 + ? importReview.remoteRequestId + : null; + const isCurrentRemoteRequest = () => + remoteRequestId == null + || remoteImportRequestControllerRef.current?.isCurrent(remoteRequestId) === true; setStatus('importing'); setError(''); setNotice(''); @@ -1448,26 +1524,44 @@ export default function App() { : { imported: [], errors: [] }; const collectionResults = []; for (const request of collectionRequests) { - collectionResults.push(await invoke('apply_import_collection', { - request: { - collection_id: request.collectionId, - worktree_root: request.worktreeRoot, - preview_id: request.previewId, - selections: request.selections.map((selection) => ({ - relative_path: selection.relativePath, - group_id: selection.groupId, - variant_id: selection.variantId, - skill_type: selection.skillType - })), - actor: 'desktop' - } - })); + const command = request.sourceKind === 'github_remote' + ? 'apply_github_skill_collection' + : 'apply_import_collection'; + const requestBody = { + collection_id: request.collectionId, + preview_id: request.previewId, + selections: request.selections.map((selection) => ({ + relative_path: selection.relativePath, + group_id: selection.groupId, + variant_id: selection.variantId, + skill_type: selection.skillType + })), + actor: 'desktop' + }; + if (request.sourceKind === 'github_remote') { + requestBody.source_url = request.sourceUrl; + } else { + requestBody.worktree_root = request.worktreeRoot; + } + collectionResults.push(await invoke(command, { request: requestBody })); + if (!isCurrentRemoteRequest()) { + return; + } } + if (!isCurrentRemoteRequest()) { + return; + } setImportReview({ open: false, candidates: [], collections: [], errors: [], noticePrefix: '' }); await refresh(); + if (!isCurrentRemoteRequest()) { + return; + } if (page === 'rankings') { await loadUsageRankings(usageRankingFilters); + if (!isCurrentRemoteRequest()) { + return; + } } const collectionCount = collectionResults.reduce( (count, collection) => count + (collection.imported || []).length, @@ -1479,6 +1573,9 @@ export default function App() { ].filter(Boolean).join(' '); setNotice(importNotice(noticePrefix, summary || 'Import completed.')); } catch (importError) { + if (!isCurrentRemoteRequest()) { + return; + } setError(importError.message || 'Unable to import selected skills.'); setStatus('ready'); } @@ -3711,6 +3808,7 @@ export default function App() { function closeRemoteInstallDialog() { if (remoteInstallDialog.applying) return; + remoteImportRequestControllerRef.current?.invalidate(); setRemoteInstallDialog((current) => ({ ...current, open: false, error: '' })); } diff --git a/apps/desktop/src/components/importReview.jsx b/apps/desktop/src/components/importReview.jsx index 155f184..464a9ca 100644 --- a/apps/desktop/src/components/importReview.jsx +++ b/apps/desktop/src/components/importReview.jsx @@ -399,6 +399,7 @@ function CollectionReviewCard({ }) { const [expanded, setExpanded] = useState(false); const isInstalledSource = collection.sourceKind === 'installed_source'; + const isGithubRemote = collection.sourceKind === 'github_remote'; const disclosureId = `${collection.id}-children`; const selectedCount = collection.children.filter((child) => { const group = groups.find((candidateGroup) => candidateGroup.id === child.groupId); @@ -413,12 +414,19 @@ function CollectionReviewCard({
{collection.displayName} - {isInstalledSource ? 'Installed source collection' : 'Git collection'} + {isInstalledSource ? 'Installed source collection' : isGithubRemote ? 'GitHub collection' : 'Git collection'} {collectionSkillCountLabel(collection.children.length)}
{isInstalledSource ? ( Source: {collection.originUrl || 'Installed source metadata'} + ) : isGithubRemote ? ( + <> + + Ref {collection.requestedReference || collection.branch || 'default'} · {shortSha} + + {collection.sourceUrl || collection.originUrl} + ) : ( <> @@ -478,6 +486,12 @@ function CollectionReviewCard({ {child.relativePath} Calls {group.usageCount || 0} + {isGithubRemote && child.diff ? ( +
+ Preview SKILL.md diff +
{child.diff}
+
+ ) : null} {child.locations.length > 1 ? ( {child.locations.length} runtime/source locations resolve to this child. ) : null} diff --git a/apps/desktop/src/importCandidates.js b/apps/desktop/src/importCandidates.js index 9647424..dbfef9e 100644 --- a/apps/desktop/src/importCandidates.js +++ b/apps/desktop/src/importCandidates.js @@ -117,6 +117,8 @@ export function normalizeImportCollections(collections = []) { canonicalWorktreeRoot: collection.canonicalWorktreeRoot || collection.canonical_worktree_root || '', canonicalRepositoryId: collection.canonicalRepositoryId || collection.canonical_repository_id || '', originUrl: collection.originUrl || collection.origin_url || '', + sourceUrl: collection.sourceUrl || collection.source_url || '', + requestedReference: collection.requestedReference || collection.requested_reference || '', branch: collection.branch || '', detached: Boolean(collection.detached), reviewedHeadSha: collection.reviewedHeadSha || collection.reviewed_head_sha || '', @@ -130,6 +132,7 @@ export function normalizeImportCollections(collections = []) { sourcePath: child.sourcePath || child.source_path || '', realPath: child.realPath || child.real_path || '', snapshotHash: child.snapshotHash || child.snapshot_hash || '', + diff: child.diff || '', contentHash: child.contentHash || child.content_hash || '', importStatus: child.importStatus || child.import_status || 'importable', suggestedTypes: child.suggestedTypes || child.suggested_types || [], @@ -146,10 +149,23 @@ export function normalizeImportCollections(collections = []) { })); } +export function normalizeGithubSkillCollectionPreviewResult(result = {}) { + if (result.kind === 'collection' && result.preview) { + return { kind: 'collection', preview: result.preview }; + } + if (result.kind === 'single_skill' || result.kind === 'explicit_reference_required') { + return { + kind: result.kind, + message: result.message || 'GitHub collection preview could not continue.' + }; + } + throw new Error('GitHub collection preview returned an invalid result.'); +} + export function importCollectionGroupIds(collections = [], { liveOnly = false } = {}) { return new Set( collections - .filter((collection) => !liveOnly || collection.sourceKind === 'git_worktree') + .filter((collection) => !liveOnly || ['git_worktree', 'github_remote'].includes(collection.sourceKind)) .flatMap((collection) => collection.children.map((child) => child.groupId)) ); } @@ -181,13 +197,13 @@ export function filterImportCollectionsByQuery(collections = [], query = '') { } export function selectedImportCollectionRequests(groups = [], collections = []) { - return collections.filter((collection) => collection.sourceKind === 'git_worktree').map((collection) => { + return collections.filter((collection) => ['git_worktree', 'github_remote'].includes(collection.sourceKind)).map((collection) => { const selections = collection.children .map((child) => { const group = groups.find((candidateGroup) => candidateGroup.id === child.groupId); if (!group || !group.isSelected || group.selectedVariantId !== child.variantId) return null; const variant = group.variants.find((candidateVariant) => candidateVariant.id === child.variantId); - if (!variant || !variant.selectedType || variant.candidate.conflict) return null; + if (!variant || !variant.selectedType || variant.candidate.conflict || child.conflict) return null; return { relativePath: child.relativePath, groupId: child.groupId, @@ -200,6 +216,8 @@ export function selectedImportCollectionRequests(groups = [], collections = []) ? null : { collectionId: collection.id, + sourceKind: collection.sourceKind, + sourceUrl: collection.sourceUrl || collection.originUrl || '', worktreeRoot: collection.canonicalWorktreeRoot, previewId: collection.previewId, selections diff --git a/apps/desktop/src/importScanProgress.js b/apps/desktop/src/importScanProgress.js index 4e1a620..82ef07c 100644 --- a/apps/desktop/src/importScanProgress.js +++ b/apps/desktop/src/importScanProgress.js @@ -79,6 +79,34 @@ export function createImportScanRequestController() { }; } +export function createRemoteImportRequestController() { + let latestRequestId = 0; + let activeRequestId = 0; + + return { + begin() { + if (activeRequestId !== 0) { + return null; + } + latestRequestId += 1; + activeRequestId = latestRequestId; + return activeRequestId; + }, + finish(requestId) { + if (activeRequestId === requestId) { + activeRequestId = 0; + } + }, + invalidate() { + latestRequestId += 1; + activeRequestId = 0; + }, + isCurrent(requestId) { + return requestId === latestRequestId; + } + }; +} + export function browserImportScanOptions(search = '') { const params = new URLSearchParams(search); const requestedDelay = Number(params.get('import-scan-delay-ms') || 0); diff --git a/apps/desktop/src/styles.css b/apps/desktop/src/styles.css index cce1ce8..6ea71c2 100644 --- a/apps/desktop/src/styles.css +++ b/apps/desktop/src/styles.css @@ -6367,6 +6367,32 @@ dd { font-size: 11px; } +.collectionChildDiff { + max-width: 100%; + margin-top: 7px; + color: var(--skillbox-text-muted); + font-size: 11px; +} + +.collectionChildDiff summary { + cursor: pointer; + font-weight: 650; +} + +.collectionChildDiff pre { + max-width: 100%; + max-height: 180px; + margin: 6px 0 0; + overflow: auto; + border: 1px solid var(--skillbox-border-soft); + border-radius: 6px; + background: var(--skillbox-surface-muted); + padding: 8px; + color: var(--skillbox-text-secondary); + font: 10px/1.45 "SF Mono", ui-monospace, Menlo, monospace; + white-space: pre-wrap; +} + .collectionChildUnlinkedLocations { display: grid; gap: 3px; diff --git a/crates/skillbox-cli/Cargo.toml b/crates/skillbox-cli/Cargo.toml index c8e3897..ecd1966 100644 --- a/crates/skillbox-cli/Cargo.toml +++ b/crates/skillbox-cli/Cargo.toml @@ -7,4 +7,5 @@ edition = "2021" serde = "1" serde_json = "1" skillbox-core = { path = "../skillbox-core" } +skillbox-git = { path = "../skillbox-git" } skillbox-github = { path = "../skillbox-github" } diff --git a/crates/skillbox-cli/src/main.rs b/crates/skillbox-cli/src/main.rs index e62601d..0d9975e 100644 --- a/crates/skillbox-cli/src/main.rs +++ b/crates/skillbox-cli/src/main.rs @@ -58,6 +58,35 @@ fn run(args: Vec) -> Result<(), String> { managed_root(command_args), )?) } + "github-collection-preview" => { + let source_url = positional(command_args).into_iter().next().ok_or_else(|| { + "Usage: skillbox github-collection-preview [--managed-root ]" + .to_string() + })?; + print_json(&skillbox_core::preview_github_skill_collection_result( + skillbox_core::PreviewGithubSkillCollectionRequest { source_url }, + managed_root(command_args), + )?) + } + "github-collection-apply" => { + let source_url = positional(command_args).into_iter().next().ok_or_else(|| { + "Usage: skillbox github-collection-apply --collection-id --preview-id --select " + .to_string() + })?; + let collection_id = required_option_value(command_args, "--collection-id")?; + let preview_id = required_option_value(command_args, "--preview-id")?; + let selections = collection_selections(command_args)?; + print_json(&skillbox_core::apply_github_skill_collection( + skillbox_core::GithubSkillCollectionApplyRequest { + source_url, + collection_id, + preview_id, + selections, + actor: "cli".to_string(), + }, + managed_root(command_args), + )?) + } "collection-apply" => { let root = positional(command_args).into_iter().next().ok_or_else(|| { "Usage: skillbox collection-apply --collection-id --preview-id --select " @@ -668,6 +697,8 @@ Commands: skillbox collections [--managed-root ] skillbox collection-preview [--managed-root ] skillbox collection-apply --collection-id --preview-id --select [--managed-root ] + skillbox github-collection-preview [--managed-root ] + skillbox github-collection-apply --collection-id --preview-id --select [--managed-root ] skillbox install-preview [--target ] [--managed-root ] skillbox install --preview-id [--target ] [--confirm-warnings] [--managed-root ] skillbox import --type user|remote [--managed-root ] @@ -922,6 +953,8 @@ mod tests { assert!(help.contains("skillbox collections [--managed-root ]")); assert!(help.contains("skillbox collection-preview ")); assert!(help.contains("skillbox collection-apply ")); + assert!(help.contains("skillbox github-collection-preview ")); + assert!(help.contains("skillbox github-collection-apply ")); let selections = collection_selections(&[ "--select".to_string(), @@ -935,6 +968,24 @@ mod tests { assert_eq!(selections[1].skill_type, skillbox_core::SkillKind::Remote); } + #[test] + fn github_collection_preview_command_uses_structured_core_result() { + let root = temp_dir("cli-github-collection-preview").join("SkillBox"); + let remote = bare_remote_with_skill_content("cli-github-collection-preview-origin", "demo"); + let _rewrite = github_repo_rewrite("acme", "cli-github-collection-preview", &remote); + + run(vec![ + "github-collection-preview".to_string(), + "https://github.com/acme/cli-github-collection-preview/tree/main".to_string(), + "--managed-root".to_string(), + root.to_string_lossy().to_string(), + ]) + .unwrap(); + + assert!(!root.join("user-skills").exists()); + assert!(!root.join("remote-skills").exists()); + } + #[test] fn inbound_apply_request_requires_preview_id() { let error = user_skills_inbound_apply_request(&[]).unwrap_err(); @@ -1498,18 +1549,11 @@ tools: struct GitConfigRewriteGuard { _lock: std::sync::MutexGuard<'static, ()>, - previous: Vec<(&'static str, Option)>, + _rewrite: skillbox_git::TestTrustedUrlRewriteGuard, } impl Drop for GitConfigRewriteGuard { - fn drop(&mut self) { - for (key, value) in self.previous.drain(..) { - match value { - Some(value) => std::env::set_var(key, value), - None => std::env::remove_var(key), - } - } - } + fn drop(&mut self) {} } fn github_repo_rewrite( @@ -1518,25 +1562,14 @@ tools: remote: &std::path::Path, ) -> GitConfigRewriteGuard { let lock = GIT_CONFIG_LOCK.lock().unwrap(); - let keys = ["GIT_CONFIG_COUNT", "GIT_CONFIG_KEY_0", "GIT_CONFIG_VALUE_0"]; - let previous = keys - .into_iter() - .map(|key| (key, std::env::var_os(key))) - .collect::>(); - - std::env::set_var("GIT_CONFIG_COUNT", "1"); - std::env::set_var( - "GIT_CONFIG_KEY_0", - format!("url.file://{}.insteadOf", remote.display()), - ); - std::env::set_var( - "GIT_CONFIG_VALUE_0", + let rewrite = skillbox_git::test_trusted_url_rewrite( + format!("file://{}", remote.display()), format!("https://github.com/{owner}/{repo}.git"), ); GitConfigRewriteGuard { _lock: lock, - previous, + _rewrite: rewrite, } } } diff --git a/crates/skillbox-core/src/collections.rs b/crates/skillbox-core/src/collections.rs index 848d0ce..615155d 100644 --- a/crates/skillbox-core/src/collections.rs +++ b/crates/skillbox-core/src/collections.rs @@ -2,11 +2,87 @@ use crate::*; use skillbox_git::{GitRepositoryIdentity, GitService}; use std::collections::{BTreeMap, HashMap, HashSet}; use std::fs; +use std::os::unix::fs::MetadataExt; use std::path::{Component, Path, PathBuf}; const MAX_COLLECTION_CHILDREN: usize = 500; const MAX_COLLECTION_ERRORS: usize = 32; +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum CollectionFsEntryKind { + Directory, + Symlink, + RegularFile, + Other, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct CollectionFsIdentity { + kind: CollectionFsEntryKind, + device: u64, + inode: u64, +} + +fn collection_fs_identity(path: &Path) -> Result { + let metadata = fs::symlink_metadata(path).map_err(|error| error.to_string())?; + let file_type = metadata.file_type(); + let kind = if file_type.is_dir() { + CollectionFsEntryKind::Directory + } else if file_type.is_symlink() { + CollectionFsEntryKind::Symlink + } else if file_type.is_file() { + CollectionFsEntryKind::RegularFile + } else { + CollectionFsEntryKind::Other + }; + Ok(CollectionFsIdentity { + kind, + device: metadata.dev(), + inode: metadata.ino(), + }) +} + +#[cfg(test)] +type CollectionImportTestHook = Box; + +#[cfg(test)] +thread_local! { + static TEST_COLLECTION_IMPORT_HOOK: std::cell::RefCell> = + std::cell::RefCell::new(None); +} + +#[cfg(test)] +pub(crate) struct CollectionImportHookGuard; + +#[cfg(test)] +pub(crate) fn install_collection_import_test_hook(hook: F) -> CollectionImportHookGuard +where + F: Fn(&[ImportedCandidate]) + 'static, +{ + TEST_COLLECTION_IMPORT_HOOK.with(|slot| { + *slot.borrow_mut() = Some(Box::new(hook)); + }); + CollectionImportHookGuard +} + +#[cfg(test)] +fn run_collection_import_test_hook(imported: &[ImportedCandidate]) { + TEST_COLLECTION_IMPORT_HOOK.with(|slot| { + if let Some(hook) = slot.borrow().as_ref() { + hook(imported); + } + }); +} + +#[cfg(test)] +impl Drop for CollectionImportHookGuard { + fn drop(&mut self) { + TEST_COLLECTION_IMPORT_HOOK.with(|slot| { + *slot.borrow_mut() = None; + }); + } +} + #[derive(Debug, Default)] pub(crate) struct CollectionDiscoveryStats { pub unique_repository_count: usize, @@ -164,6 +240,7 @@ where real_path: candidate.real_path.clone(), content_hash: candidate.content_hash.clone(), snapshot_hash, + diff: String::new(), import_status: candidate.import_status, conflict: candidate.conflict.clone(), usage_count: candidate.usage_count, @@ -267,6 +344,8 @@ impl CollectionBuilder { branch: self.identity.branch, detached, reviewed_head_sha: self.identity.head, + source_url: None, + requested_reference: None, children, errors: self .errors @@ -305,7 +384,7 @@ fn safe_collection_relative_path(root: &Path, path: &Path) -> Option { Some(relative.to_path_buf()) } -fn sanitize_origin_url(value: &str) -> String { +pub(crate) fn sanitize_origin_url(value: &str) -> String { let mut value = value.trim().to_string(); if let Some(index) = value.find(['?', '#']) { value.truncate(index); @@ -321,13 +400,37 @@ fn sanitize_origin_url(value: &str) -> String { value } +fn collection_source_kind_string(kind: ImportCandidateCollectionSourceKind) -> &'static str { + match kind { + ImportCandidateCollectionSourceKind::GitWorktree => "git_worktree", + ImportCandidateCollectionSourceKind::InstalledSource => "installed_source", + ImportCandidateCollectionSourceKind::GithubRemote => "github_remote", + } +} + +fn parse_collection_source_kind( + value: &str, +) -> rusqlite::Result { + match value { + "git_worktree" => Ok(ImportCandidateCollectionSourceKind::GitWorktree), + "installed_source" => Ok(ImportCandidateCollectionSourceKind::InstalledSource), + "github_remote" => Ok(ImportCandidateCollectionSourceKind::GithubRemote), + _ => Err(rusqlite::Error::InvalidColumnType( + 0, + "source_kind".to_string(), + rusqlite::types::Type::Text, + )), + } +} + pub fn list_skill_collections(managed_root: impl AsRef) -> Result> { let paths = ensure_managed_layout(expand_home(managed_root.as_ref().to_path_buf()))?; let connection = open_database(&paths.database_path)?; let mut statement = connection .prepare( "SELECT id, display_name, canonical_worktree_root, canonical_repository_id, - origin_url, branch, detached, reviewed_head_sha, available + origin_url, branch, detached, reviewed_head_sha, source_kind, source_url, + requested_reference, available FROM skill_collections ORDER BY display_name COLLATE NOCASE, id", ) @@ -343,7 +446,10 @@ pub fn list_skill_collections(managed_root: impl AsRef) -> Result(6)? != 0, reviewed_head_sha: row.get(7)?, - available: row.get::<_, i64>(8)? != 0, + source_kind: parse_collection_source_kind(&row.get::<_, String>(8)?)?, + source_url: row.get(9)?, + requested_reference: row.get(10)?, + available: row.get::<_, i64>(11)? != 0, members: Vec::new(), }) }) @@ -376,7 +482,9 @@ pub fn list_skill_collections(managed_root: impl AsRef) -> Result, _>>() .map_err(|error| error.to_string())?; - collection.available = collection.canonical_worktree_root.is_dir(); + if collection.source_kind == ImportCandidateCollectionSourceKind::GitWorktree { + collection.available = collection.canonical_worktree_root.is_dir(); + } } Ok(collections) } @@ -414,7 +522,6 @@ pub fn apply_import_collection( let mut selected_names = HashSet::new(); let mut items = Vec::new(); let mut selected_children = Vec::new(); - let mut targets = Vec::new(); for selection in &request.selections { let child = collection .children @@ -470,7 +577,6 @@ pub fn apply_import_collection( skill.name )); } - targets.push(target); items.push(ImportRequestItem { source_path, skill_type: selection.skill_type, @@ -478,45 +584,37 @@ pub fn apply_import_collection( }); selected_children.push(child.clone()); } - let batch = import_candidates_with_paths(&paths, items)?; - if !batch.errors.is_empty() { - return Err(collection_apply_error( - format!( - "Collection import did not complete: {} skill(s) failed.", - batch.errors.len() - ), - rollback_collection_imports(&paths, &targets), - )); + if let Some(existing_reviewed_sha) = + persisted_collection_reviewed_sha(&paths.database_path, &collection.id)? + { + if existing_reviewed_sha.as_ref() != collection.reviewed_head_sha.as_ref() { + return Err( + "Collection already has a different reviewed SHA; collection updates are not available until Phase D." + .to_string(), + ); + } } - let collection_record = match persist_collection( - &paths.database_path, + apply_collection_import_with_audit( + &paths, &collection, &selected_children, - &batch.imported, - ) { - Ok(record) => record, - Err(error) => { - return Err(collection_apply_error( - format!("Collection metadata could not be saved: {error}"), - rollback_collection_imports(&paths, &targets), - )); - } - }; - Ok(ImportCollectionApplyResult { - collection: collection_record, - imported: batch.imported, - errors: batch.errors, - }) + items, + &request.actor, + ) } -struct CollectionImportTarget { - name: String, - path: PathBuf, - remote_root: Option, - expected_snapshot_hash: String, +pub(crate) struct CollectionImportTarget { + pub(crate) name: String, + pub(crate) path: PathBuf, + pub(crate) remote_root: Option, + pub(crate) expected_snapshot_hash: String, + pub(crate) expected_path_identity: Option, + pub(crate) expected_current_identity: Option, + pub(crate) expected_versions_identity: Option, + pub(crate) expected_remote_root_identity: Option, } -fn collection_import_target( +pub(crate) fn collection_import_target( paths: &ManagedPaths, skill: &Skill, kind: SkillKind, @@ -528,6 +626,10 @@ fn collection_import_target( remote_root: None, expected_snapshot_hash: skill_directory_snapshot_hash(&skill.real_path) .unwrap_or_default(), + expected_path_identity: None, + expected_current_identity: None, + expected_versions_identity: None, + expected_remote_root_identity: None, }, SkillKind::Remote => CollectionImportTarget { name: skill.name.clone(), @@ -539,12 +641,140 @@ fn collection_import_target( remote_root: Some(paths.remote_skills_root.join(&skill.name)), expected_snapshot_hash: skill_directory_snapshot_hash(&skill.real_path) .unwrap_or_default(), + expected_path_identity: None, + expected_current_identity: None, + expected_versions_identity: None, + expected_remote_root_identity: None, }, } } -fn managed_index_contains(database_path: &Path, skill_name: &str) -> Result { - let connection = open_database(database_path)?; +fn collection_import_targets_from_imported( + imported: &[ImportedCandidate], +) -> (Vec, Vec) { + let mut targets = Vec::new(); + let mut errors = Vec::new(); + for result in imported { + let remote_root = match result.kind { + SkillKind::User => None, + SkillKind::Remote => result + .managed_path + .parent() + .and_then(Path::parent) + .map(Path::to_path_buf), + }; + let expected_snapshot_hash = match skill_directory_snapshot_hash(&result.managed_path) { + Ok(snapshot) => snapshot, + Err(error) => { + errors.push(format!( + "Unable to record rollback snapshot for {}: {error}", + result.name + )); + continue; + } + }; + let expected_path_identity = match collection_fs_identity(&result.managed_path) { + Ok(identity) if identity.kind == CollectionFsEntryKind::Directory => identity, + Ok(_) => { + errors.push(format!( + "Unable to record a directory rollback identity for {}.", + result.name + )); + continue; + } + Err(error) => { + errors.push(format!( + "Unable to record rollback identity for {}: {error}", + result.name + )); + continue; + } + }; + let expected_current_identity = match remote_root.as_ref() { + None => None, + Some(remote_root) => match collection_fs_identity(&remote_root.join("current")) { + Ok(identity) if identity.kind == CollectionFsEntryKind::Symlink => Some(identity), + Ok(_) => { + errors.push(format!( + "Unable to record a symlink rollback identity for {}.", + result.name + )); + continue; + } + Err(error) => { + errors.push(format!( + "Unable to record rollback identity for {} current: {error}", + result.name + )); + continue; + } + }, + }; + let (expected_versions_identity, expected_remote_root_identity) = match remote_root.as_ref() + { + None => (None, None), + Some(remote_root) => { + let versions = remote_root.join("versions"); + let versions_identity = match collection_fs_identity(&versions) { + Ok(identity) if identity.kind == CollectionFsEntryKind::Directory => identity, + Ok(_) => { + errors.push(format!( + "Unable to record a versions-directory rollback identity for {}.", + result.name + )); + continue; + } + Err(error) => { + errors.push(format!( + "Unable to record rollback versions identity for {}: {error}", + result.name + )); + continue; + } + }; + let root_identity = match collection_fs_identity(remote_root) { + Ok(identity) if identity.kind == CollectionFsEntryKind::Directory => identity, + Ok(_) => { + errors.push(format!( + "Unable to record a collection rollback root identity for {}.", + result.name + )); + continue; + } + Err(error) => { + errors.push(format!( + "Unable to record rollback root identity for {}: {error}", + result.name + )); + continue; + } + }; + (Some(versions_identity), Some(root_identity)) + } + }; + targets.push(CollectionImportTarget { + name: result.name.clone(), + path: result.managed_path.clone(), + remote_root, + expected_snapshot_hash, + expected_path_identity: Some(expected_path_identity), + expected_current_identity, + expected_versions_identity, + expected_remote_root_identity, + }); + } + (targets, errors) +} + +pub(crate) fn managed_index_contains(database_path: &Path, skill_name: &str) -> Result { + if !database_path.is_file() { + return Ok(false); + } + let connection = rusqlite::Connection::open_with_flags( + database_path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, + ) + .map_err(|error| error.to_string())?; connection .query_row( "SELECT EXISTS(SELECT 1 FROM skills WHERE name = ?1)", @@ -554,89 +784,181 @@ fn managed_index_contains(database_path: &Path, skill_name: &str) -> Result Result>> { + if !database_path.is_file() { + return Ok(None); + } + let connection = rusqlite::Connection::open_with_flags( + database_path, + rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY, + ) + .map_err(|error| error.to_string())?; + connection + .query_row( + "SELECT reviewed_head_sha FROM skill_collections WHERE id = ?1", + rusqlite::params![collection_id], + |row| row.get(0), + ) + .optional() + .map_err(|error| error.to_string()) +} + +pub(crate) fn rollback_collection_imports( paths: &ManagedPaths, targets: &[CollectionImportTarget], ) -> Result<()> { let mut errors = Vec::new(); for target in targets { - let exists = match fs::symlink_metadata(&target.path) { - Ok(metadata) if metadata.file_type().is_dir() => true, - Ok(metadata) if metadata.file_type().is_symlink() => { + let current = target.remote_root.as_ref().map(|root| root.join("current")); + let mut current_removed = current.is_none(); + if let Some(current) = ¤t { + match (target.expected_current_identity, collection_fs_identity(current)) { + (Some(expected), Ok(actual)) if actual == expected => { + if let Err(error) = fs::remove_file(current) { + errors.push(format!( + "Unable to remove rollback current for {}: {error}", + target.name + )); + } else { + current_removed = true; + } + } + (Some(_), Ok(_)) => errors.push(format!( + "Preserved rollback current for {} because its filesystem identity changed.", + target.name + )), + (Some(_), Err(error)) => errors.push(format!( + "Preserved rollback current for {} because its identity could not be verified: {error}", + target.name + )), + (None, _) => errors.push(format!( + "Preserved rollback current for {} because no filesystem identity was recorded.", + target.name + )), + } + } + + let target_removed = match ( + target.expected_path_identity, + collection_fs_identity(&target.path), + ) { + (Some(expected), Ok(actual)) if actual == expected => { + match skill_directory_snapshot_hash(&target.path) { + Ok(snapshot) if snapshot == target.expected_snapshot_hash => { + if let Err(error) = fs::remove_dir_all(&target.path) { + errors.push(format!( + "Unable to remove rollback target for {}: {error}", + target.name + )); + false + } else { + if let Err(error) = + remove_skill_index(&paths.database_path, &target.name) + { + errors.push(format!( + "Unable to restore index for {}: {error}", + target.name + )); + } + true + } + } + Ok(_) => { + errors.push(format!( + "Preserved rollback target for {} because its contents changed.", + target.name + )); + false + } + Err(error) => { + errors.push(format!( + "Preserved rollback target for {} because its contents could not be verified: {error}", + target.name + )); + false + } + } + } + (Some(_), Ok(_)) => { errors.push(format!( - "Refusing to remove unexpected symlink at {}", - target.path.display() + "Preserved rollback target for {} because its filesystem identity changed.", + target.name )); false } - Ok(_) => { + (Some(_), Err(error)) => { errors.push(format!( - "Refusing to remove unexpected non-directory at {}", - target.path.display() + "Preserved rollback target for {} because its identity could not be verified: {error}", + target.name )); false } - Err(error) if error.kind() == std::io::ErrorKind::NotFound => false, - Err(error) => { + (None, _) => { errors.push(format!( - "Unable to inspect {}: {error}", - target.path.display() + "Preserved rollback target for {} because no filesystem identity was recorded.", + target.name )); false } }; - if exists { - match skill_directory_snapshot_hash(&target.path) { - Ok(snapshot) if snapshot == target.expected_snapshot_hash => { - if let Err(error) = fs::remove_dir_all(&target.path) { - errors.push(format!( - "Unable to remove {}: {error}", - target.path.display() - )); - } else if let Err(error) = - remove_skill_index(&paths.database_path, &target.name) - { + + if let Some(remote_root) = &target.remote_root { + if target_removed && current_removed { + let versions = remote_root.join("versions"); + let versions_owned = matches!( + (target.expected_versions_identity, collection_fs_identity(&versions)), + (Some(expected), Ok(actual)) if expected == actual + ); + let root_owned = matches!( + ( + target.expected_remote_root_identity, + collection_fs_identity(remote_root) + ), + (Some(expected), Ok(actual)) if expected == actual + ); + if !versions_owned || !root_owned { + errors.push(format!( + "Preserved rollback root for {} because its filesystem identity changed.", + target.name + )); + continue; + } + if let Err(error) = fs::remove_dir(&versions) { + errors.push(format!( + "Unable to clean rollback versions for {}: {error}", + target.name + )); + continue; + } + let root_still_owned = matches!( + ( + target.expected_remote_root_identity, + collection_fs_identity(remote_root) + ), + (Some(expected), Ok(actual)) if expected == actual + ); + if root_still_owned { + if let Err(error) = fs::remove_dir(remote_root) { errors.push(format!( - "Unable to restore index for {}: {error}", + "Unable to clean rollback root for {}: {error}", target.name )); } + } else { + errors.push(format!( + "Preserved rollback root for {} because its filesystem identity changed.", + target.name + )); } - Ok(_) => errors.push(format!( - "Preserved {} because its contents changed during collection import.", - target.path.display() - )), - Err(error) => errors.push(format!( - "Preserved {} because its contents could not be verified: {error}", - target.path.display() - )), - } - } - if let Some(remote_root) = &target.remote_root { - let current = remote_root.join("current"); - match fs::symlink_metadata(¤t) { - Ok(metadata) if metadata.file_type().is_symlink() => { - let points_to_target = fs::canonicalize(¤t) - .ok() - .zip(fs::canonicalize(&target.path).ok()) - .is_some_and(|(current, target)| current == target); - if points_to_target { - if let Err(error) = fs::remove_file(¤t) { - errors.push(format!("Unable to remove {}: {error}", current.display())); - } - } - } - Ok(_) => errors.push(format!( - "Preserved unexpected remote current entry at {}", - current.display() - )), - Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} - Err(error) => { - errors.push(format!("Unable to inspect {}: {error}", current.display())) - } + } else { + errors.push(format!( + "Preserved rollback root for {} because an owned entry was not safely removed.", + target.name + )); } - let _ = fs::remove_dir(remote_root.join("versions")); - let _ = fs::remove_dir(remote_root); } } if errors.is_empty() { @@ -646,14 +968,7 @@ fn rollback_collection_imports( } } -fn collection_apply_error(primary: String, rollback: Result<()>) -> String { - match rollback { - Ok(()) => primary, - Err(error) => format!("{primary} Collection rollback was incomplete: {error}"), - } -} - -fn import_candidates_with_paths( +pub(crate) fn import_candidates_with_paths( paths: &ManagedPaths, items: Vec, ) -> Result { @@ -662,7 +977,7 @@ fn import_candidates_with_paths( let mut errors = Vec::new(); for item in items { let source_path = item.source_path.clone(); - match import_one_candidate(paths, item) { + match import_one_candidate_unlogged(paths, item) { Ok(candidate) => imported.push(candidate), Err(error) => errors.push(ImportCandidateError { source_path, error }), } @@ -670,7 +985,151 @@ fn import_candidates_with_paths( Ok(ImportBatchResult { imported, errors }) } -fn persist_collection( +pub(crate) fn apply_collection_import_with_audit( + paths: &ManagedPaths, + collection: &ImportCandidateCollection, + selected_children: &[ImportCandidateCollectionChild], + items: Vec, + actor: &str, +) -> Result { + let selected_names = selected_children + .iter() + .map(|child| child.name.clone()) + .collect::>(); + let operation = start_operation( + OperationStart { + operation_type: "import_collection".to_string(), + actor: actor.to_string(), + entity_type: "skill_collection".to_string(), + entity_name: collection.display_name.clone(), + summary: format!("Import selected skills from {}", collection.display_name), + payload: serde_json::json!({ + "collectionId": collection.id, + "sourceKind": collection.source_kind, + "sourceUrl": collection.source_url, + "reviewedHeadSha": collection.reviewed_head_sha, + "selectedSkillNames": selected_names, + "phase": "validated" + }), + }, + &paths.root, + )?; + + let fail = |primary: String, + phase: &str, + _imported: &[ImportedCandidate], + receipt_targets: &[CollectionImportTarget], + receipt_errors: &[String]| + -> Result { + let rollback = rollback_collection_imports(paths, receipt_targets); + let rollback_error = match rollback { + Ok(()) if receipt_errors.is_empty() => None, + Ok(()) => Some(receipt_errors.join(" ")), + Err(error) if receipt_errors.is_empty() => Some(error), + Err(error) => Some(format!("{} {error}", receipt_errors.join(" "))), + }; + let rollback_outcome = if rollback_error.is_some() { + "partial" + } else { + "succeeded" + }; + let mut error_message = primary.clone(); + if let Some(error) = rollback_error.as_deref() { + error_message.push_str(&format!(" Collection rollback was incomplete: {error}")); + } + let payload = serde_json::json!({ + "collectionId": collection.id, + "reviewedHeadSha": collection.reviewed_head_sha, + "selectedSkillNames": selected_names, + "phase": phase, + "rollback": rollback_outcome, + "rollbackError": rollback_error, + "partialRecovery": rollback_outcome == "partial" + }); + match finish_operation( + OperationFinish { + id: operation.id.clone(), + status: OperationStatus::Failed, + summary: format!("Collection import failed for {}", collection.display_name), + error: Some(error_message.clone()), + payload, + }, + &paths.root, + ) { + Ok(_) => Err(error_message), + Err(log_error) => Err(format!( + "{error_message} (operation log failed: {log_error})" + )), + } + }; + + let batch = match import_candidates_with_paths(paths, items) { + Ok(batch) => batch, + Err(error) => return fail(error, "import_validation", &[], &[], &[]), + }; + // Capture ownership receipts immediately after successful imports and + // before any later-child failure or test/concurrency seam can replace them. + let (receipt_targets, receipt_errors) = + collection_import_targets_from_imported(&batch.imported); + #[cfg(test)] + run_collection_import_test_hook(&batch.imported); + if !batch.errors.is_empty() { + return fail( + format!( + "Collection import did not complete: {} skill(s) failed.", + batch.errors.len() + ), + "import", + &batch.imported, + &receipt_targets, + &receipt_errors, + ); + } + let collection_record = match persist_collection( + &paths.database_path, + collection, + selected_children, + &batch.imported, + ) { + Ok(record) => record, + Err(error) => { + return fail( + format!("Collection metadata could not be saved: {error}"), + "persist", + &batch.imported, + &receipt_targets, + &receipt_errors, + ) + } + }; + let mut warnings = Vec::new(); + if let Err(error) = finish_operation( + OperationFinish { + id: operation.id, + status: OperationStatus::Succeeded, + summary: format!("Imported selected skills from {}", collection.display_name), + error: None, + payload: serde_json::json!({ + "collectionId": collection.id, + "reviewedHeadSha": collection.reviewed_head_sha, + "selectedSkillNames": selected_names, + "importedSkillNames": batch.imported.iter().map(|item| item.name.clone()).collect::>(), + "phase": "completed" + }), + }, + &paths.root, + ) { + warnings.push(format!("Collection import completed, but its operation history could not be finalized: {error}")); + } + Ok(ImportCollectionApplyResult { + collection: collection_record, + imported: batch.imported, + errors: batch.errors, + warnings, + }) +} + +pub(crate) fn persist_collection( database_path: &Path, preview: &ImportCandidateCollection, children: &[ImportCandidateCollectionChild], @@ -680,12 +1139,29 @@ fn persist_collection( let transaction = connection .transaction() .map_err(|error| error.to_string())?; + let existing_reviewed_sha: Option> = transaction + .query_row( + "SELECT reviewed_head_sha FROM skill_collections WHERE id = ?1", + rusqlite::params![preview.id], + |row| row.get(0), + ) + .optional() + .map_err(|error| error.to_string())?; + if let Some(existing_reviewed_sha) = existing_reviewed_sha { + if existing_reviewed_sha.as_ref() != preview.reviewed_head_sha.as_ref() { + return Err( + "Collection already has a different reviewed SHA; collection updates are not available until Phase D." + .to_string(), + ); + } + } transaction .execute( "INSERT INTO skill_collections ( id, display_name, canonical_worktree_root, canonical_repository_id, - origin_url, branch, detached, reviewed_head_sha, available, updated_at - ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, 1, ?9) + origin_url, branch, detached, reviewed_head_sha, source_kind, source_url, + requested_reference, available, updated_at + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, 1, ?12) ON CONFLICT(id) DO UPDATE SET display_name = excluded.display_name, canonical_worktree_root = excluded.canonical_worktree_root, @@ -694,6 +1170,9 @@ fn persist_collection( branch = excluded.branch, detached = excluded.detached, reviewed_head_sha = excluded.reviewed_head_sha, + source_kind = excluded.source_kind, + source_url = excluded.source_url, + requested_reference = excluded.requested_reference, available = 1, updated_at = excluded.updated_at", rusqlite::params![ @@ -705,23 +1184,26 @@ fn persist_collection( preview.branch, i64::from(preview.detached), preview.reviewed_head_sha, + collection_source_kind_string(preview.source_kind), + preview.source_url, + preview.requested_reference, current_rfc3339_timestamp(), ], ) .map_err(|error| error.to_string())?; - transaction - .execute( - "DELETE FROM skill_collection_members WHERE collection_id = ?1", - rusqlite::params![preview.id], - ) - .map_err(|error| error.to_string())?; for (child, imported) in children.iter().zip(imported.iter()) { transaction .execute( "INSERT INTO skill_collection_members ( collection_id, skill_name, relative_path, reviewed_head_sha, snapshot_hash, content_hash, managed_skill_name - ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)", + ) VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7) + ON CONFLICT(collection_id, relative_path) DO UPDATE SET + skill_name = excluded.skill_name, + reviewed_head_sha = excluded.reviewed_head_sha, + snapshot_hash = excluded.snapshot_hash, + content_hash = excluded.content_hash, + managed_skill_name = excluded.managed_skill_name", rusqlite::params![ preview.id, child.name, @@ -734,6 +1216,29 @@ fn persist_collection( ) .map_err(|error| error.to_string())?; } + let members = transaction + .prepare( + "SELECT collection_id, skill_name, relative_path, reviewed_head_sha, + snapshot_hash, content_hash, managed_skill_name + FROM skill_collection_members + WHERE collection_id = ?1 + ORDER BY relative_path", + ) + .map_err(|error| error.to_string())? + .query_map(rusqlite::params![preview.id], |row| { + Ok(SkillCollectionMember { + collection_id: row.get(0)?, + skill_name: row.get(1)?, + relative_path: row.get(2)?, + reviewed_head_sha: row.get(3)?, + snapshot_hash: row.get(4)?, + content_hash: row.get(5)?, + managed_skill_name: row.get(6)?, + }) + }) + .map_err(|error| error.to_string())? + .collect::, _>>() + .map_err(|error| error.to_string())?; transaction.commit().map_err(|error| error.to_string())?; Ok(SkillCollection { id: preview.id.clone(), @@ -744,20 +1249,11 @@ fn persist_collection( branch: preview.branch.clone(), detached: preview.detached, reviewed_head_sha: preview.reviewed_head_sha.clone(), + source_kind: preview.source_kind, + source_url: preview.source_url.clone(), + requested_reference: preview.requested_reference.clone(), available: true, - members: children - .iter() - .zip(imported) - .map(|(child, imported)| SkillCollectionMember { - collection_id: preview.id.clone(), - skill_name: child.name.clone(), - relative_path: child.relative_path.clone(), - reviewed_head_sha: preview.reviewed_head_sha.clone(), - snapshot_hash: child.snapshot_hash.clone(), - content_hash: child.content_hash.clone(), - managed_skill_name: imported.name.clone(), - }) - .collect(), + members, }) } diff --git a/crates/skillbox-core/src/db.rs b/crates/skillbox-core/src/db.rs index 3fd0c07..ed11d4a 100644 --- a/crates/skillbox-core/src/db.rs +++ b/crates/skillbox-core/src/db.rs @@ -2,7 +2,7 @@ use crate::*; use fs2::FileExt; use std::fs::{File, OpenOptions}; -pub(crate) const LATEST_DATABASE_SCHEMA_VERSION: i64 = 8; +pub(crate) const LATEST_DATABASE_SCHEMA_VERSION: i64 = 9; pub(crate) fn open_database(database_path: &Path) -> Result { let connection = Connection::open(database_path).map_err(|error| error.to_string())?; @@ -96,6 +96,7 @@ pub(crate) fn run_database_migrations(connection: &mut Connection) -> Result<()> (6_i64, "runtime_profiles"), (7_i64, "usage_evidence_classification"), (8_i64, "skill_collections"), + (9_i64, "github_skill_collections"), ] { let applied: bool = connection .query_row( @@ -120,6 +121,7 @@ pub(crate) fn run_database_migrations(connection: &mut Connection) -> Result<()> 6 => apply_runtime_profiles_migration(&transaction)?, 7 => apply_usage_evidence_classification_migration(&transaction)?, 8 => apply_skill_collections_migration(&transaction)?, + 9 => apply_github_skill_collections_migration(&transaction)?, _ => return Err(format!("Unknown database migration version: {version}")), } transaction @@ -428,6 +430,22 @@ fn apply_skill_collections_migration(connection: &Connection) -> Result<()> { .map_err(|error| error.to_string()) } +fn apply_github_skill_collections_migration(connection: &Connection) -> Result<()> { + ensure_database_column( + connection, + "skill_collections", + "source_kind", + "TEXT NOT NULL DEFAULT 'git_worktree'", + )?; + ensure_database_column(connection, "skill_collections", "source_url", "TEXT")?; + ensure_database_column( + connection, + "skill_collections", + "requested_reference", + "TEXT", + ) +} + fn usage_evidence_repair_required(connection: &Connection) -> Result { let table_exists: bool = connection .query_row( diff --git a/crates/skillbox-core/src/github_collections.rs b/crates/skillbox-core/src/github_collections.rs new file mode 100644 index 0000000..ddb0918 --- /dev/null +++ b/crates/skillbox-core/src/github_collections.rs @@ -0,0 +1,993 @@ +use crate::*; +use std::collections::{HashMap, HashSet}; +use std::fs; +use std::path::{Component, Path, PathBuf}; +use std::time::Instant; + +const MAX_REMOTE_COLLECTION_CHILDREN: usize = 500; +const MAX_REMOTE_COLLECTION_ENTRIES: usize = skillbox_git::MAX_STRICT_TREE_ENTRIES; +const MAX_REMOTE_COLLECTION_FILE_BYTES: u64 = skillbox_git::MAX_STRICT_TREE_FILE_BYTES; +const MAX_REMOTE_COLLECTION_TOTAL_BYTES: u64 = skillbox_git::MAX_STRICT_TREE_TOTAL_BYTES; +const MAX_REMOTE_COLLECTION_DEPTH: usize = 12; + +pub fn preview_github_skill_collection( + request: PreviewGithubSkillCollectionRequest, + managed_root: impl AsRef, +) -> Result { + let source = skillbox_github::parse_github_skill_url(&request.source_url)?; + if !source.is_root { + return Err( + "This URL points to one skill. Use the single-skill GitHub install preview for it." + .to_string(), + ); + } + if !source.reference_explicit { + return Err( + "GitHub repository URLs need an explicit ref for collection preview. Use /tree/ so the reviewed commit is unambiguous." + .to_string(), + ); + } + + let started = Instant::now(); + let temp = temporary_work_dir("github-collection-preview"); + let result = (|| { + let checkout = temp.join("checkout"); + let git = skillbox_git::GitService::new(); + let fetch = + git.fetch_ref_tree_with_diagnostics(&source.repo_url, &source.reference, &checkout)?; + let paths = managed_paths(managed_root.as_ref().to_path_buf()); + build_github_skill_collection_preview( + &source, + &fetch.resolved_sha, + &checkout, + &paths, + fetch.fetch_count, + started, + ) + })(); + let _ = fs::remove_dir_all(&temp); + result +} + +pub fn preview_github_skill_collection_result( + request: PreviewGithubSkillCollectionRequest, + managed_root: impl AsRef, +) -> Result { + let source = skillbox_github::parse_github_skill_url(&request.source_url)?; + if !source.is_root { + return Ok(GithubSkillCollectionPreviewResult::SingleSkill { + message: + "This URL points to one skill. Use the single-skill GitHub install preview for it." + .to_string(), + }); + } + if !source.reference_explicit { + return Ok(GithubSkillCollectionPreviewResult::ExplicitReferenceRequired { + message: "GitHub repository URLs need an explicit ref for collection preview. Use /tree/ so the reviewed commit is unambiguous." + .to_string(), + }); + } + Ok(GithubSkillCollectionPreviewResult::Collection { + preview: Box::new(preview_github_skill_collection(request, managed_root)?), + }) +} + +pub fn apply_github_skill_collection( + request: GithubSkillCollectionApplyRequest, + managed_root: impl AsRef, +) -> Result { + if request.selections.is_empty() { + return Err("Select at least one skill from the collection.".to_string()); + } + + let mutation_lock = acquire_user_skills_mutation_lock(managed_root.as_ref())?; + let truth_root = mutation_lock.truth_root().to_path_buf(); + let paths = managed_paths(truth_root.clone()); + let source = skillbox_github::parse_github_skill_url(&request.source_url)?; + if !source.is_root { + return Err( + "This URL points to one skill. Use the single-skill GitHub install flow for it." + .to_string(), + ); + } + if !source.reference_explicit { + return Err( + "GitHub repository URLs need an explicit ref for collection apply. Re-open preview with /tree/." + .to_string(), + ); + } + + let temp = temporary_work_dir("github-collection-apply"); + let result = (|| { + let checkout = temp.join("checkout"); + let git = skillbox_git::GitService::new(); + let fetch = + git.fetch_ref_tree_with_diagnostics(&source.repo_url, &source.reference, &checkout)?; + let preview = build_github_skill_collection_preview( + &source, + &fetch.resolved_sha, + &checkout, + &paths, + fetch.fetch_count, + Instant::now(), + )?; + let collection = &preview.collection; + if collection.id != request.collection_id + || collection.preview_id != request.preview_id + || collection.source_url.as_deref() != Some(source.url.as_str()) + { + return Err( + "GitHub collection preview is stale. Re-open the preview and try again." + .to_string(), + ); + } + if let Some(existing_reviewed_sha) = + persisted_collection_reviewed_sha(&paths.database_path, &collection.id)? + { + if existing_reviewed_sha.as_ref() != collection.reviewed_head_sha.as_ref() { + return Err( + "Collection already has a different reviewed SHA; collection updates are not available until Phase D." + .to_string(), + ); + } + } + + let mut selected_names = HashSet::new(); + let mut items = Vec::new(); + let mut selected_children = Vec::new(); + for selection in &request.selections { + let child = collection + .children + .iter() + .find(|child| child.relative_path == selection.relative_path) + .ok_or_else(|| { + "Selected GitHub collection child is not part of the reviewed preview." + .to_string() + })?; + if child.group_id != selection.group_id + || child.variant_id != selection.variant_id + || child.import_status != ImportCandidateStatus::Importable + || child.conflict.is_some() + { + return Err(format!( + "Selected GitHub collection child {} is stale or not importable.", + child.name + )); + } + if !selected_names.insert(child.name.to_ascii_lowercase()) { + return Err(format!( + "Import review may select only one source variant for skill {}.", + child.name + )); + } + let source_path = checkout.join(&child.relative_path); + let skill = read_skill(&source_path) + .map_err(|error| format!("GitHub collection preview is stale: {error}"))?; + validate_skill_name(&skill.name)?; + if skill.name != child.name + || skill.content_hash != child.content_hash + || skill_directory_snapshot_hash(&source_path)? != child.snapshot_hash + { + return Err( + "GitHub collection preview is stale. Re-open the preview and try again." + .to_string(), + ); + } + let target = collection_import_target(&paths, &skill, selection.skill_type); + if managed_index_contains(&paths.database_path, &skill.name)? + || fs::symlink_metadata(&target.path).is_ok() + || target + .remote_root + .as_ref() + .is_some_and(|path| fs::symlink_metadata(path).is_ok()) + { + return Err(format!( + "Managed target for {} changed after preview. Review the collection again.", + skill.name + )); + } + items.push(ImportRequestItem { + source_path, + skill_type: selection.skill_type, + deploy_back_to_source: false, + }); + selected_children.push(child.clone()); + } + + // Keep layout/database creation after source, selection, snapshot, and + // managed-target validation so stale previews do not initialize state. + let paths = ensure_managed_layout(truth_root.clone())?; + apply_collection_import_with_audit( + &paths, + collection, + &selected_children, + items, + &request.actor, + ) + })(); + let _ = fs::remove_dir_all(&temp); + result +} + +fn build_github_skill_collection_preview( + source: &skillbox_github::GitHubSkillSource, + resolved_sha: &str, + checkout: &Path, + paths: &ManagedPaths, + fetch_count: usize, + started: Instant, +) -> Result { + let checkout = fs::canonicalize(checkout).map_err(|error| error.to_string())?; + let mut skill_dirs = Vec::new(); + let mut errors = Vec::new(); + let mut entry_count = 0; + let mut total_bytes = 0; + if is_regular_file(&checkout.join("SKILL.md")) { + skill_dirs.push(checkout.to_path_buf()); + } + walk_remote_tree( + &checkout, + &checkout, + 0, + &mut entry_count, + &mut total_bytes, + &mut skill_dirs, + &mut errors, + )?; + skill_dirs.sort(); + skill_dirs.dedup(); + if skill_dirs.iter().enumerate().any(|(index, parent)| { + skill_dirs[index + 1..] + .iter() + .any(|child| child.starts_with(parent)) + }) { + return Err( + "GitHub collection contains overlapping skill directories. Keep SKILL.md roots disjoint." + .to_string(), + ); + } + if skill_dirs.is_empty() { + return Err("The repository contains no valid SKILL.md directories.".to_string()); + } + if skill_dirs.len() > MAX_REMOTE_COLLECTION_CHILDREN { + return Err(format!( + "GitHub collection contains {} skill directories, exceeding the safety limit of {}. Narrow the repository or select a smaller source.", + skill_dirs.len(), + MAX_REMOTE_COLLECTION_CHILDREN + )); + } + + let usage_by_skill = if paths.database_path.is_file() { + load_usage_by_skill(&paths.database_path).unwrap_or_default() + } else { + HashMap::new() + }; + let mut candidates = Vec::new(); + for skill_dir in &skill_dirs { + match read_remote_skill(skill_dir) { + Ok(skill) => { + let conflict = managed_target_conflict(paths, &skill, SkillKind::Remote)?; + let imported = managed_skill_matches(paths, &skill); + let status = if imported { + ImportCandidateStatus::Imported + } else { + ImportCandidateStatus::Importable + }; + candidates.push(ImportCandidate { + name: skill.name, + description: skill.description, + source_path: skill.path.clone(), + source_root: Some(checkout.to_path_buf()), + real_path: skill.real_path, + is_symlink: false, + symlink_target_path: None, + content_hash: skill.content_hash, + additional_source_paths: Vec::new(), + suggested_type: SkillKind::Remote, + suggestion_reason: "GitHub repository source".to_string(), + import_status: status, + is_selected: status == ImportCandidateStatus::Importable && conflict.is_none(), + conflict, + usage_count: 0, + }); + } + Err(error) => errors.push(ImportCandidateError { + source_path: safe_relative_display(&checkout, skill_dir), + error, + }), + } + } + if candidates.is_empty() { + return Err("The repository contains no valid SKILL.md directories.".to_string()); + } + + let (mut groups, _) = group_import_candidates(&candidates, &usage_by_skill); + let mut children = Vec::new(); + let mut relative_paths_by_name = HashMap::::new(); + for group in &groups { + for variant in &group.variants { + for location in &variant.locations { + let relative_path = safe_collection_relative_path(&checkout, &location.real_path) + .ok_or_else(|| { + "Remote collection contained an unsafe child path.".to_string() + })?; + let relative_path = relative_path.to_string_lossy().to_string(); + *relative_paths_by_name + .entry(variant.candidate.name.to_ascii_lowercase()) + .or_default() += 1; + let snapshot_hash = if variant.snapshot_hash.is_empty() { + skill_directory_snapshot_hash(&location.real_path)? + } else { + variant.snapshot_hash.clone() + }; + let diff_path = if relative_path.is_empty() { + "SKILL.md".to_string() + } else { + format!("{relative_path}/SKILL.md") + }; + let diff = new_file_diff(&checkout, &diff_path).unwrap_or_default(); + children.push(ImportCandidateCollectionChild { + id: format!( + "child-{}", + &sha256(&format!("{}\n{}", group.id, relative_path))[..16] + ), + group_id: group.id.clone(), + variant_id: variant.id.clone(), + name: variant.candidate.name.clone(), + relative_path, + source_path: location.source_path.clone(), + real_path: location.real_path.clone(), + content_hash: variant.candidate.content_hash.clone(), + snapshot_hash, + diff, + import_status: variant.candidate.import_status, + conflict: variant.candidate.conflict.clone(), + usage_count: group.usage_count, + locations: vec![location.clone()], + unlinked_locations: Vec::new(), + suggested_types: variant.suggested_types.clone(), + requires_type_review: variant.requires_type_review, + selected_type: variant.selected_type, + is_selected: variant.candidate.is_selected, + }); + } + } + } + for child in &mut children { + if relative_paths_by_name + .get(&child.name.to_ascii_lowercase()) + .copied() + .unwrap_or_default() + > 1 + { + child.conflict = Some("Duplicate skill name in repository".to_string()); + child.is_selected = false; + } + } + sanitize_remote_group_paths(&mut groups, &checkout); + sanitize_remote_collection_children(&mut children, &checkout); + children.sort_by(|left, right| left.relative_path.cmp(&right.relative_path)); + let duplicate_name_count = relative_paths_by_name + .values() + .filter(|count| **count > 1) + .count(); + let child_seed = children + .iter() + .map(|child| { + format!( + "{}\n{}\n{}\n{:?}\n{}", + child.relative_path, + child.name, + child.snapshot_hash, + child.import_status, + child.conflict.as_deref().unwrap_or_default() + ) + }) + .collect::>() + .join("\n"); + let source_url = sanitize_origin_url(&source.url); + let collection_identity = format!( + "skillbox-github-collection-identity-v1\n{}\n{}\n{}", + "github_remote", source_url, source.reference + ); + let collection_id = format!("github-collection-{}", &sha256(&collection_identity)[..16]); + let preview_identity = format!( + "skillbox-github-collection-preview-v1\n{}\n{}\n{}\n{}", + source_url, source.reference, resolved_sha, child_seed + ); + let preview_id = format!("github-collection-preview-{}", sha256(&preview_identity)); + let collection = ImportCandidateCollection { + id: collection_id, + preview_id, + display_name: source.repo.clone(), + source_kind: ImportCandidateCollectionSourceKind::GithubRemote, + canonical_worktree_root: PathBuf::new(), + canonical_repository_id: PathBuf::from(source.repo_url.clone()), + origin_url: Some(source_url.clone()), + branch: Some(source.reference.clone()), + detached: false, + reviewed_head_sha: Some(resolved_sha.to_string()), + source_url: Some(source_url), + requested_reference: Some(source.reference.clone()), + children, + errors, + }; + let diagnostics = GithubSkillCollectionDiagnostics { + fetch_count, + child_count: skill_dirs.len(), + valid_child_count: candidates.len(), + invalid_child_count: collection.errors.len(), + duplicate_name_count, + elapsed_ms: started.elapsed().as_millis().min(u128::from(u64::MAX)) as u64, + }; + let preview_errors = collection.errors.clone(); + Ok(GithubSkillCollectionPreview { + collection, + groups, + errors: preview_errors, + diagnostics, + }) +} + +fn sanitize_remote_group_paths(groups: &mut [ImportCandidateGroup], checkout: &Path) { + for group in groups { + for variant in &mut group.variants { + let relative_candidate_path = + safe_relative_display(checkout, &variant.candidate.real_path); + variant.candidate.source_path = relative_candidate_path.clone(); + variant.candidate.source_root = Some(PathBuf::from(".")); + variant.candidate.real_path = relative_candidate_path.clone(); + variant.candidate.additional_source_paths = variant + .candidate + .additional_source_paths + .iter() + .map(|path| safe_relative_display(checkout, path)) + .collect(); + for location in &mut variant.locations { + let relative = safe_relative_display(checkout, &location.real_path); + location.source_path = relative.clone(); + location.source_root = Some(PathBuf::from(".")); + location.real_path = relative; + location.symlink_target_path = None; + } + } + } +} + +fn sanitize_remote_collection_children( + children: &mut [ImportCandidateCollectionChild], + checkout: &Path, +) { + for child in children { + child.source_path = safe_relative_display(checkout, &child.real_path); + child.real_path = child.source_path.clone(); + for location in &mut child.locations { + let relative = safe_relative_display(checkout, &location.real_path); + location.source_path = relative.clone(); + location.source_root = Some(PathBuf::from(".")); + location.real_path = relative; + location.symlink_target_path = None; + } + for location in &mut child.unlinked_locations { + let relative = safe_relative_display(checkout, &location.real_path); + location.source_path = relative.clone(); + location.source_root = Some(PathBuf::from(".")); + location.real_path = relative; + location.symlink_target_path = None; + } + } +} + +fn walk_remote_tree( + root: &Path, + current: &Path, + depth: usize, + entry_count: &mut usize, + total_bytes: &mut u64, + skill_dirs: &mut Vec, + errors: &mut Vec, +) -> Result<()> { + if depth > MAX_REMOTE_COLLECTION_DEPTH { + return Err("GitHub collection tree exceeds the directory depth safety limit.".to_string()); + } + let entries = fs::read_dir(current).map_err(|error| error.to_string())?; + for entry in entries { + *entry_count += 1; + if *entry_count > MAX_REMOTE_COLLECTION_ENTRIES { + return Err("GitHub collection tree exceeds the entry safety limit.".to_string()); + } + let entry = entry.map_err(|error| error.to_string())?; + let path = entry.path(); + if safe_collection_relative_path(root, &path).is_none() + || path + .strip_prefix(root) + .ok() + .map(|relative| { + relative.components().any(|component| { + matches!(component, Component::Normal(value) if value == ".git" || value.to_string_lossy().contains(':')) + }) + }) + .unwrap_or(true) + { + return Err("GitHub collection tree contains an unsafe path.".to_string()); + } + let file_type = entry.file_type().map_err(|error| error.to_string())?; + if file_type.is_symlink() { + errors.push(ImportCandidateError { + source_path: safe_relative_display(root, &path), + error: "Symlinks are not supported in GitHub skill collections.".to_string(), + }); + continue; + } + if file_type.is_dir() { + if entry.file_name() == ".git" { + return Err("Git metadata is not allowed in a GitHub collection tree.".to_string()); + } + if is_regular_file(&path.join("SKILL.md")) { + skill_dirs.push(path.clone()); + } + walk_remote_tree( + root, + &path, + depth + 1, + entry_count, + total_bytes, + skill_dirs, + errors, + )?; + } else if file_type.is_file() { + if entry.file_name() == ".git" { + return Err("Git metadata is not allowed in a GitHub collection tree.".to_string()); + } + let length = entry.metadata().map_err(|error| error.to_string())?.len(); + if length > MAX_REMOTE_COLLECTION_FILE_BYTES { + return Err("GitHub collection contains a file over the safety limit.".to_string()); + } + *total_bytes = total_bytes.saturating_add(length); + if *total_bytes > MAX_REMOTE_COLLECTION_TOTAL_BYTES { + return Err("GitHub collection exceeds the total byte safety limit.".to_string()); + } + } else { + return Err("GitHub collection contains an unsupported file type.".to_string()); + } + } + Ok(()) +} + +fn read_remote_skill(path: &Path) -> Result { + let skill_md = path.join("SKILL.md"); + if !is_regular_file(&skill_md) { + return Err("SKILL.md must be a regular non-symlink file.".to_string()); + } + validate_remote_skill_tree(path)?; + let content = fs::read_to_string(&skill_md).map_err(|error| error.to_string())?; + parse_skill_frontmatter_document(&content)?; + let skill = read_skill(path)?; + validate_skill_name(&skill.name)?; + Ok(skill) +} + +fn validate_remote_skill_tree(root: &Path) -> Result<()> { + let mut entries = vec![root.to_path_buf()]; + while let Some(current) = entries.pop() { + for entry in fs::read_dir(¤t).map_err(|error| error.to_string())? { + let entry = entry.map_err(|error| error.to_string())?; + let file_type = entry.file_type().map_err(|error| error.to_string())?; + if file_type.is_symlink() { + return Err("Skill directories may not contain symlinks.".to_string()); + } + if file_type.is_dir() { + entries.push(entry.path()); + } else if !file_type.is_file() { + return Err("Skill directories may not contain unsupported file types.".to_string()); + } + } + } + Ok(()) +} + +fn is_regular_file(path: &Path) -> bool { + fs::symlink_metadata(path) + .map(|metadata| metadata.file_type().is_file()) + .unwrap_or(false) +} + +fn managed_skill_matches(paths: &ManagedPaths, skill: &Skill) -> bool { + [ + paths.user_skills_root.join(&skill.name), + paths.remote_skills_root.join(&skill.name).join("current"), + ] + .iter() + .any(|path| { + read_skill(path) + .ok() + .is_some_and(|managed| managed.content_hash == skill.content_hash) + }) +} + +fn safe_collection_relative_path(root: &Path, path: &Path) -> Option { + let relative = path.strip_prefix(root).ok()?; + for component in relative.components() { + match component { + Component::Normal(name) => { + let name = name.to_string_lossy(); + if name == ".git" + || name.contains(':') + || name.contains('\\') + || name.chars().any(char::is_control) + { + return None; + } + } + _ => return None, + } + } + Some(relative.to_path_buf()) +} + +fn safe_relative_display(root: &Path, path: &Path) -> PathBuf { + safe_collection_relative_path(root, path).unwrap_or_else(|| PathBuf::from("")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn source() -> skillbox_github::GitHubSkillSource { + skillbox_github::GitHubSkillSource { + owner: "acme".to_string(), + repo: "skills".to_string(), + reference: "main".to_string(), + reference_explicit: true, + path: String::new(), + is_root: true, + url: "https://github.com/acme/skills".to_string(), + repo_url: "https://github.com/acme/skills.git".to_string(), + kind: "github".to_string(), + } + } + + fn write_skill(root: &Path, name: &str) { + let path = root.join("skills").join(name); + fs::create_dir_all(&path).unwrap(); + fs::write( + path.join("SKILL.md"), + format!("---\nname: {name}\ndescription: Test skill\n---\n"), + ) + .unwrap(); + } + + fn write_root_skill(root: &Path, name: &str) { + fs::create_dir_all(root).unwrap(); + fs::write( + root.join("SKILL.md"), + format!("---\nname: {name}\ndescription: Root skill\n---\n"), + ) + .unwrap(); + } + + #[test] + fn preview_groups_many_children_without_managed_writes() { + let temp = temporary_work_dir("github-collection-test"); + let checkout = temp.join("checkout"); + write_skill(&checkout, "alpha"); + write_skill(&checkout, "beta"); + let paths = managed_paths(temp.join("managed")); + + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + + assert_eq!( + preview.collection.source_kind, + ImportCandidateCollectionSourceKind::GithubRemote + ); + assert_eq!(preview.collection.children.len(), 2); + assert_eq!(preview.diagnostics.fetch_count, 1); + assert!(preview + .collection + .children + .iter() + .all(|child| child.diff.contains("SKILL.md"))); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_scales_to_one_hundred_children_with_one_fetch_diagnostic() { + let temp = temporary_work_dir("github-collection-large-fixture"); + let checkout = temp.join("checkout"); + for index in 0..100 { + write_skill(&checkout, &format!("skill-{index:03}")); + } + let paths = managed_paths(temp.join("managed")); + + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + + assert_eq!(preview.collection.children.len(), 100); + assert_eq!(preview.diagnostics.fetch_count, 1); + assert_eq!(preview.diagnostics.child_count, 100); + assert_eq!(preview.diagnostics.valid_child_count, 100); + assert!(preview.diagnostics.elapsed_ms < 30_000); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_rejects_more_than_five_hundred_children_without_partial_collection() { + let temp = temporary_work_dir("github-collection-child-limit"); + let checkout = temp.join("checkout"); + for index in 0..=MAX_REMOTE_COLLECTION_CHILDREN { + write_skill(&checkout, &format!("skill-{index:03}")); + } + let paths = managed_paths(temp.join("managed")); + + let error = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap_err(); + + assert!(error.contains("exceeding the safety limit"), "{error}"); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_result_routes_single_skill_and_bare_repository_structurally() { + let single = preview_github_skill_collection_result( + PreviewGithubSkillCollectionRequest { + source_url: "https://github.com/acme/repo/tree/main/skills/demo".to_string(), + }, + temporary_work_dir("github-collection-route-single"), + ) + .unwrap(); + let single_json = serde_json::to_value(single).unwrap(); + assert_eq!(single_json["kind"], "single_skill"); + assert!(single_json["message"] + .as_str() + .unwrap() + .contains("single-skill")); + + let bare = preview_github_skill_collection_result( + PreviewGithubSkillCollectionRequest { + source_url: "https://github.com/acme/repo".to_string(), + }, + temporary_work_dir("github-collection-route-bare"), + ) + .unwrap(); + let bare_json = serde_json::to_value(bare).unwrap(); + assert_eq!(bare_json["kind"], "explicit_reference_required"); + assert!(bare_json["message"] + .as_str() + .unwrap() + .contains("/tree/")); + } + + #[test] + fn preview_keeps_valid_children_when_one_child_is_malformed() { + let temp = temporary_work_dir("github-collection-invalid-child-test"); + let checkout = temp.join("checkout"); + write_skill(&checkout, "valid"); + let invalid = checkout.join("skills/invalid"); + fs::create_dir_all(&invalid).unwrap(); + fs::write(invalid.join("SKILL.md"), "---\nname: invalid\n").unwrap(); + let paths = managed_paths(temp.join("managed")); + + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + + assert_eq!(preview.collection.children.len(), 1); + assert_eq!(preview.collection.children[0].name, "valid"); + assert_eq!(preview.diagnostics.invalid_child_count, 1); + assert!(preview + .errors + .iter() + .any(|error| error.source_path == Path::new("skills/invalid"))); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_rejects_empty_repository_without_managed_writes() { + let temp = temporary_work_dir("github-collection-empty-test"); + let checkout = temp.join("checkout"); + fs::create_dir_all(&checkout).unwrap(); + fs::write(checkout.join("README.md"), "no skills\n").unwrap(); + let paths = managed_paths(temp.join("managed")); + + let error = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap_err(); + + assert!(error.contains("no valid SKILL.md")); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_supports_repository_root_skill_without_creating_state() { + let temp = temporary_work_dir("github-collection-root-skill-test"); + let checkout = temp.join("checkout"); + write_root_skill(&checkout, "root-skill"); + let paths = managed_paths(temp.join("managed")); + + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + + assert_eq!(preview.collection.children.len(), 1); + assert_eq!(preview.collection.children[0].relative_path, ""); + assert!(preview.collection.children[0].diff.contains("SKILL.md")); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_rejects_root_and_nested_skill_overlap() { + let temp = temporary_work_dir("github-collection-root-overlap-test"); + let checkout = temp.join("checkout"); + write_root_skill(&checkout, "root-skill"); + write_skill(&checkout, "nested-skill"); + let paths = managed_paths(temp.join("managed")); + + let error = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap_err(); + + assert!(error.contains("overlapping skill directories")); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_rejects_parent_and_nested_skill_overlap() { + let temp = temporary_work_dir("github-collection-parent-overlap-test"); + let checkout = temp.join("checkout"); + write_skill(&checkout, "parent"); + write_skill(&checkout.join("skills/parent"), "nested"); + let paths = managed_paths(temp.join("managed")); + + let error = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap_err(); + + assert!(error.contains("overlapping skill directories")); + assert!(!paths.database_path.exists()); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_marks_duplicate_child_names_as_unselectable() { + let temp = temporary_work_dir("github-collection-duplicate-name-test"); + let checkout = temp.join("checkout"); + write_skill(&checkout, "alpha"); + fs::create_dir_all(checkout.join("other/alpha")).unwrap(); + fs::write( + checkout.join("other/alpha/SKILL.md"), + "---\nname: alpha\ndescription: Duplicate\n---\n", + ) + .unwrap(); + let paths = managed_paths(temp.join("managed")); + + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + + assert_eq!(preview.collection.children.len(), 2); + assert!(preview + .collection + .children + .iter() + .all(|child| child.conflict.as_deref() == Some("Duplicate skill name in repository"))); + assert_eq!(preview.diagnostics.duplicate_name_count, 1); + let _ = fs::remove_dir_all(temp); + } + + #[test] + fn preview_keeps_valid_children_when_tree_contains_a_symlink_diagnostic() { + let temp = temporary_work_dir("github-collection-symlink-test"); + let checkout = temp.join("checkout"); + write_skill(&checkout, "alpha"); + fs::write(checkout.join("README.md"), "readme").unwrap(); + #[cfg(unix)] + std::os::unix::fs::symlink("outside", checkout.join("unsafe-link")).unwrap(); + let paths = managed_paths(temp.join("managed")); + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + assert_eq!(preview.collection.children.len(), 1); + assert_eq!(preview.collection.errors.len(), 1); + let _ = fs::remove_dir_all(temp); + } + + #[cfg(unix)] + #[test] + fn preview_rejects_symlink_inside_skill_but_keeps_other_valid_children() { + let temp = temporary_work_dir("github-collection-skill-symlink-test"); + let checkout = temp.join("checkout"); + write_skill(&checkout, "alpha"); + write_skill(&checkout, "beta"); + std::os::unix::fs::symlink("outside", checkout.join("skills/alpha/reference.txt")).unwrap(); + let paths = managed_paths(temp.join("managed")); + + let preview = build_github_skill_collection_preview( + &source(), + "0123456789012345678901234567890123456789", + &checkout, + &paths, + 1, + Instant::now(), + ) + .unwrap(); + + assert_eq!(preview.collection.children.len(), 1); + assert_eq!(preview.collection.children[0].name, "beta"); + assert!(preview + .errors + .iter() + .any(|error| error.error.contains("symlinks"))); + let _ = fs::remove_dir_all(temp); + } +} diff --git a/crates/skillbox-core/src/import.rs b/crates/skillbox-core/src/import.rs index ffc9de9..6bfa3a0 100644 --- a/crates/skillbox-core/src/import.rs +++ b/crates/skillbox-core/src/import.rs @@ -620,7 +620,7 @@ pub(crate) fn import_one_candidate( ) } -fn import_one_candidate_unlogged( +pub(crate) fn import_one_candidate_unlogged( paths: &ManagedPaths, item: ImportRequestItem, ) -> Result { diff --git a/crates/skillbox-core/src/installed_sources.rs b/crates/skillbox-core/src/installed_sources.rs index ee20a89..a540e70 100644 --- a/crates/skillbox-core/src/installed_sources.rs +++ b/crates/skillbox-core/src/installed_sources.rs @@ -153,10 +153,12 @@ pub(crate) fn discover_installed_source_collections( source_kind: ImportCandidateCollectionSourceKind::InstalledSource, canonical_worktree_root: PathBuf::new(), canonical_repository_id: PathBuf::new(), - origin_url: Some(source_url), + origin_url: Some(source_url.clone()), branch: None, detached: false, reviewed_head_sha: None, + source_url: Some(source_url), + requested_reference: None, children, errors: Vec::new(), }) @@ -356,6 +358,7 @@ fn installed_source_child( real_path: candidate.real_path.clone(), content_hash: candidate.content_hash.clone(), snapshot_hash: variant.snapshot_hash.clone(), + diff: String::new(), import_status: candidate.import_status, conflict: candidate.conflict.clone(), usage_count: candidate.usage_count, diff --git a/crates/skillbox-core/src/lib.rs b/crates/skillbox-core/src/lib.rs index e12dc63..9333c91 100644 --- a/crates/skillbox-core/src/lib.rs +++ b/crates/skillbox-core/src/lib.rs @@ -75,6 +75,7 @@ mod db; mod doctor; mod fsutil; mod git_sync; +mod github_collections; mod hooks; mod import; mod inbound_git_sync; @@ -102,6 +103,7 @@ pub(crate) use db::*; pub use doctor::*; pub(crate) use fsutil::*; pub use git_sync::*; +pub use github_collections::*; pub use hooks::*; pub use import::*; pub use inbound_git_sync::*; diff --git a/crates/skillbox-core/src/marketplace.rs b/crates/skillbox-core/src/marketplace.rs index e7c35f2..887db82 100644 --- a/crates/skillbox-core/src/marketplace.rs +++ b/crates/skillbox-core/src/marketplace.rs @@ -51,7 +51,13 @@ pub(crate) fn fetch_remote_source_skill_path( fs::remove_dir_all(checkout).map_err(|error| error.to_string())?; } - match git.fetch_ref_path(repo_url, reference, candidate, checkout) { + match git.fetch_ref_path_with_timeout_allow_legacy_tree( + repo_url, + reference, + candidate, + checkout, + std::time::Duration::from_secs(30), + ) { Ok(sha) => return Ok((sha, candidate.clone())), Err(error) => { if first_error.is_none() { diff --git a/crates/skillbox-core/src/remote.rs b/crates/skillbox-core/src/remote.rs index 552d0a7..b8f8415 100644 --- a/crates/skillbox-core/src/remote.rs +++ b/crates/skillbox-core/src/remote.rs @@ -598,12 +598,22 @@ fn fetch_github_skill_source_snapshot( root_snapshot: &Path, ) -> Result<(String, PathBuf)> { if source.is_root { - let sha = git.fetch_ref_tree(&source.repo_url, &source.reference, checkout)?; + let sha = git.fetch_ref_tree_with_timeout_allow_legacy_tree( + &source.repo_url, + &source.reference, + checkout, + Duration::from_secs(30), + )?; copy_skill_dir_from_checkout(checkout, root_snapshot, checkout)?; Ok((sha, root_snapshot.to_path_buf())) } else { - let sha = - git.fetch_ref_path(&source.repo_url, &source.reference, &source.path, checkout)?; + let sha = git.fetch_ref_path_with_timeout_allow_legacy_tree( + &source.repo_url, + &source.reference, + &source.path, + checkout, + Duration::from_secs(30), + )?; Ok((sha, checkout.join(&source.path))) } } @@ -1268,10 +1278,12 @@ pub(crate) fn remote_skill_path_changed( }; let checkout = temp.join("checkout"); let latest_path = if source_is_root { - git.fetch_ref_tree_with_timeout(repo_url, latest_sha, &checkout, timeout)?; + git.fetch_ref_tree_with_timeout_allow_legacy_tree( + repo_url, latest_sha, &checkout, timeout, + )?; checkout } else { - git.fetch_ref_path_with_timeout( + git.fetch_ref_path_with_timeout_allow_legacy_tree( repo_url, latest_sha, source_repo_path, @@ -1561,10 +1573,21 @@ pub(crate) fn remote_version_preview_target( let checkout = temp.join("checkout"); let git = skillbox_git::GitService::new(); if source_is_root { - git.fetch_ref_tree(&repo_url, to_version, &checkout)?; + git.fetch_ref_tree_with_timeout_allow_legacy_tree( + &repo_url, + to_version, + &checkout, + Duration::from_secs(30), + )?; Ok(checkout) } else { - git.fetch_ref_path(&repo_url, to_version, &source_path, &checkout)?; + git.fetch_ref_path_with_timeout_allow_legacy_tree( + &repo_url, + to_version, + &source_path, + &checkout, + Duration::from_secs(30), + )?; Ok(checkout.join(source_path)) } } @@ -1922,7 +1945,12 @@ pub(crate) fn ensure_github_version_snapshot( let result = (|| { let checkout = temp.join("checkout"); let git = skillbox_git::GitService::new(); - git.fetch_ref_tree(&repo_url, target_sha, &checkout)?; + git.fetch_ref_tree_with_timeout_allow_legacy_tree( + &repo_url, + target_sha, + &checkout, + Duration::from_secs(30), + )?; let fetched_skill_path = if source_is_root { checkout.clone() } else { diff --git a/crates/skillbox-core/src/tests.rs b/crates/skillbox-core/src/tests.rs index 3119077..cf9299a 100644 --- a/crates/skillbox-core/src/tests.rs +++ b/crates/skillbox-core/src/tests.rs @@ -171,7 +171,8 @@ fn database_initialization_records_ordered_schema_migrations() { (5, "canonical_usage_agent_ids".to_string()), (6, "runtime_profiles".to_string()), (7, "usage_evidence_classification".to_string()), - (8, "skill_collections".to_string()) + (8, "skill_collections".to_string()), + (9, "github_skill_collections".to_string()) ] ); assert_eq!( @@ -210,7 +211,7 @@ fn database_initialization_records_ordered_schema_migrations() { } #[test] -fn schema_v8_collection_migration_is_idempotent_for_existing_database() { +fn schema_v9_github_collection_migration_is_idempotent_for_existing_database() { let root = temp_dir("skill-collections-v8-migration"); let managed_root = root.join("SkillBox"); let paths = ensure_managed_layout(&managed_root).unwrap(); @@ -221,7 +222,7 @@ fn schema_v8_collection_migration_is_idempotent_for_existing_database() { " DROP TABLE skill_collection_members; DROP TABLE skill_collections; - DELETE FROM schema_migrations WHERE version = 8; + DELETE FROM schema_migrations WHERE version >= 8; ", ) .unwrap(); @@ -237,6 +238,14 @@ fn schema_v8_collection_migration_is_idempotent_for_existing_database() { assert!(table_column_names(&connection, "skill_collections") .unwrap() .contains(&"reviewed_head_sha".to_string())); + for column in ["source_kind", "source_url", "requested_reference"] { + assert!( + table_column_names(&connection, "skill_collections") + .unwrap() + .contains(&column.to_string()), + "missing {column}" + ); + } assert!(table_column_names(&connection, "skill_collection_members") .unwrap() .contains(&"managed_skill_name".to_string())); @@ -7774,6 +7783,639 @@ fn install_github_root_skill_rejects_preview_after_branch_advances() { assert_eq!(indexed, None); } +#[test] +fn github_collection_preview_and_apply_selects_children_at_one_reviewed_sha() { + let root = temp_dir("github-collection-apply"); + let managed_root = root.join("SkillBox"); + let (remote, _work) = bare_remote_with_multiple_skill_content( + "github-collection-apply-origin", + &["alpha", "beta"], + ); + let _rewrite = github_repo_rewrite("acme", "github-collection-apply", &remote); + let source_url = "https://github.com/acme/github-collection-apply/tree/main"; + let preview = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + assert_eq!(preview.diagnostics.fetch_count, 1); + assert_eq!(preview.collection.children.len(), 2); + let reviewed_sha = preview.collection.reviewed_head_sha.clone().unwrap(); + let selections = preview + .collection + .children + .iter() + .map(|child| ImportCollectionChildSelection { + relative_path: child.relative_path.clone(), + group_id: child.group_id.clone(), + variant_id: child.variant_id.clone(), + skill_type: SkillKind::User, + }) + .collect(); + + let result = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: preview.collection.id, + preview_id: preview.collection.preview_id, + selections, + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap(); + + assert_eq!(result.imported.len(), 2); + assert_eq!( + result.collection.source_kind, + ImportCandidateCollectionSourceKind::GithubRemote + ); + assert_eq!( + result.collection.reviewed_head_sha.as_deref(), + Some(reviewed_sha.as_str()) + ); + let paths = managed_paths(&managed_root); + assert!(paths.user_skills_root.join("alpha/SKILL.md").exists()); + assert!(paths.user_skills_root.join("beta/SKILL.md").exists()); + let connection = open_database(&paths.database_path).unwrap(); + let (source_kind, source_url, member_count): (String, String, i64) = connection + .query_row( + "SELECT c.source_kind, c.source_url, (SELECT COUNT(*) FROM skill_collection_members m WHERE m.collection_id = c.id) FROM skill_collections c WHERE c.id = ?1", + [&result.collection.id], + |row| Ok((row.get(0)?, row.get(1)?, row.get(2)?)), + ) + .unwrap(); + assert_eq!(source_kind, "github_remote"); + assert_eq!( + source_url, + "https://github.com/acme/github-collection-apply/tree/main" + ); + assert_eq!(member_count, 2); +} + +#[test] +fn github_collection_incremental_apply_preserves_members_at_one_reviewed_sha() { + let root = temp_dir("github-collection-incremental-apply"); + let managed_root = root.join("SkillBox"); + let (remote, work) = bare_remote_with_multiple_skill_content( + "github-collection-incremental-apply-origin", + &["alpha", "beta"], + ); + let _rewrite = github_repo_rewrite("acme", "github-collection-incremental-apply", &remote); + let source_url = "https://github.com/acme/github-collection-incremental-apply/tree/main"; + + let first = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + let reviewed_sha = first.collection.reviewed_head_sha.clone(); + let alpha = first + .collection + .children + .iter() + .find(|child| child.name == "alpha") + .unwrap() + .clone(); + let first_result = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: first.collection.id.clone(), + preview_id: first.collection.preview_id, + selections: vec![ImportCollectionChildSelection { + relative_path: alpha.relative_path.clone(), + group_id: alpha.group_id, + variant_id: alpha.variant_id, + skill_type: SkillKind::User, + }], + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap(); + assert_eq!(first_result.collection.members.len(), 1); + + let second = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + assert_eq!(second.collection.reviewed_head_sha, reviewed_sha); + let beta = second + .collection + .children + .iter() + .find(|child| child.name == "beta") + .unwrap() + .clone(); + let second_result = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: second.collection.id.clone(), + preview_id: second.collection.preview_id, + selections: vec![ImportCollectionChildSelection { + relative_path: beta.relative_path, + group_id: beta.group_id, + variant_id: beta.variant_id, + skill_type: SkillKind::User, + }], + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap(); + + assert_eq!(second_result.collection.members.len(), 2); + assert_eq!( + list_skill_collections(&managed_root).unwrap()[0] + .members + .len(), + 2 + ); + + fs::write( + work.join("skills/beta/SKILL.md"), + "---\nname: beta\ndescription: Changed\n---\n\n# Changed\n", + ) + .unwrap(); + run_git(&work, &["add", "."]); + run_git( + &work, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Advance collection", + ], + ); + run_git(&work, &["push", "origin", "main"]); + let third = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + let changed_beta = third + .collection + .children + .iter() + .find(|child| child.name == "beta") + .unwrap() + .clone(); + let error = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: third.collection.id, + preview_id: third.collection.preview_id, + selections: vec![ImportCollectionChildSelection { + relative_path: changed_beta.relative_path, + group_id: changed_beta.group_id, + variant_id: changed_beta.variant_id, + skill_type: SkillKind::User, + }], + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap_err(); + assert!( + error.contains("different reviewed SHA") || error.contains("Managed target"), + "{error}" + ); + assert_eq!( + list_skill_collections(&managed_root).unwrap()[0] + .members + .len(), + 2 + ); +} + +#[test] +fn github_collection_identity_is_stable_across_commits_on_one_ref() { + let root = temp_dir("github-collection-stable-identity"); + let managed_root = root.join("SkillBox"); + let (remote, work) = bare_remote_with_multiple_skill_content( + "github-collection-stable-identity-origin", + &["alpha", "beta"], + ); + let _rewrite = github_repo_rewrite("acme", "github-collection-stable-identity", &remote); + let source_url = + "https://github.com/acme/github-collection-stable-identity/tree/main".to_string(); + + let first = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.clone(), + }, + &managed_root, + ) + .unwrap(); + fs::write( + work.join("skills/alpha/SKILL.md"), + "---\nname: alpha\ndescription: Changed\n---\n\n# Changed\n", + ) + .unwrap(); + run_git(&work, &["add", "."]); + run_git( + &work, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Change alpha", + ], + ); + run_git(&work, &["push", "origin", "main"]); + let second = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { source_url }, + &managed_root, + ) + .unwrap(); + + assert_eq!(first.collection.id, second.collection.id); + assert_ne!(first.collection.preview_id, second.collection.preview_id); + assert_ne!( + first.collection.reviewed_head_sha, + second.collection.reviewed_head_sha + ); +} + +#[test] +fn collection_batch_failure_rolls_back_remote_current_index_and_audits_one_operation() { + let root = temp_dir("collection-batch-failure-audit"); + let repository = root.join("collection"); + let managed_root = root.join("SkillBox"); + fs::create_dir_all(&repository).unwrap(); + run_git(&repository, &["init", "-b", "main"]); + make_skill(&repository.join("skills/alpha"), "alpha", "Alpha"); + run_git(&repository, &["add", "."]); + run_git( + &repository, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Collection", + ], + ); + + let scan = scan_import_candidates(std::slice::from_ref(&repository), &managed_root).unwrap(); + let mut collection = scan.collections.into_iter().next().unwrap(); + let child = collection.children[0].clone(); + collection.source_kind = ImportCandidateCollectionSourceKind::GithubRemote; + collection.source_url = Some("https://github.com/acme/collection/tree/main".to_string()); + collection.requested_reference = Some("main".to_string()); + let paths = ensure_managed_layout(managed_root.clone()).unwrap(); + let missing = repository.join("skills/missing"); + let external_missing_root = paths.remote_skills_root.join("missing"); + make_skill( + &external_missing_root.join("versions/external"), + "missing", + "External content that rollback must preserve", + ); + let error = apply_collection_import_with_audit( + &paths, + &collection, + std::slice::from_ref(&child), + vec![ + ImportRequestItem { + source_path: child.source_path.clone(), + skill_type: SkillKind::Remote, + deploy_back_to_source: false, + }, + ImportRequestItem { + source_path: missing, + skill_type: SkillKind::Remote, + deploy_back_to_source: false, + }, + ], + "test", + ) + .unwrap_err(); + + assert!(error.contains("Collection import did not complete")); + assert!(!paths.remote_skills_root.join("alpha").exists()); + assert!(external_missing_root + .join("versions/external/SKILL.md") + .is_file()); + let operations = list_operations(OperationFilter::default(), &managed_root) + .unwrap() + .operations; + let collection_operations = operations + .iter() + .filter(|operation| operation.operation_type == "import_collection") + .collect::>(); + assert_eq!(collection_operations.len(), 1); + assert!(operations + .iter() + .all(|operation| operation.operation_type != "import_candidate")); + assert_eq!(collection_operations[0].status, OperationStatus::Failed); + assert_eq!( + collection_operations[0] + .payload + .get("partialRecovery") + .and_then(serde_json::Value::as_bool), + Some(false) + ); + assert!(list_skill_collections(&managed_root).unwrap().is_empty()); +} + +#[cfg(unix)] +#[test] +fn collection_batch_rollback_preserves_same_content_identity_replacement() { + let root = temp_dir("collection-batch-identity-replacement"); + let repository = root.join("collection"); + let managed_root = root.join("SkillBox"); + fs::create_dir_all(&repository).unwrap(); + run_git(&repository, &["init", "-b", "main"]); + make_skill(&repository.join("skills/alpha"), "alpha", "Alpha"); + run_git(&repository, &["add", "."]); + run_git( + &repository, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Collection", + ], + ); + + let scan = scan_import_candidates(std::slice::from_ref(&repository), &managed_root).unwrap(); + let mut collection = scan.collections.into_iter().next().unwrap(); + let child = collection.children[0].clone(); + collection.source_kind = ImportCandidateCollectionSourceKind::GithubRemote; + collection.source_url = Some("https://github.com/acme/collection/tree/main".to_string()); + collection.requested_reference = Some("main".to_string()); + let paths = ensure_managed_layout(managed_root.clone()).unwrap(); + let missing = repository.join("skills/missing"); + let _hook = install_collection_import_test_hook(|imported| { + assert_eq!(imported.len(), 1); + let imported_path = &imported[0].managed_path; + let replacement_path = imported_path.with_file_name("external-replacement"); + let skill_md = fs::read(imported_path.join("SKILL.md")).unwrap(); + fs::rename(imported_path, &replacement_path).unwrap(); + fs::create_dir_all(imported_path).unwrap(); + fs::write(imported_path.join("SKILL.md"), skill_md).unwrap(); + + let current = imported_path + .parent() + .and_then(Path::parent) + .unwrap() + .join("current"); + fs::remove_file(¤t).unwrap(); + std::os::unix::fs::symlink(&replacement_path, ¤t).unwrap(); + }); + + let error = apply_collection_import_with_audit( + &paths, + &collection, + std::slice::from_ref(&child), + vec![ + ImportRequestItem { + source_path: child.source_path.clone(), + skill_type: SkillKind::Remote, + deploy_back_to_source: false, + }, + ImportRequestItem { + source_path: missing, + skill_type: SkillKind::Remote, + deploy_back_to_source: false, + }, + ], + "test", + ) + .unwrap_err(); + + assert!(error.contains("filesystem identity changed"), "{error}"); + let replacement_path = paths + .remote_skills_root + .join("alpha/versions/external-replacement"); + assert!(replacement_path.join("SKILL.md").is_file()); + let current = paths.remote_skills_root.join("alpha/current"); + assert_eq!( + fs::canonicalize(current).unwrap(), + fs::canonicalize(replacement_path).unwrap() + ); + let operations = list_operations(OperationFilter::default(), &managed_root) + .unwrap() + .operations; + let collection_operation = operations + .iter() + .find(|operation| operation.operation_type == "import_collection") + .unwrap(); + assert_eq!(collection_operation.status, OperationStatus::Failed); + assert_eq!( + collection_operation + .payload + .get("partialRecovery") + .and_then(serde_json::Value::as_bool), + Some(true) + ); + assert!(list_skill_collections(&managed_root).unwrap().is_empty()); +} + +#[test] +fn github_collection_apply_rejects_new_sha_before_new_child_writes() { + let root = temp_dir("github-collection-new-child-sha"); + let managed_root = root.join("SkillBox"); + let (remote, work) = bare_remote_with_multiple_skill_content( + "github-collection-new-child-sha-origin", + &["alpha", "beta"], + ); + let _rewrite = github_repo_rewrite("acme", "github-collection-new-child-sha", &remote); + let source_url = "https://github.com/acme/github-collection-new-child-sha/tree/main"; + + let first = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + let alpha = first + .collection + .children + .iter() + .find(|child| child.name == "alpha") + .unwrap() + .clone(); + let first_result = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: first.collection.id.clone(), + preview_id: first.collection.preview_id, + selections: vec![ImportCollectionChildSelection { + relative_path: alpha.relative_path, + group_id: alpha.group_id, + variant_id: alpha.variant_id, + skill_type: SkillKind::User, + }], + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap(); + let operations_before = list_operations(OperationFilter::default(), &managed_root) + .unwrap() + .operations + .len(); + + make_skill(&work.join("skills/charlie"), "charlie", "Charlie"); + run_git(&work, &["add", "."]); + run_git( + &work, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Add Charlie", + ], + ); + run_git(&work, &["push", "origin", "main"]); + + let second = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + assert_ne!( + first_result.collection.reviewed_head_sha, + second.collection.reviewed_head_sha + ); + let charlie = second + .collection + .children + .iter() + .find(|child| child.name == "charlie") + .unwrap() + .clone(); + let error = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: second.collection.id, + preview_id: second.collection.preview_id, + selections: vec![ImportCollectionChildSelection { + relative_path: charlie.relative_path, + group_id: charlie.group_id, + variant_id: charlie.variant_id, + skill_type: SkillKind::User, + }], + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap_err(); + + assert!(error.contains("different reviewed SHA"), "{error}"); + assert!(!managed_root.join("user-skills/charlie").exists()); + let paths = managed_paths(&managed_root); + assert!(!managed_index_contains(&paths.database_path, "charlie").unwrap()); + assert_eq!( + list_skill_collections(&managed_root).unwrap()[0] + .members + .iter() + .map(|member| member.managed_skill_name.as_str()) + .collect::>(), + vec!["alpha"] + ); + assert_eq!( + list_operations(OperationFilter::default(), &managed_root) + .unwrap() + .operations + .len(), + operations_before + ); +} + +#[test] +fn github_collection_apply_rejects_new_remote_sha_before_managed_writes() { + let root = temp_dir("github-collection-stale"); + let managed_root = root.join("SkillBox"); + let (remote, work) = bare_remote_with_multiple_skill_content( + "github-collection-stale-origin", + &["alpha", "beta"], + ); + let _rewrite = github_repo_rewrite("acme", "github-collection-stale", &remote); + let source_url = "https://github.com/acme/github-collection-stale/tree/main"; + let preview = preview_github_skill_collection( + PreviewGithubSkillCollectionRequest { + source_url: source_url.to_string(), + }, + &managed_root, + ) + .unwrap(); + fs::write( + work.join("skills/alpha/SKILL.md"), + "---\nname: alpha\ndescription: Changed\n---\n", + ) + .unwrap(); + run_git(&work, &["add", "."]); + run_git( + &work, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Advance collection", + ], + ); + run_git(&work, &["push", "origin", "main"]); + let selections = preview + .collection + .children + .iter() + .map(|child| ImportCollectionChildSelection { + relative_path: child.relative_path.clone(), + group_id: child.group_id.clone(), + variant_id: child.variant_id.clone(), + skill_type: SkillKind::User, + }) + .collect(); + + let error = apply_github_skill_collection( + GithubSkillCollectionApplyRequest { + source_url: source_url.to_string(), + collection_id: preview.collection.id, + preview_id: preview.collection.preview_id, + selections, + actor: "test".to_string(), + }, + &managed_root, + ) + .unwrap_err(); + + assert!(error.contains("stale")); + assert!(!managed_root.join("user-skills/alpha").exists()); + assert!(!managed_root.join("user-skills/beta").exists()); + assert!(!managed_root.join("skillbox.sqlite").exists()); +} + #[test] fn install_github_warning_target_requires_confirmation_before_any_install_state() { let root = temp_dir("install-github-warning-confirmation"); @@ -11967,6 +12609,40 @@ description: \"{description}\" remote } +fn bare_remote_with_multiple_skill_content(label: &str, names: &[&str]) -> (PathBuf, PathBuf) { + let remote = bare_remote(label); + let work = temp_dir(&format!("{label}-work")); + run_git(&work, &["init", "-b", "main"]); + for name in names { + let skill_dir = work.join("skills").join(name); + fs::create_dir_all(&skill_dir).unwrap(); + fs::write( + skill_dir.join("SKILL.md"), + format!("---\nname: {name}\ndescription: \"{name} skill\"\n---\n\n# {name}\n"), + ) + .unwrap(); + } + run_git(&work, &["add", "."]); + run_git( + &work, + &[ + "-c", + "user.name=SkillBox", + "-c", + "user.email=skillbox@example.invalid", + "commit", + "-m", + "Add skills", + ], + ); + run_git( + &work, + &["remote", "add", "origin", remote.to_str().unwrap()], + ); + run_git(&work, &["push", "-u", "origin", "main"]); + (remote, work) +} + fn bare_remote_with_root_skill_content( label: &str, skill_name: &str, @@ -12025,54 +12701,23 @@ static GIT_CONFIG_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); struct GitConfigRewriteGuard { _lock: std::sync::MutexGuard<'static, ()>, - previous: Vec<(&'static str, Option)>, + _rewrite: skillbox_git::TestTrustedUrlRewriteGuard, } impl Drop for GitConfigRewriteGuard { - fn drop(&mut self) { - let previous_count = self - .previous - .iter() - .find_map(|(key, value)| (*key == "GIT_CONFIG_COUNT").then(|| value.clone())) - .flatten(); - std::env::remove_var("GIT_CONFIG_COUNT"); - for (key, value) in self - .previous - .drain(..) - .filter(|(key, _)| *key != "GIT_CONFIG_COUNT") - { - match value { - Some(value) => std::env::set_var(key, value), - None => std::env::remove_var(key), - } - } - if let Some(value) = previous_count { - std::env::set_var("GIT_CONFIG_COUNT", value); - } - } + fn drop(&mut self) {} } fn github_repo_rewrite(owner: &str, repo: &str, remote: &std::path::Path) -> GitConfigRewriteGuard { let lock = GIT_CONFIG_LOCK.lock().unwrap(); - let keys = ["GIT_CONFIG_COUNT", "GIT_CONFIG_KEY_0", "GIT_CONFIG_VALUE_0"]; - let previous = keys - .into_iter() - .map(|key| (key, std::env::var_os(key))) - .collect::>(); - - std::env::set_var( - "GIT_CONFIG_KEY_0", - format!("url.file://{}.insteadOf", remote.display()), - ); - std::env::set_var( - "GIT_CONFIG_VALUE_0", + let rewrite = skillbox_git::test_trusted_url_rewrite( + format!("file://{}", remote.display()), format!("https://github.com/{owner}/{repo}.git"), ); - std::env::set_var("GIT_CONFIG_COUNT", "1"); GitConfigRewriteGuard { _lock: lock, - previous, + _rewrite: rewrite, } } diff --git a/crates/skillbox-core/src/types.rs b/crates/skillbox-core/src/types.rs index df9ad92..b4d5540 100644 --- a/crates/skillbox-core/src/types.rs +++ b/crates/skillbox-core/src/types.rs @@ -1104,6 +1104,7 @@ pub struct ImportCandidateCollectionChild { pub real_path: PathBuf, pub content_hash: String, pub snapshot_hash: String, + pub diff: String, pub import_status: ImportCandidateStatus, pub conflict: Option, pub usage_count: usize, @@ -1127,6 +1128,8 @@ pub struct ImportCandidateCollection { pub branch: Option, pub detached: bool, pub reviewed_head_sha: Option, + pub source_url: Option, + pub requested_reference: Option, pub children: Vec, pub errors: Vec, } @@ -1136,6 +1139,7 @@ pub struct ImportCandidateCollection { pub enum ImportCandidateCollectionSourceKind { GitWorktree, InstalledSource, + GithubRemote, } #[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -1252,6 +1256,9 @@ pub struct SkillCollection { pub branch: Option, pub detached: bool, pub reviewed_head_sha: Option, + pub source_kind: ImportCandidateCollectionSourceKind, + pub source_url: Option, + pub requested_reference: Option, pub available: bool, pub members: Vec, } @@ -1261,6 +1268,55 @@ pub struct ImportCollectionApplyResult { pub collection: SkillCollection, pub imported: Vec, pub errors: Vec, + #[serde(default)] + pub warnings: Vec, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct PreviewGithubSkillCollectionRequest { + pub source_url: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct GithubSkillCollectionApplyRequest { + pub source_url: String, + pub collection_id: String, + pub preview_id: String, + pub selections: Vec, + pub actor: String, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct GithubSkillCollectionDiagnostics { + pub fetch_count: usize, + pub child_count: usize, + pub valid_child_count: usize, + pub invalid_child_count: usize, + pub duplicate_name_count: usize, + pub elapsed_ms: u64, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct GithubSkillCollectionPreview { + pub collection: ImportCandidateCollection, + pub groups: Vec, + pub errors: Vec, + pub diagnostics: GithubSkillCollectionDiagnostics, +} + +#[derive(Debug, Clone, Serialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +pub enum GithubSkillCollectionPreviewResult { + Collection { + preview: Box, + }, + SingleSkill { + message: String, + }, + ExplicitReferenceRequired { + message: String, + }, } #[derive(Debug, Copy, Clone, PartialEq, Eq, Serialize, Deserialize)] diff --git a/crates/skillbox-git/src/lib.rs b/crates/skillbox-git/src/lib.rs index c44f646..ae2847d 100644 --- a/crates/skillbox-git/src/lib.rs +++ b/crates/skillbox-git/src/lib.rs @@ -8,13 +8,22 @@ use std::process::{Command, Output, Stdio}; use std::sync::atomic::{AtomicU64, Ordering}; use std::time::{Duration, Instant}; +thread_local! { + static TEST_TRUSTED_URL_REWRITES: std::cell::RefCell> = const { + std::cell::RefCell::new(Vec::new()) + }; +} + const DEFAULT_LS_REMOTE_TIMEOUT: Duration = Duration::from_secs(30); const FETCH_REF_TIMEOUT: Duration = Duration::from_secs(30); const PUSH_TIMEOUT: Duration = Duration::from_secs(30); const LOCAL_GIT_TIMEOUT: Duration = Duration::from_secs(30); const MAX_REF_DIFF_FILES: usize = 500; const MAX_DIFF_BYTES_PER_FILE: usize = 256 * 1024; -const MAX_TREE_ENTRIES: usize = 20_000; +pub const MAX_STRICT_TREE_ENTRIES: usize = 20_000; +pub const MAX_STRICT_TREE_FILE_BYTES: u64 = 8 * 1024 * 1024; +pub const MAX_STRICT_TREE_TOTAL_BYTES: u64 = 64 * 1024 * 1024; +const MAX_TREE_ENTRIES: usize = MAX_STRICT_TREE_ENTRIES; const MAX_SHOW_FILE_BYTES: usize = 2 * 1024 * 1024; const BACKUP_REF_PREFIX: &str = "refs/skillbox/backups/"; static COMMAND_OUTPUT_COUNTER: AtomicU64 = AtomicU64::new(0); @@ -75,6 +84,12 @@ pub struct GitTreeEntry { pub path: String, } +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GitTreeFetchResult { + pub resolved_sha: String, + pub fetch_count: usize, +} + #[derive(Debug, Clone, PartialEq, Eq)] pub struct GitRepositoryIdentity { pub worktree_root: std::path::PathBuf, @@ -115,6 +130,33 @@ pub struct GitService { preflight_timeout: Duration, } +#[doc(hidden)] +pub struct TestTrustedUrlRewriteGuard { + previous: Vec<(String, String)>, +} + +#[doc(hidden)] +pub fn test_trusted_url_rewrite( + replacement: impl Into, + source_prefix: impl Into, +) -> TestTrustedUrlRewriteGuard { + let rewrite = (replacement.into(), source_prefix.into()); + let previous = TEST_TRUSTED_URL_REWRITES.with(|rewrites| { + let mut rewrites = rewrites.borrow_mut(); + std::mem::replace(&mut *rewrites, vec![rewrite]) + }); + TestTrustedUrlRewriteGuard { previous } +} + +impl Drop for TestTrustedUrlRewriteGuard { + fn drop(&mut self) { + let previous = std::mem::take(&mut self.previous); + TEST_TRUSTED_URL_REWRITES.with(|rewrites| { + *rewrites.borrow_mut() = previous; + }); + } +} + impl Default for GitService { fn default() -> Self { Self::new() @@ -1397,6 +1439,14 @@ impl GitService { validate_git_remote_arg(repo_url)?; validate_git_reference_arg(reference)?; let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, true); + TEST_TRUSTED_URL_REWRITES.with(|rewrites| { + for (replacement, source_prefix) in rewrites.borrow().iter() { + command + .arg("-c") + .arg(format!("url.{replacement}.insteadOf={source_prefix}")); + } + }); command .arg("ls-remote") .arg("--") @@ -1443,32 +1493,67 @@ impl GitService { path: &str, checkout_root: impl AsRef, timeout: Duration, + ) -> Result { + self.fetch_ref_path_with_timeout_internal( + repo_url, + reference, + path, + checkout_root, + timeout, + true, + ) + } + + /// Fetch a requested path for legacy single-skill import/update flows. + /// Those flows intentionally snapshot safe internal symlinks; collection + /// previews use the strict full-tree API instead. + pub fn fetch_ref_path_with_timeout_allow_legacy_tree( + &self, + repo_url: &str, + reference: &str, + path: &str, + checkout_root: impl AsRef, + timeout: Duration, + ) -> Result { + self.fetch_ref_path_with_timeout_internal( + repo_url, + reference, + path, + checkout_root, + timeout, + false, + ) + } + + fn fetch_ref_path_with_timeout_internal( + &self, + repo_url: &str, + reference: &str, + path: &str, + checkout_root: impl AsRef, + timeout: Duration, + reject_unsafe_tree_entries: bool, ) -> Result { validate_git_remote_arg(repo_url)?; validate_git_reference_arg(reference)?; let checkout_root = checkout_root.as_ref(); + let deadline = Instant::now() + timeout.min(self.preflight_timeout); fs::create_dir_all(checkout_root).map_err(|error| error.to_string())?; - self.run(checkout_root, &["init", "-b", "main"])?; - self.run(checkout_root, &["remote", "add", "origin", repo_url])?; - self.run_network( - checkout_root, - &["fetch", "--depth", "1", "origin", "--", reference], - timeout, - "git fetch", - )?; - let sha = self - .run(checkout_root, &["rev-parse", "FETCH_HEAD"])? - .trim() - .to_string(); - self.run_owned( + self.run_fetch_setup(checkout_root, repo_url, deadline)?; + self.run_fetch(checkout_root, reference, deadline)?; + if reject_unsafe_tree_entries { + // Validate the object tree while it is still only in Git's object + // database. Checkout is the first step allowed to materialize the + // untrusted remote worktree. + self.validate_fetched_tree(checkout_root, deadline)?; + } + let sha = self.run_fetch_command( checkout_root, - &[ - "checkout".to_string(), - "FETCH_HEAD".to_string(), - "--".to_string(), - path.to_string(), - ], + &["rev-parse", "FETCH_HEAD"], + deadline, + "git rev-parse", )?; + self.checkout_fetched_ref(checkout_root, Some(path), deadline)?; Ok(sha) } @@ -1488,23 +1573,85 @@ impl GitService { checkout_root: impl AsRef, timeout: Duration, ) -> Result { + self.fetch_ref_tree_with_timeout_diagnostics(repo_url, reference, checkout_root, timeout) + .map(|result| result.resolved_sha) + } + + /// Fetch a full tree for the legacy single-skill update path. That path + /// may snapshot a safe, internal symlink as part of the requested skill; + /// collection previews use the strict diagnostics API below instead. + pub fn fetch_ref_tree_with_timeout_allow_legacy_tree( + &self, + repo_url: &str, + reference: &str, + checkout_root: impl AsRef, + timeout: Duration, + ) -> Result { + self.fetch_ref_tree_with_timeout_diagnostics_internal( + repo_url, + reference, + checkout_root, + timeout, + false, + ) + .map(|result| result.resolved_sha) + } + + pub fn fetch_ref_tree_with_diagnostics( + &self, + repo_url: &str, + reference: &str, + checkout_root: impl AsRef, + ) -> Result { + self.fetch_ref_tree_with_timeout_diagnostics( + repo_url, + reference, + checkout_root, + FETCH_REF_TIMEOUT, + ) + } + + pub fn fetch_ref_tree_with_timeout_diagnostics( + &self, + repo_url: &str, + reference: &str, + checkout_root: impl AsRef, + timeout: Duration, + ) -> Result { + self.fetch_ref_tree_with_timeout_diagnostics_internal( + repo_url, + reference, + checkout_root, + timeout, + true, + ) + } + + fn fetch_ref_tree_with_timeout_diagnostics_internal( + &self, + repo_url: &str, + reference: &str, + checkout_root: impl AsRef, + timeout: Duration, + reject_unsafe_tree_entries: bool, + ) -> Result { validate_git_remote_arg(repo_url)?; validate_git_reference_arg(reference)?; let checkout_root = checkout_root.as_ref(); + let deadline = Instant::now() + timeout.min(self.preflight_timeout); fs::create_dir_all(checkout_root).map_err(|error| error.to_string())?; - self.run(checkout_root, &["init", "-b", "main"])?; - self.run(checkout_root, &["remote", "add", "origin", repo_url])?; - self.run_network( + self.run_fetch_setup(checkout_root, repo_url, deadline)?; + self.run_fetch(checkout_root, reference, deadline)?; + if reject_unsafe_tree_entries { + self.validate_fetched_tree(checkout_root, deadline)?; + } + let sha = self.run_fetch_command( checkout_root, - &["fetch", "--depth", "1", "origin", "--", reference], - timeout, - "git fetch", + &["rev-parse", "FETCH_HEAD"], + deadline, + "git rev-parse", )?; - let sha = self - .run(checkout_root, &["rev-parse", "FETCH_HEAD"])? - .trim() - .to_string(); - self.run(checkout_root, &["checkout", "FETCH_HEAD"])?; + self.checkout_fetched_ref(checkout_root, None, deadline)?; let git_metadata = checkout_root.join(".git"); match fs::symlink_metadata(&git_metadata) { Ok(metadata) if metadata.is_dir() => { @@ -1516,7 +1663,129 @@ impl GitService { Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} Err(error) => return Err(error.to_string()), } - Ok(sha) + Ok(GitTreeFetchResult { + resolved_sha: sha, + fetch_count: 1, + }) + } + + fn run_fetch_setup( + &self, + repo: &Path, + repo_url: &str, + deadline: Instant, + ) -> Result<(), String> { + self.run_fetch_command(repo, &["init", "-b", "main"], deadline, "git init")?; + self.run_fetch_command( + repo, + &["remote", "add", "origin", repo_url], + deadline, + "git remote add", + )?; + Ok(()) + } + + fn run_fetch(&self, repo: &Path, reference: &str, deadline: Instant) -> Result<(), String> { + let mut command = self.hardened_network_command(repo, deadline)?; + command.args(["fetch", "--depth", "1", "origin", "--", reference]); + let output = self.command_output_with_timeout( + command, + remaining_git_deadline(deadline)?, + "git fetch", + )?; + if output.status.success() { + Ok(()) + } else { + Err(network_git_error( + "git fetch", + String::from_utf8_lossy(&output.stderr).trim(), + )) + } + } + + fn validate_fetched_tree(&self, repo: &Path, deadline: Instant) -> Result<(), String> { + let listing = self.run_fetch_command_bytes( + repo, + &["ls-tree", "-r", "-z", "--long", "FETCH_HEAD"], + deadline, + "git validate fetched tree", + )?; + validate_strict_tree_listing(&listing) + } + + fn run_fetch_command( + &self, + repo: &Path, + args: &[&str], + deadline: Instant, + label: &str, + ) -> Result { + let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, false); + command + .arg("-c") + .arg("core.hooksPath=/dev/null") + .arg("-C") + .arg(repo) + .args(args) + .env("LC_ALL", "C"); + let output = + self.command_output_with_timeout(command, remaining_git_deadline(deadline)?, label)?; + self.success_stdout(output) + .map(|value| value.trim().to_string()) + } + + fn run_fetch_command_bytes( + &self, + repo: &Path, + args: &[&str], + deadline: Instant, + label: &str, + ) -> Result, String> { + let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, false); + command + .arg("-c") + .arg("core.hooksPath=/dev/null") + .arg("-C") + .arg(repo) + .args(args) + .env("LC_ALL", "C"); + let output = + self.command_output_with_timeout(command, remaining_git_deadline(deadline)?, label)?; + self.success_stdout_bytes(output) + } + + fn checkout_fetched_ref( + &self, + repo: &Path, + path: Option<&str>, + deadline: Instant, + ) -> Result<(), String> { + let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, false); + command + .arg("-c") + .arg("core.hooksPath=/dev/null") + .arg("-c") + .arg("core.attributesfile=/dev/null") + .arg("-C") + .arg(repo) + .args(["checkout", "FETCH_HEAD"]) + .env("GIT_CONFIG_NOSYSTEM", "1") + .env("GIT_CONFIG_GLOBAL", "/dev/null") + .env("GIT_TERMINAL_PROMPT", "0") + .env("GIT_ASKPASS", "true") + .env("LC_ALL", "C"); + if let Some(path) = path { + command.args(["--", path]); + } + let output = self.command_output_with_timeout( + command, + remaining_git_deadline(deadline)?, + "git checkout fetched tree", + )?; + self.success_stdout(output).map(|_| ()) } pub fn diff_no_index_tree( @@ -1610,8 +1879,19 @@ impl GitService { fn hardened_network_command(&self, repo: &Path, deadline: Instant) -> Result { self.reject_untrusted_repository_network_config(repo, deadline)?; - let trusted_network_config = self.trusted_global_network_config(deadline)?; + let mut trusted_network_config = self.trusted_global_network_config(deadline)?; + TEST_TRUSTED_URL_REWRITES.with(|rewrites| { + trusted_network_config.extend(rewrites.borrow().iter().map( + |(replacement, source_prefix)| { + ( + format!("url.{replacement}.insteadof"), + source_prefix.clone(), + ) + }, + )); + }); let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, false); command .arg("-c") .arg("core.hooksPath=/dev/null") @@ -1649,6 +1929,7 @@ impl GitService { deadline: Instant, ) -> Result, String> { let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, true); command .args(["config", "--global", "--null", "--list"]) .env("LC_ALL", "C"); @@ -1662,9 +1943,7 @@ impl GitService { } Ok(parse_null_git_config(&output.stdout) .into_iter() - .filter(|(key, _)| { - is_credential_helper_key(key) || key.eq_ignore_ascii_case("core.sshcommand") - }) + .filter(|(key, _)| is_trusted_global_network_key(key)) .collect()) } @@ -1701,6 +1980,7 @@ impl GitService { deadline: Instant, ) -> Result { let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, false); command .arg("-C") .arg(repo) @@ -1740,6 +2020,7 @@ impl GitService { deadline: Instant, ) -> Result, String> { let mut command = Command::new("git"); + isolate_git_config_environment(&mut command, false); command .arg("-C") .arg(repo) @@ -2011,11 +2292,52 @@ fn parse_null_git_config(output: &[u8]) -> Vec<(String, String)> { .collect() } +fn isolate_git_config_environment(command: &mut Command, preserve_global: bool) { + let trusted_global = preserve_global.then(|| std::env::var_os("GIT_CONFIG_GLOBAL")); + for key in [ + "GIT_CONFIG_COUNT", + "GIT_CONFIG_PARAMETERS", + "GIT_CONFIG_SYSTEM", + "GIT_CONFIG_GLOBAL", + "GIT_CONFIG_NOSYSTEM", + "GIT_CONFIG_PATH", + "GIT_CONFIG", + "GIT_TEMPLATE_DIR", + ] { + command.env_remove(key); + } + for index in 0..128 { + command.env_remove(format!("GIT_CONFIG_KEY_{index}")); + command.env_remove(format!("GIT_CONFIG_VALUE_{index}")); + } + command + .env("GIT_CONFIG_NOSYSTEM", "1") + .env("GIT_CONFIG_SYSTEM", "/dev/null") + .env("GIT_TEMPLATE_DIR", "/dev/null"); + if preserve_global { + if let Some(path) = trusted_global.flatten() { + command.env("GIT_CONFIG_GLOBAL", path); + } else { + command.env_remove("GIT_CONFIG_GLOBAL"); + } + } else { + command.env("GIT_CONFIG_GLOBAL", "/dev/null"); + } +} + fn is_credential_helper_key(key: &str) -> bool { let key = key.to_ascii_lowercase(); key == "credential.helper" || (key.starts_with("credential.") && key.ends_with(".helper")) } +fn is_trusted_global_network_key(key: &str) -> bool { + let key = key.to_ascii_lowercase(); + is_credential_helper_key(&key) + || key == "core.sshcommand" + || (key.starts_with("url.") + && (key.ends_with(".insteadof") || key.ends_with(".pushinsteadof"))) +} + fn local_network_config_is_untrusted(key: &str) -> bool { let key = key.to_ascii_lowercase(); is_credential_helper_key(&key) @@ -2390,6 +2712,90 @@ fn parse_merge_tree_analysis(output: &[u8]) -> Result Result<(), String> { + let mut entry_count = 0usize; + let mut total_bytes = 0u64; + + for record in output + .split(|byte| *byte == 0) + .filter(|record| !record.is_empty()) + { + entry_count = entry_count.saturating_add(1); + if entry_count > MAX_STRICT_TREE_ENTRIES { + return Err(format!( + "Fetched Git tree contains more than {MAX_STRICT_TREE_ENTRIES} entries." + )); + } + + let tab_index = record + .iter() + .position(|byte| *byte == b'\t') + .ok_or_else(|| "Fetched Git tree contains an invalid entry.".to_string())?; + let (metadata, path) = record.split_at(tab_index); + let path = &path[1..]; + let metadata = std::str::from_utf8(metadata) + .map_err(|_| "Fetched Git tree contains invalid entry metadata.".to_string())?; + let path = std::str::from_utf8(path) + .map_err(|_| "Fetched Git tree contains a non-UTF-8 path.".to_string())?; + let mut fields = metadata.split_ascii_whitespace(); + let mode = fields + .next() + .ok_or_else(|| "Fetched Git tree omitted an entry mode.".to_string())?; + let object_type = fields + .next() + .ok_or_else(|| "Fetched Git tree omitted an object type.".to_string())?; + let _object_id = fields + .next() + .ok_or_else(|| "Fetched Git tree omitted an object id.".to_string())?; + let size = fields + .next() + .ok_or_else(|| "Fetched Git tree omitted an entry size.".to_string())?; + + if path.is_empty() + || path.starts_with('/') + || path.contains('\\') + || path.chars().any(char::is_control) + || path.split('/').any(|part| { + part.is_empty() + || part == "." + || part == ".." + || part == ".git" + || part.contains(':') + }) + { + return Err("Fetched Git tree contains an unsafe path.".to_string()); + } + + match mode { + "100644" | "100755" => { + if object_type != "blob" { + return Err("Fetched Git tree contains an unsupported entry type.".to_string()); + } + let size = size + .parse::() + .map_err(|_| "Fetched Git tree returned an invalid entry size.".to_string())?; + if size > MAX_STRICT_TREE_FILE_BYTES { + return Err(format!( + "Fetched Git tree contains a file over the {MAX_STRICT_TREE_FILE_BYTES} byte safety limit." + )); + } + total_bytes = total_bytes.checked_add(size).ok_or_else(|| { + "Fetched Git tree exceeds the total byte safety limit.".to_string() + })?; + if total_bytes > MAX_STRICT_TREE_TOTAL_BYTES { + return Err(format!( + "Fetched Git tree exceeds the {MAX_STRICT_TREE_TOTAL_BYTES} byte safety limit." + )); + } + } + "120000" => return Err("Fetched Git tree contains a symlink entry.".to_string()), + "160000" => return Err("Fetched Git tree contains a Git submodule entry.".to_string()), + _ => return Err("Fetched Git tree contains an unsupported file mode.".to_string()), + } + } + Ok(()) +} + fn parse_tree_entries(output: &[u8]) -> Result, String> { let records = output .split(|byte| *byte == 0) @@ -2872,14 +3278,313 @@ mod tests { let temp = temp_dir("git-snapshot-tree-work"); let checkout = temp.join("checkout"); - let sha = fetch_ref_tree(remote.to_str().unwrap(), "main", &checkout).unwrap(); + let fetch = GitService::new() + .fetch_ref_tree_with_diagnostics(remote.to_str().unwrap(), "main", &checkout) + .unwrap(); - assert!(!sha.is_empty()); + assert!(!fetch.resolved_sha.is_empty()); + assert_eq!(fetch.fetch_count, 1); assert!(checkout.join("skills/demo/SKILL.md").exists()); assert!(checkout.join("README.md").exists()); assert!(!checkout.join(".git").exists()); } + #[test] + fn snapshot_fetch_ref_tree_ignores_global_filter_selected_by_remote_attributes() { + const CHILD: &str = "SKILLBOX_REMOTE_ATTRIBUTE_FILTER_CHILD"; + if let Some(root) = std::env::var_os(CHILD) { + let root = PathBuf::from(root); + let git = GitService::new(); + let remote = root.join("remote.git"); + Command::new("git") + .args(["init", "--bare"]) + .arg(&remote) + .output() + .unwrap(); + let work = root.join("work"); + git.init_main(&work).unwrap(); + fs::create_dir_all(work.join("skills/demo")).unwrap(); + fs::write( + work.join(".gitattributes"), + "skills/demo/SKILL.md filter=marker\n", + ) + .unwrap(); + fs::write(work.join("skills/demo/SKILL.md"), "name: demo\n").unwrap(); + git.add_all(&work).unwrap(); + git.commit(&work, "Remote attribute fixture").unwrap(); + git.set_origin_url(&work, remote.to_str().unwrap()).unwrap(); + git.push_origin_main(&work, true).unwrap(); + + let marker = root.join("filter-invoked"); + let filter = root.join("filter.sh"); + fs::write( + &filter, + format!("#!/bin/sh\nprintf invoked > '{}'\ncat\n", marker.display()), + ) + .unwrap(); + fs::set_permissions(&filter, fs::Permissions::from_mode(0o700)).unwrap(); + let global_config = root.join("global.gitconfig"); + fs::write( + &global_config, + format!("[filter \"marker\"]\n\tsmudge = !{}\n", filter.display()), + ) + .unwrap(); + std::env::set_var("GIT_CONFIG_GLOBAL", &global_config); + + let checkout = root.join("checkout"); + let sha = git + .fetch_ref_tree(remote.to_str().unwrap(), "main", &checkout) + .unwrap(); + assert!(!sha.is_empty()); + assert!(checkout.join("skills/demo/SKILL.md").exists()); + assert!( + !marker.exists(), + "remote .gitattributes activated a global filter during checkout" + ); + return; + } + + let root = temp_dir("git-remote-attribute-filter-child"); + let output = Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "tests::snapshot_fetch_ref_tree_ignores_global_filter_selected_by_remote_attributes", + "--nocapture", + ]) + .env(CHILD, &root) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + } + + #[test] + fn snapshot_fetch_ref_tree_ignores_process_git_config_filter_injection() { + const CHILD: &str = "SKILLBOX_PROCESS_FILTER_CHILD"; + if let Some(root) = std::env::var_os(CHILD) { + let root = PathBuf::from(root); + let git = GitService::new(); + let remote = root.join("remote.git"); + Command::new("git") + .args(["init", "--bare"]) + .arg(&remote) + .output() + .unwrap(); + let work = root.join("work"); + git.init_main(&work).unwrap(); + fs::create_dir_all(work.join("skills/demo")).unwrap(); + fs::write( + work.join(".gitattributes"), + "skills/demo/SKILL.md filter=marker\n", + ) + .unwrap(); + fs::write(work.join("skills/demo/SKILL.md"), "name: demo\n").unwrap(); + git.add_all(&work).unwrap(); + git.commit(&work, "Process config fixture").unwrap(); + git.set_origin_url(&work, remote.to_str().unwrap()).unwrap(); + git.push_origin_main(&work, true).unwrap(); + + let marker = root.join("process-filter-invoked"); + let filter = root.join("process-filter.sh"); + fs::write( + &filter, + format!("#!/bin/sh\nprintf invoked > '{}'\ncat\n", marker.display()), + ) + .unwrap(); + fs::set_permissions(&filter, fs::Permissions::from_mode(0o700)).unwrap(); + std::env::set_var("GIT_CONFIG_COUNT", "1"); + std::env::set_var("GIT_CONFIG_KEY_0", "filter.marker.smudge"); + std::env::set_var("GIT_CONFIG_VALUE_0", format!("!{}", filter.display())); + std::env::set_var( + "GIT_CONFIG_PARAMETERS", + "'filter.marker.smudge'='!injected'", + ); + + let checkout = root.join("checkout"); + let sha = git + .fetch_ref_tree(remote.to_str().unwrap(), "main", &checkout) + .unwrap(); + assert!(!sha.is_empty()); + assert!(checkout.join("skills/demo/SKILL.md").exists()); + assert!(!marker.exists(), "process-level Git filter executed"); + return; + } + + let root = temp_dir("git-process-filter-child"); + let output = Command::new(std::env::current_exe().unwrap()) + .args([ + "--exact", + "tests::snapshot_fetch_ref_tree_ignores_process_git_config_filter_injection", + "--nocapture", + ]) + .env(CHILD, &root) + .output() + .unwrap(); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + } + + #[cfg(unix)] + #[test] + fn snapshot_fetch_ref_tree_rejects_symlink_and_gitlink_entries_before_checkout() { + let git = GitService::new(); + + let symlink_remote = temp_dir("git-snapshot-symlink-remote").join("remote.git"); + Command::new("git") + .args(["init", "--bare"]) + .arg(&symlink_remote) + .output() + .unwrap(); + let symlink_work = temp_dir("git-snapshot-symlink-work"); + git.init_main(&symlink_work).unwrap(); + fs::create_dir_all(symlink_work.join("skills/demo")).unwrap(); + fs::write(symlink_work.join("skills/demo/SKILL.md"), "name: demo\n").unwrap(); + std::os::unix::fs::symlink("outside", symlink_work.join("link")).unwrap(); + git.add_all(&symlink_work).unwrap(); + git.commit(&symlink_work, "Symlink fixture").unwrap(); + git.set_origin_url(&symlink_work, symlink_remote.to_str().unwrap()) + .unwrap(); + git.push_origin_main(&symlink_work, true).unwrap(); + let symlink_error = git + .fetch_ref_tree( + symlink_remote.to_str().unwrap(), + "main", + temp_dir("git-snapshot-symlink-checkout").join("checkout"), + ) + .unwrap_err(); + assert!(symlink_error.contains("symlink entry"), "{symlink_error}"); + + let nested = temp_dir("git-snapshot-gitlink-nested"); + git.init_main(&nested).unwrap(); + fs::write(nested.join("nested.txt"), "nested\n").unwrap(); + git.add_all(&nested).unwrap(); + let nested_sha = git.commit(&nested, "Nested fixture").unwrap(); + let gitlink_remote = temp_dir("git-snapshot-gitlink-remote").join("remote.git"); + Command::new("git") + .args(["init", "--bare"]) + .arg(&gitlink_remote) + .output() + .unwrap(); + let gitlink_work = temp_dir("git-snapshot-gitlink-work"); + git.init_main(&gitlink_work).unwrap(); + fs::create_dir_all(gitlink_work.join("skills/demo")).unwrap(); + fs::write(gitlink_work.join("skills/demo/SKILL.md"), "name: demo\n").unwrap(); + run_git( + &gitlink_work, + &[ + "update-index", + "--add", + "--cacheinfo", + &format!("160000,{nested_sha},vendor/submodule"), + ], + ) + .unwrap(); + run_git(&gitlink_work, &["add", "--", "skills/demo/SKILL.md"]).unwrap(); + git.commit(&gitlink_work, "Gitlink fixture").unwrap(); + git.set_origin_url(&gitlink_work, gitlink_remote.to_str().unwrap()) + .unwrap(); + git.push_origin_main(&gitlink_work, true).unwrap(); + let gitlink_error = git + .fetch_ref_tree( + gitlink_remote.to_str().unwrap(), + "main", + temp_dir("git-snapshot-gitlink-checkout").join("checkout"), + ) + .unwrap_err(); + assert!( + gitlink_error.contains("Git submodule entry"), + "{gitlink_error}" + ); + } + + #[test] + fn strict_tree_listing_rejects_limits_and_unsafe_paths_before_checkout() { + let entry = + |path: &str, size: u64| format!("100644 blob {} {}\t{path}\0", "0".repeat(40), size); + + let oversized = entry("large.bin", MAX_STRICT_TREE_FILE_BYTES + 1); + let error = validate_strict_tree_listing(oversized.as_bytes()).unwrap_err(); + assert!(error.contains("file over"), "{error}"); + + let total = (0..9) + .map(|index| entry(&format!("file-{index}.bin"), MAX_STRICT_TREE_FILE_BYTES)) + .collect::(); + let error = validate_strict_tree_listing(total.as_bytes()).unwrap_err(); + assert!(error.contains("safety limit"), "{error}"); + + let entries = (0..=MAX_STRICT_TREE_ENTRIES) + .map(|index| entry(&format!("file-{index}.bin"), 1)) + .collect::(); + let error = validate_strict_tree_listing(entries.as_bytes()).unwrap_err(); + assert!(error.contains("more than"), "{error}"); + + for path in [".git/config", "bad:name", "/absolute", "a/../b", "a//b"] { + let error = validate_strict_tree_listing(entry(path, 1).as_bytes()).unwrap_err(); + assert!(error.contains("unsafe path"), "{path}: {error}"); + } + } + + #[test] + fn snapshot_fetch_ref_tree_rejects_oversized_and_colon_entries_before_checkout() { + let git = GitService::new(); + let publish = |label: &str, extra_path: &str, extra_size: Option| { + let remote = temp_dir(&format!("{label}-remote")).join("remote.git"); + Command::new("git") + .args(["init", "--bare"]) + .arg(&remote) + .output() + .unwrap(); + let work = temp_dir(&format!("{label}-work")); + git.init_main(&work).unwrap(); + fs::create_dir_all(work.join("skills/demo")).unwrap(); + fs::write(work.join("skills/demo/SKILL.md"), "name: demo\n").unwrap(); + let extra = work.join(extra_path); + if let Some(size) = extra_size { + if let Some(parent) = extra.parent() { + fs::create_dir_all(parent).unwrap(); + } + fs::File::create(extra).unwrap().set_len(size).unwrap(); + } else { + fs::write(extra, "unsafe\n").unwrap(); + } + git.add_all(&work).unwrap(); + git.commit(&work, label).unwrap(); + git.set_origin_url(&work, remote.to_str().unwrap()).unwrap(); + git.push_origin_main(&work, true).unwrap(); + remote + }; + + let oversized_remote = publish( + "git-snapshot-oversized", + "large.bin", + Some(MAX_STRICT_TREE_FILE_BYTES + 1), + ); + let oversized_checkout = temp_dir("git-snapshot-oversized-checkout").join("checkout"); + let error = git + .fetch_ref_tree( + oversized_remote.to_str().unwrap(), + "main", + &oversized_checkout, + ) + .unwrap_err(); + assert!(error.contains("file over"), "{error}"); + assert!(!oversized_checkout.join("skills/demo/SKILL.md").exists()); + + let colon_remote = publish("git-snapshot-colon", "bad:name.txt", None); + let colon_checkout = temp_dir("git-snapshot-colon-checkout").join("checkout"); + let error = git + .fetch_ref_tree(colon_remote.to_str().unwrap(), "main", &colon_checkout) + .unwrap_err(); + assert!(error.contains("unsafe path"), "{error}"); + assert!(!colon_checkout.join("skills/demo/SKILL.md").exists()); + } + #[test] fn snapshot_diff_no_index_tree_reports_changed_files() { let temp = temp_dir("git-diff-no-index"); @@ -3011,17 +3716,17 @@ mod tests { let fetch_ref_path_start = source.find("pub fn fetch_ref_path").unwrap(); let diff_no_index_start = source.find("pub fn diff_no_index_tree").unwrap(); let fetch_ref_path_source = &source[fetch_ref_path_start..diff_no_index_start]; - let run_network_start = source.find("fn run_network").unwrap(); - let run_with_config_start = source.find("fn run_with_config").unwrap(); - let run_network_source = &source[run_network_start..run_with_config_start]; + let run_fetch_start = source.find("fn run_fetch(").unwrap(); + let run_fetch_command_start = source.find("fn run_fetch_command").unwrap(); + let run_fetch_source = &source[run_fetch_start..run_fetch_command_start]; let hardened_network_start = source.find("fn hardened_network_command").unwrap(); let trusted_config_start = source.find("fn trusted_global_network_config").unwrap(); let hardened_network_source = &source[hardened_network_start..trusted_config_start]; - assert!(fetch_ref_path_source.contains("run_network")); + assert!(fetch_ref_path_source.contains("run_fetch")); assert!(fetch_ref_path_source.contains("FETCH_REF_TIMEOUT")); - assert!(run_network_source.contains("command_output_with_timeout")); - assert!(run_network_source.contains("hardened_network_command")); + assert!(run_fetch_source.contains("command_output_with_timeout")); + assert!(run_fetch_source.contains("hardened_network_command")); assert!(hardened_network_source.contains("GIT_TERMINAL_PROMPT")); assert!(hardened_network_source.contains("GIT_ASKPASS")); assert!(hardened_network_source.contains("GCM_INTERACTIVE")); diff --git a/crates/skillbox-github/src/lib.rs b/crates/skillbox-github/src/lib.rs index 1903da8..5a2452b 100644 --- a/crates/skillbox-github/src/lib.rs +++ b/crates/skillbox-github/src/lib.rs @@ -7,6 +7,7 @@ pub struct GitHubSkillSource { pub owner: String, pub repo: String, pub reference: String, + pub reference_explicit: bool, pub path: String, pub is_root: bool, pub url: String, @@ -117,6 +118,8 @@ pub fn parse_github_skill_url(input: &str) -> Result owner, repo, reference, + reference_explicit: matches!(kind.as_str(), "tree" | "blob" | "raw") + || (kind == "api" && url.query_pairs().any(|(key, _)| key == "ref")), path: skill_path, is_root, kind, @@ -332,6 +335,16 @@ mod tests { ); } + #[test] + fn does_not_infer_slash_ref_for_ambiguous_repository_root_url() { + let source = + parse_github_skill_url("https://github.com/acme/repo/tree/release/1.0").unwrap(); + + assert_eq!(source.reference, "release"); + assert_eq!(source.path, "1.0"); + assert!(!source.is_root); + } + #[test] fn normalizes_blob_raw_and_api_urls_to_skill_directory() { assert_eq!( @@ -406,6 +419,7 @@ mod tests { source.url, "https://github.com/acme/repo/tree/main", "{url}" ); + assert_eq!(source.reference_explicit, url.contains("/tree/")); } } diff --git a/docs/architecture.md b/docs/architecture.md index 501d4e4..858d3c9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -70,6 +70,8 @@ React UI - `bind_remote_source` -> `skillbox_core::bind_remote_source` - `preview_github_remote_skill_install` -> `skillbox_core::preview_github_remote_skill_install` - `install_github_remote_skill` -> `skillbox_core::install_github_remote_skill` +- `preview_github_skill_collection` -> `skillbox_core::preview_github_skill_collection` +- `apply_github_skill_collection` -> `skillbox_core::apply_github_skill_collection` - `list_remote_skill_versions` -> `skillbox_core::list_remote_skill_versions` - `preview_remote_version_change` -> `skillbox_core::preview_remote_version_change` - `apply_remote_version_change` -> `skillbox_core::apply_remote_version_change` @@ -208,10 +210,16 @@ Code、Cursor;它只决定 discovery/recommendation 顺序,不授权自动 ### Local Git Skill Collections -Phase A+B 的 Collection model 只覆盖本地 Git discovery、Import Review -grouping 和成功导入后的 provenance persistence。Rust `GitService` 返回 +Phase A+B 的 Collection model 覆盖本地 Git discovery、Import Review +grouping 和成功导入后的 provenance persistence。面向 v0.9.0 的 Phase C +实现扩展同一 +model 支持 GitHub repository/root 的一次 fetch collection preview/apply。Rust `GitService` 返回 canonical worktree root、Git common directory、branch/detached state、HEAD 和 sanitized origin;core 以 worktree/common-dir pair 作为 collection identity。 +GitHub remote collections 使用稳定的 canonical source URL + explicit requested ref +作为 collection identity;resolved SHA、完整 child tree、selection 和 status 只进入 +每次 preview identity。因此同一 repo/ref 的新 commit 会得到新的 preview,但不会伪造 +成另一个长期 collection;Phase D 仍未提供更新/回滚语义。 对于没有 live Git metadata 的复制安装,Import Review 可以读取配置 runtime root 旁边受支持的 v3 `.skill-lock.json`。Rust 只解析 bounded JSON,校验 @@ -233,9 +241,13 @@ target,然后才导入并保存 `skill_collections` / `skill_collection_member 前不创建 collection rows;失败时使用受保护的 compensatable rollback,并报告无法 回滚的部分,而不是声称跨文件系统 transaction。 -Phase C 的 GitHub multi-skill one-fetch install,以及 Phase D 的 collection-level -update/rollback 尚未实现。当前实现也不自动部署、不执行 hooks、filters、submodules、 -repository scripts、custom helpers 或 arbitrary shell。 +Phase C 的 GitHub multi-skill one-fetch install 只允许显式 child selection,并在 apply +前重新验证 canonical source URL、ref、resolved SHA、child snapshot 和 managed target; +裸 repository URL 不假设 `main`,必须通过结构化结果要求显式 ref;root-only skill 也 +拒绝与 nested `SKILL.md` roots 重叠。它已按 v0.9.0 实现但尚未完成 release +qualification,尚未发布。 +Phase D 的 collection-level update/rollback 尚未实现。当前实现也不自动部署、不执行 hooks、filters、submodules、repository +scripts、custom helpers 或 arbitrary shell。 不要在没有 adapter 语义的情况下猜测某个 agent 的目录布局。新增 agent 支持时,先定义 adapter 的发现路径、原生格式、部署方式和冲突处理。 @@ -324,8 +336,7 @@ GitHub remote source 可以是仓库中的 skill 子目录,也可以是根目 - Rust CLI 有 `remote-source-candidates`、`remote-source-preview`、`bind-remote-source`、`remote-versions`、`remote-preview-change`、`remote-apply-change`、`usage-record`、`usage-rankings`、`usage-audit`、各 provider history backfill、`usage-hook`、`usage-hook-status`、`usage-hook-install`、`doctor` 和 `operations`。 - Rust CLI 有 `workspaces`、`workspace-scan`、`workspace-add`、`workspace-forget` 来管理 workspace registry。 - Rust core、CLI 和 Tauri 已覆盖 `~/.skillbox/user-skills` 的 outbound Git - commit/push;reviewed inbound `origin/main` fast-forward 已实现并处于 v0.7 - Draft qualification,尚未作为 released capability 声明。 + commit/push;reviewed inbound `origin/main` fast-forward 已随 v0.7.0 发布。 - Rust core 已覆盖 remote skill 的 GitHub install preview/apply、GitHub update check、source binding、diff preview、update/rollback apply 和 operation log。 - Rust core 和 Tauri 已覆盖 usage stats 显式上报,以及 Codex App、Codex CLI、Claude Code CLI 的 Stop hook 注入入口。schema v7 把本机 evidence 分为 `confirmed`、`inferred` 和 `reference`;用户可见 `Calls` 只包含前两类,History references 单独展示。Rankings 支持 time range、User/Remote/System skill type、Agent 和 Workspace 的结构化过滤,并返回同一过滤快照内的 evidence totals、时间覆盖和可重叠 provenance source counts,以及 Codex、Claude Code、Cursor 最近一次 history scan 的文件/session 数。桌面 `Sync histories` 顺序调用三个 provider;单个 provider 失败不会撤销其他 provider 已成功写入或升级的幂等事件。 - Codex 本地 store 没有稳定的 provider-native skill-run total。Codex 结构化逐回合 skill carrier 只能作为 defensible `inferred` Calls;`usage-audit` 明确报告这个已知 undercount,不读取或返回聊天正文。 diff --git a/docs/data-model.md b/docs/data-model.md index 25cc7ec..a7e6c8e 100644 --- a/docs/data-model.md +++ b/docs/data-model.md @@ -329,6 +329,18 @@ selected-child import 成功后写入,写入使用 SQLite transaction,重复 现有 skill/deployment/usage/import rows,也不要求用户 rescan;collection source 不可用 时保留 provenance,并在读取 collection detail 时派生 `available=false`。 +schema v9 为 `skill_collections` 增加 `source_kind`、`source_url` 和 +`requested_reference`。已有本地 worktree rows 回填为 `git_worktree`,其 +canonical root/HEAD/availability 语义不变;GitHub Phase C rows 使用 +`github_remote`,保存无凭据的 canonical repository URL 和用户请求的 ref。 +GitHub collection id 稳定绑定 source kind、canonical source URL 和 explicit requested +ref;preview id 另外绑定 resolved SHA、完整 tree、child snapshot/status 和 selection。 +裸 repository URL 不自动声明 `main`,必须返回 explicit-ref-required 结果。远程 +collection apply 只在一次 bounded fetch 后写入选中的 child,并在 apply +前重新验证 source URL、ref、resolved SHA、child relative path、snapshot 和 +managed target;Phase C 按 v0.9.0 实现但尚未完成 release qualification,尚未发布;Phase D 的 +collection-level update/rollback 不属于此迁移。 + Import Review 返回的 `ImportCandidateCollection` 还有一个只读的 `source_kind`:`git_worktree` 表示可绑定 canonical worktree/HEAD 的本地 Git 来源;`installed_source` 表示由受支持的 v3 installer lockfile 归并的来源 diff --git a/docs/decisions/0007-git-backed-skill-collections.md b/docs/decisions/0007-git-backed-skill-collections.md index d8e9ddf..e348648 100644 --- a/docs/decisions/0007-git-backed-skill-collections.md +++ b/docs/decisions/0007-git-backed-skill-collections.md @@ -2,8 +2,10 @@ ## Status -Accepted for v0.8.0 Phase A+B. GitHub multi-skill fetch and collection-level -update/rollback remain planned Phase C/D work. +Accepted for the v0.8.0/v0.9.x collection milestone. Phase A+B shipped in +v0.8.0. The one-fetch GitHub multi-skill preview/apply boundary is implemented +for v0.9.0 and remains unreleased pending v0.9.0 release qualification. Collection-level +update/rollback remains planned Phase D work for a later v0.9.x release. ## Context @@ -63,5 +65,11 @@ preserved and the failure explains the recovery boundary. versions remain independent and do not require a rescan after migration. - Collection availability is derived from the canonical worktree at read time; moving or deleting the worktree does not delete managed skills or provenance. -- Phase C will add one-fetch GitHub repository preview/apply; Phase D will add - reviewed collection update/rollback. Neither is part of the current contract. +- Phase C adds one-fetch GitHub repository preview/apply through a separate + remote collection source kind and schema-v9 source metadata. It re-fetches + and revalidates the full reviewed tree before selected child writes; it does + not provide collection update/rollback. A repository URL must carry an explicit + ref; bare URLs return a structured explicit-ref-required result rather than + assuming `main`. Root-only skills are accepted only when no nested skill root + overlaps them, and fetched Git trees reject symlinks and gitlinks before + checkout materialization. Phase D remains outside the current contract. diff --git a/docs/implementation-status.md b/docs/implementation-status.md index b274fd2..124c805 100644 --- a/docs/implementation-status.md +++ b/docs/implementation-status.md @@ -39,7 +39,7 @@ - Implemented SQLite-backed workspace registry for global and project-local skills roots, including `.codex/skills`, `.agents/skills`, `.claude/skills`, scan-time auto registration, imported skill counts, preview-confirmed single-root project initialization, manual add/forget, Rust CLI compatibility commands, Tauri commands, and a searchable desktop Workspaces page with type filters and per-workspace skill review/import. - Implemented import records and import revert in Rust core, Rust CLI, Tauri commands, and Skill Detail UI, including backup restoration, conservative legacy reconciliation, multi-workspace blocking, and warning/danger confirmation states. - Implemented Rust-owned Import Review groups with stable content/status variant identities, full-snapshot-equivalent locations, location-level User/Remote advice with explicit mixed-suggestion confirmation, explicit selection for materially different same-name variants, group-level Calls/search/tab counts, and one-primary-only import enforcement that preserves existing revert guarantees. -- Implemented v0.8.0 Skill Collections Phase A+B: Rust discovers the nearest safe Git worktree for local Import Review, groups repository children with canonical worktree/HEAD identity, keeps external copies unlinked, and persists reviewed collection/member provenance after a stale-checked child import. GitHub multi-skill fetch and collection-level update/rollback remain planned. +- Implemented v0.8.0 Skill Collections Phase A+B: Rust discovers the nearest safe Git worktree for local Import Review, groups repository children with canonical worktree/HEAD identity, keeps external copies unlinked, and persists reviewed collection/member provenance after a stale-checked child import. GitHub multi-skill fetch and collection-level update/rollback are not part of the published v0.8.0 release. - Added a bounded installer-lockfile fallback for copied skills: valid v3 GitHub provenance can form a display-only `installed_source` collection after filesystem scanning, while live Git identity wins and selected children retain the ordinary per-skill import/apply contract. - Added signed macOS app update checks and user-confirmed install/restart through the Tauri updater plugin, plus release workflow assets for updater archives, signatures, and `latest.json`. - Added daily macOS updater metadata checks with a SQLite-backed successful-result cache, a sidebar Update reminder, one-click metadata recheck plus signed install/restart, and retry-safe pending updates without automatic downloads. @@ -55,9 +55,12 @@ ## Next Implementation Targets The `0.6` implementation and release qualification are complete. SkillBox -`v0.7.1` is the current shipped release, including reviewed inbound sync and -the History query/layout patch. The next milestones follow [the versioned -evolution path](roadmap.md#versioned-evolution-path): +`v0.8.0` is the current shipped release, including reviewed inbound sync, +History query/layout fixes, and Skill Collections Phase A+B. The current +`v0.9.0` work follows [issue #46](https://github.com/santosli/SkillBox/issues/46): +Phase C is implemented for `v0.9.0` and remains unreleased pending `v0.9.0` +release qualification; Phase D remains planned for a later `v0.9.x` release. +The next milestones follow [the versioned evolution path](roadmap.md#versioned-evolution-path): ### 0.7 — Safe Sync, Deployment, And Recovery @@ -82,17 +85,19 @@ evolution path](roadmap.md#versioned-evolution-path): classification, and Calls/reference semantics backward compatible. - Keep native non-`SKILL.md` formats behind the future adapter boundary. -### 0.8 — Git-backed Skill Collections +### 0.9.x — Git-backed Skill Collections - [Git-backed Skill Collections](https://github.com/santosli/SkillBox/issues/46) - are the active v0.8.0 milestone. Phase A+B are implemented: local repository + are the active v0.9.0 milestone. Phase A+B shipped in v0.8.0: local repository grouping and schema-backed collection/member relationships are available in Import Review. A collection records canonical worktree/repository identity, reviewed HEAD, and child-relative provenance; child skills remain independent for selection, import, deployment, Calls, and history. -- Phase C, one-fetch GitHub multi-skill preview/apply, and Phase D, - commit-consistent collection update/rollback, remain planned and are not - shipped. +- Phase C, one-fetch GitHub multi-skill preview/apply, is implemented for + v0.9.0 and remains unreleased pending v0.9.0 release qualification. It is + not shipped yet. + Phase D, commit-consistent collection update/rollback, remains planned for a + later v0.9.x release. - Collection scans must remain read-only. Apply must revalidate repository identity, HEAD/ref, and tree snapshot; execute no hooks, submodules, filters, scripts, or arbitrary shell; and preserve existing path, overwrite, diff --git a/docs/roadmap.md b/docs/roadmap.md index 836dff9..8896411 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,9 +4,9 @@ SkillBox is early-stage software. This roadmap describes the public direction, not a date-based commitment. Implementation details can change as the app gets more real-world use. -## Current Focus: 0.8.x +## Current Focus: 0.9.x -SkillBox `v0.7.1` is shipped. The 0.6 line made `SKILL.md` deployment targets +SkillBox `v0.8.0` is shipped. The 0.6 line made `SKILL.md` deployment targets explicit and portable without moving runtime knowledge into React: - a versioned Rust runtime-profile registry for Agents, Codex, Claude Code, @@ -22,18 +22,22 @@ and notarized `v0.7.1` distribution. The current release identity and distribution invariants remain documented in `docs/release.md`. Copy-snapshot deployment and broader recovery hardening remain planned product -work. Git-backed Skill Collections are the active `v0.8.0` milestone; Phase A+B -are now implemented, while remote multi-skill fetch and collection-level -update/rollback remain planned. +work. Git-backed Skill Collections are the active `v0.9.0` milestone; Phase A+B +shipped in `v0.8.0`, and the Phase C implementation is complete for `v0.9.0` +and remains unreleased pending `v0.9.0` release qualification. Collection-level update/rollback +remains planned Phase D work for a later `v0.9.x` release. ## Near-Term Priorities These are the next areas where focused contributions are most useful: -- **Git-backed Skill Collections (v0.8.0 Phase A+B).** Rust now treats one +- **Git-backed Skill Collections (v0.9.0).** Rust now treats one canonical Git repository/worktree and reviewed SHA as a local collection source while keeping child `SKILL.md` directories independently selectable - and deployable. Track the remaining scope in [GitHub issue #46](https://github.com/santosli/SkillBox/issues/46). + and deployable. Phase C's one-fetch GitHub preview/apply is implemented for + `v0.9.0` and remains unreleased pending `v0.9.0` release qualification; collection + update/rollback remains open. Track + the remaining scope in [GitHub issue #46](https://github.com/santosli/SkillBox/issues/46). - **Deployment portability.** Add copy-snapshot deployment as an explicit alternative to the current compatibility-checked symlink path. - **Recovery workflows.** Strengthen restore previews, backup inspection, and @@ -63,8 +67,8 @@ verified; completing a feature list alone does not qualify a release. | **0.5 — Local usage discovery and release awareness** | Evidence-aware local skill rankings with time-range, skill-type, agent, and workspace filters; separate Calls and history references; auditable multi-provider history sync; daily signed app-update awareness without automatic downloads. | CLI and desktop reconcile confirmed/inferred/reference evidence consistently; Calls never include low-signal references or claim provider-native totals; history providers resolve real local skills, deduplicate and upgrade stable identities, and preserve successful imports when another provider fails; schema upgrades and representative ranking queries are tested; update checks are rate-limited and every install is revalidated after an explicit click. | | **0.6 — Runtime profiles and portability** | Rust-owned runtime profiles model roots, precedence, frontmatter capabilities, and compatibility without hard-coding agent behavior in React. | Each supported profile has fixtures and compatibility tests; unsupported fields are reported before deployment; runtime-specific behavior remains behind an adapter boundary. | | **0.7 — Safe sync, deployment, and recovery** | Reviewed inbound user-skills Git updates are shipped; copy-snapshot deployment and broader restore/audit hardening remain planned follow-ups to the existing symlink path. | Worktree and branch-relation states are explicit; conflicts are never auto-merged; incoming trees, stale previews, deployed deletion/rename blockers, backup refs, index reconciliation, overwrite protection, and recovery paths have automated coverage. | -| **0.8 — Skill Collections and product hardening** | Git-backed Skill Collections add repository-level discovery, reviewed multi-skill sources, and commit-consistent update/rollback while large-library performance, diagnostics, accessibility, onboarding, and recovery mature for sustained daily use. | Collections use one canonical repository/worktree identity and reviewed SHA, perform one bounded fetch/check per remote repository, require explicit child selection, preserve per-skill deploy/Calls independence, and pass rollback/recovery and untrusted-tree tests with CLI/Tauri parity. Performance budgets and critical UI workflows are verified; supported upgrade and recovery procedures are documented and exercised. | -| **0.9 — Release candidate** | Feature scope is frozen while security, migration compatibility, packaging, updater, Homebrew, and real-world beta feedback are closed out. | Threat-model review is complete; upgrades from every supported prior release are tested; blocker defects are closed; signed and notarized distribution rehearsals pass. | +| **0.8 — Skill Collections foundation and product hardening** | Git-backed Skill Collections add repository-level discovery, persisted reviewed provenance, installed-source display grouping, and large-library import-review performance. | Collections use one canonical local repository/worktree identity, preserve per-skill selection/deploy/Calls independence, and pass local grouping, migration, recovery, and untrusted-input tests with CLI/Tauri parity. | +| **0.9 — Reviewed GitHub Skill Collection install** | Phase C adds one-fetch GitHub multi-skill preview/apply with explicit child selection and one reviewed commit SHA. Collection-level update/rollback remains later Phase D work. | Remote trees are bounded and fail closed before checkout; source/ref/tree/selection/managed state are stale-checked; selected children apply transactionally/compensatably; CLI/Tauri parity, recovery, and desktop review gates pass. | | **1.0 — Stable local skill management** | SkillBox offers a documented, supportable contract for discovering, importing, managing, deploying, updating, synchronizing, diagnosing, and recovering supported skills. | Core workflows meet their definitions of done; supported runtimes and limitations are explicit; migrations and recovery are proven; release artifacts and docs match; no open blocker or known data-loss issue remains. | ### 0.5 Usage Evidence And Ranking Boundaries @@ -128,10 +132,10 @@ targets, and unsupported deployment modes block deployment. SkillBox does not rewrite frontmatter, translate formats, select a target automatically, or write to a runtime before a fresh preview is explicitly confirmed. -### 0.8 Git-backed Skill Collections +### 0.9 Git-backed Skill Collections [Git-backed Skill Collections](https://github.com/santosli/SkillBox/issues/46) -are targeted for `v0.8.0`. Phase A+B are implemented: a collection is a +are targeted for `v0.9.0`. Phase A+B shipped in `v0.8.0`: a collection is a repository/source entity identified by its canonical Git repository or worktree plus ref/HEAD. A GitHub remote is optional. Child `SKILL.md` directories remain independent skills for selection, deployment, Calls, and @@ -156,20 +160,24 @@ permissions, and child imports continue through the ordinary per-skill safety contract. A remote repository URL will use one bounded fetch/check to preview all valid -children. Preview will report added, changed, removed, and invalid children, -plus path/name conflicts. Apply will install only explicitly selected child -snapshots while retaining collection provenance. Update and rollback remain -reviewed and commit-consistent at collection level; runtime deployment remains -an independent per-skill action. +children. Preview reports eligible/invalid children, duplicate-name and managed +conflict diagnostics, plus bounded path/name conflicts; it does not claim a +removed-child diff because collection updates are Phase D. Apply installs only +explicitly selected child snapshots while retaining collection provenance. +Runtime deployment, Calls, and History remain per-skill; collection-level +update/rollback remains Phase D. -Delivery is phased within the `v0.8.0` target: +Delivery is phased across the `v0.8.0` and `v0.9.x` releases: 1. **Implemented:** Repository detection and local Import Review grouping. 2. **Implemented:** Collection/source persistence and child relationships. -3. GitHub multi-skill install preview/apply with one repository fetch. +3. **Implemented for `v0.9.0` / awaiting qualification:** GitHub multi-skill + install preview/apply with one repository fetch. It is not part of the + published `v0.8.0` release and remains unreleased until merged and + release-qualified. 4. Collection-level update/rollback and UI detail. -Phases 3 and 4 are not implemented and must not be described as shipped. +Phase 4 is not implemented and must not be described as shipped. All scans are read-only. Apply must recheck canonical Git root, HEAD/ref, and tree snapshot and reject stale previews. Collection operations must never run diff --git a/docs/workflows.md b/docs/workflows.md index 28bd8bf..716f779 100644 --- a/docs/workflows.md +++ b/docs/workflows.md @@ -151,16 +151,58 @@ Review/apply: 时才形成 collection card;单个匹配仍作为普通 standalone candidate 显示。live Git worktree collection 保留其 repository/HEAD 语义,不受这个展示降噪门槛影响。 -当前边界:Phase C 的 GitHub repository one-fetch multi-skill preview/apply 与 Phase D -的 collection-level update/rollback 尚未实现。Collection operations 不运行 hooks、 -filters、submodules、repository scripts、custom helpers 或 arbitrary shell。 +### 2.2 GitHub Skill Collection Preview/Apply (Phase C) + +GitHub repository/root URL 的多 skill workflow 由 Rust core 承载: + +- CLI:`github-collection-preview [--managed-root ]`,然后使用 + `github-collection-apply --collection-id --preview-id + --select [--managed-root ]`。 +- Tauri:`preview_github_skill_collection` 和 + `apply_github_skill_collection`。Desktop 从 Import Review 展示 collection card, + 只提交结构化 child selections,不在 React 中解析 Git 或文件系统。 +- Preview 对 repository/root source 执行一次 bounded `fetch_ref_tree`,扫描完整 + checkout,返回 canonical sanitized URL、requested ref、resolved SHA、child groups、 + bounded errors 和 aggregate diagnostics;Git service 的结构化 fetch 结果证明实际 fetch + 次数,而不是由 UI 估算。子项可以是 importable、invalid、duplicate + name、already managed 或 managed conflict;只有显式选择的 eligible child 才会 apply。 +- Apply 重新 fetch 一次并重新校验 URL/ref/SHA、collection/preview identity、relative + path、full snapshot/content hash、duplicate name 和 managed target,然后才进行选中 + child 的普通 per-skill import。所有成功 child 绑定同一个 reviewed SHA;每个 child + 仍独立 deploy、Calls、History;collection-level update/rollback 属于 Phase D。 +- Repository-root URL(root `SKILL.md` 或多个 child)进入 collection review;指向 + 单一子目录的 URL 继续走既有 single-skill GitHub install。空仓库或没有有效 child + 返回 actionable error;无效 child 不会隐藏其它安全有效 child,除非触发全局安全上限。 +- 为了让 reviewed SHA 可重现,collection workflow 要求 repository URL 显式携带 + `/tree/`(或等价的显式 `ref`)。裸仓库 URL 不会假设 `main`,而是返回要求显式 + ref 的结构化结果;用户可以继续使用原有 single-skill URL 语义。对于 collection + root URL,`/tree/<...>` 无法安全推断带 slash 的 ref;请使用明确的 commit SHA + 或不含 slash 的 ref。只有已知 child skill-root 片段能够区分 ref 与路径时,才解析 + 带 slash 的 ref。 +- Root `SKILL.md` 只有在仓库没有其它 ancestor/descendant `SKILL.md` root 时才作为 + root-only skill 支持。父子重叠 roots 会在 preview 阶段 fail closed,避免一个 child + snapshot 包含另一个 child 或产生 overlapping apply。 +- 远程 tree 只读且不可信:禁止 `.git`、traversal、absolute/control/backslash/colon + paths、symlink、gitlink/submodule、unsafe file type、超限文件/条目/总字节,并禁用 + hooks、filters、scripts、custom helpers 和 arbitrary shell。root collection URL + 不会猜测带 slash 的 ref,也不会通过猜测路径扩大权限;请改用明确 SHA、无 slash + ref,或包含已知 child skill-root 的 unambiguous URL。 + +Phase C 已按 v0.9.0 实现,但仍未发布,待 v0.9.0 release qualification 完成后才可 +描述为 shipped。Phase D 的 collection-level update/rollback +仍未实现。Collection operations 不运行 hooks、filters、submodules、repository +scripts、custom helpers 或 arbitrary shell。 完成验证: -- `cargo test -p skillbox-core --offline schema_v8_collection_migration_is_idempotent_for_existing_database` +- `cargo test -p skillbox-core --offline schema_v9_github_collection_migration_is_idempotent_for_existing_database` - `cargo test -p skillbox-core --offline scan_import_candidates_groups_git_repository_children_and_keeps_external_copies_unlinked` - `cargo test -p skillbox-core --offline git_collection_apply_persists_selected_children_and_rejects_stale_head_before_writes` - `cargo run -p skillbox-cli --offline -- collection-preview --managed-root ` +- `cargo test -p skillbox-core --offline github_collections::tests` +- `cargo test -p skillbox-core --offline github_collection_identity_is_stable_across_commits_on_one_ref` +- `cargo test -p skillbox-core --offline collection_batch_failure_rolls_back_remote_current_index_and_audits_one_operation` +- `cargo run -p skillbox-cli --offline -- github-collection-preview --managed-root ` - `npm test` 和桌面 Import Review 的 collection collapsed/expanded/narrow visual QA。 ## 3. Revert Local Import @@ -578,7 +620,7 @@ filters、submodules、repository scripts、custom helpers 或 arbitrary shell ### Reviewed Inbound Fast-Forward -v0.7 Draft contract: +v0.7.0 reviewed inbound fast-forward contract: - Rust CLI: - `skillbox user-skills-inbound-check` @@ -1114,7 +1156,8 @@ returns structured JSON while the desktop provides interactive review. | Remote source binding, update, rollback, and versions | Full | Full | Desktop provides all-file visual review; CLI returns structured diff/version data. | | User-skills Git status and outbound commit/push | Full | Full | Desktop adds selected-file diff review. Existing push defaults and `push_failed` semantics remain unchanged. | | Reviewed inbound user-skills fast-forward (v0.7 shipped) | Full | Full | Both use Check -> Preview -> Apply with the same Rust validation and stale-preview contract. Desktop adds visual repository/skill/deployment review and conflict diagnostics. Neither interface auto-merges, rebases, resets, stashes, or resolves divergence. | -| Local Git Skill Collections Phase A+B | Full | Full | `collection-preview`/`collection-apply` and `collections` expose Rust-owned local worktree discovery, selected-child apply, and persisted provenance. GitHub multi-skill fetch and collection-level update/rollback remain planned. | +| Local Git Skill Collections Phase A+B | Full | Full | `collection-preview`/`collection-apply` and `collections` expose Rust-owned local worktree discovery, selected-child apply, and persisted provenance; shipped in v0.8.0. | +| GitHub Skill Collection Phase C | Full | Full | The unreleased `v0.9.0` implementation of `github-collection-preview`/`github-collection-apply` performs one fetch per preview/apply, explicit child selection, stale SHA/tree checks, and per-skill import. Collection-level update/rollback remains unsupported. | | Workspaces | Partial | Full | CLI lists/scans/adds/forgets exact roots. Desktop also previews a project directory, initializes one selected supported root, and offers the native folder picker. | | Usage rankings and local history sync | Full | Full | Both use the same confirmed/inferred/reference evidence model and provider backfills. | | Aggregate usage diagnostics | Full | Limited | CLI `usage-audit` is the automation-oriented aggregate report. Desktop exposes the relevant coverage summary and disclosure, not the complete diagnostic JSON. |