-
Notifications
You must be signed in to change notification settings - Fork 12
docs(cloud-agents): document customizing workspace snapshots #105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
46599f8
docs(cloud-agents): document customizing workspace snapshots
hongyi-chen 530e05f
docs(cloud-agents): address self-audit findings on snapshots page
hongyi-chen 1165e2d
Merge branch 'main' into oz/docs/snapshot-declarations
hongyi-chen 5c23a3b
docs(cloud-agents): address Oz review on snapshots page
hongyi-chen ea9bbd5
docs(cloud-agents): align 'untracked files' precision with caution
hongyi-chen a00d291
docs(cloud-agents): scrub private warp-agent-docker links from snapsh…
hongyi-chen a1e84de
docs(cloud-agents): add full bundled snapshot script alongside minima…
hongyi-chen 34990aa
Merge branch 'main' into oz/docs/snapshot-declarations
hongyi-chen bea343e
docs(cloud-agents): address Harry's review on snapshots page
hongyi-chen File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
233 changes: 233 additions & 0 deletions
233
src/content/docs/agent-platform/cloud-agents/handoff/snapshots.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,233 @@ | ||
| --- | ||
| title: Customizing workspace snapshots | ||
| description: >- | ||
| Customize which repositories and files Warp snapshots at the end of a cloud | ||
| agent run, so handoff continues to work outside the bundled cloud agent | ||
| image. | ||
| sidebar: | ||
| label: "Snapshots" | ||
| --- | ||
|
|
||
| Workspace snapshots are how [handoff](/agent-platform/cloud-agents/handoff/) carries repository changes and other workspace state across cloud agent runs. At the end of every cloud agent run, Warp asks a small declarations script which repositories and files to snapshot, then uploads the resulting git diffs and file contents so the next cloud agent run can apply them. | ||
|
|
||
| Warp's bundled cloud agent image ships with a declarations script that snapshots every git repository under the agent's workspace, so most cloud agent runs need no configuration. This page is for the cases where you need to customize what gets snapshotted — for example, when running cloud agents in a custom Docker image, on a self-hosted [Direct backend](/agent-platform/cloud-agents/self-hosting/managed-direct/), or as an [unmanaged](/agent-platform/cloud-agents/self-hosting/unmanaged/) `oz agent run` in CI. | ||
|
|
||
| ## When to customize snapshots | ||
|
|
||
| The default snapshotting behavior covers the most common case: snapshot every git repository the agent worked on, inside the bundled cloud agent image. Customize snapshots when you need different behavior: | ||
|
|
||
| * **You're running outside the bundled image** — for example, a custom Docker base image, a self-hosted Direct backend, or unmanaged `oz agent run` invocations — where Warp's default `snapshot-declarations.sh` isn't on disk. | ||
| * **You want to snapshot repos or files outside the agent's workspace** — for example, a sibling repo the agent reads but doesn't `cd` into, or a log file the agent writes to `/tmp`. | ||
| * **You want dynamic snapshotting behavior** — because the declarations file is generated by a script that Warp runs, you can script any logic you want (filter by git status, dedupe against a baseline, emit a fixed list, etc.) rather than being limited to a static set of paths. | ||
|
|
||
| If you don't customize snapshots in any of these cases, the run still completes — Warp just uploads nothing, and handoff into the next run starts without the prior session's uncommitted changes. | ||
|
|
||
| ## How snapshotting works | ||
|
|
||
| There are two ways to tell Warp what to snapshot at the end of a cloud agent run: | ||
|
|
||
| * **Script-driven** (default in the bundled image) - Warp invokes a declarations script (configured via the `OZ_SNAPSHOT_DECLARATIONS_SCRIPT` environment variable) that emits one JSON line per repository or file to snapshot. Use this when the set of repos isn't fixed ahead of time (for example, the agent may `git init` a new directory during its work) or when you want any other dynamic logic to decide what to snapshot. | ||
| * **Static** - You pre-populate the declarations file yourself (at the path given by `OZ_SNAPSHOT_DECLARATIONS_FILE`) and skip the script. Use this when the same set of repos and files should be snapshotted on every run. | ||
|
|
||
| ### Script-driven flow | ||
|
|
||
| 1. **Warp invokes your declarations script.** The script path comes from the `OZ_SNAPSHOT_DECLARATIONS_SCRIPT` environment variable. Warp runs it with the agent's workspace as the current directory. | ||
| 2. **Warp passes a per-run output file.** Warp sets `OZ_SNAPSHOT_DECLARATIONS_FILE` to an absolute path the script must write to. Each run gets its own file so concurrent runs don't clobber each other. | ||
| 3. **Warp reads the declarations file.** Warp parses the JSONL output (described below) and uploads the listed repositories and files. | ||
|
|
||
| See [Write a custom declarations script](#write-a-custom-declarations-script) for the full pattern. | ||
|
|
||
| ### Static flow | ||
|
|
||
| Set `OZ_SNAPSHOT_DECLARATIONS_FILE` to a path you've already populated and leave `OZ_SNAPSHOT_DECLARATIONS_SCRIPT` unset. Warp reads the file directly with no script invocation. See [Use a static declarations file](#use-a-static-declarations-file) for the full pattern. | ||
|
|
||
| In both flows, snapshotting is automatically enabled for cloud agent runs when cloud conversations are enabled, and can be turned off per run with `--no-snapshot`. See [Disable snapshots](#disable-snapshots) below. | ||
|
|
||
| ## Environment variables | ||
|
|
||
| * **`OZ_SNAPSHOT_DECLARATIONS_SCRIPT`** - Absolute path to the script Warp invokes at the end of each cloud agent run. The bundled cloud agent image sets this automatically. Set it yourself when running outside the bundled image. | ||
| * **`OZ_SNAPSHOT_DECLARATIONS_FILE`** - Absolute path to the JSONL file the script writes to (and Warp reads from). When `OZ_SNAPSHOT_DECLARATIONS_SCRIPT` is set, Warp picks a per-run path by default and exports it to the script. Set or override this variable yourself when you want Warp to read from a static, pre-populated declarations file instead of running a script, or whenever you want both Warp and the script to use a specific path you control. | ||
|
|
||
| ## Declarations file format | ||
|
|
||
| The declarations file is UTF-8 JSONL — one JSON object per non-empty line: | ||
|
|
||
| ```json title="snapshot-declarations.jsonl" | ||
| {"version":1,"kind":"repo","path":"/workspace/my-repo"} | ||
| {"version":1,"kind":"file","path":"/tmp/agent-output.log"} | ||
| ``` | ||
|
|
||
| Each line has exactly these fields: | ||
|
|
||
| * **`version`** - Always `1`. Reserved for future schema versions. | ||
| * **`kind`** - Either `"repo"` or `"file"`. | ||
| * **`path`** - A non-empty absolute path. | ||
|
|
||
| A few rules to follow when writing declarations: | ||
|
|
||
| * **Paths must be absolute.** Relative paths are rejected and logged as malformed. | ||
| * **Prefer `repo` over `file` for paths inside a repository.** Warp generates a git diff for each `repo` entry (tracked changes plus untracked, non-gitignored files), so individual `file` entries inside that repo are redundant and dropped before upload. Including a `repo` entry is also more extensible than enumerating files. | ||
| * **Repos are diffed, not copied wholesale.** Only changed files are uploaded for each `repo` entry, so listing a large repository is cheap when the agent's changes are small. | ||
| * **`file` entries are for paths outside any declared repo** — for example, logs the agent wrote to `/tmp` or scratch files in `$HOME`. | ||
|
|
||
| Malformed lines (invalid JSON, missing fields, unknown `kind`, non-absolute path) are logged as warnings and skipped; they never abort the upload. | ||
|
|
||
| :::caution | ||
| **Declared paths are uploaded to Warp.** `file` entries upload the file's contents verbatim, and `repo` entries upload the repo's git diff (tracked changes plus untracked, non-gitignored files). Before declaring logs, scratch files, or other agent outputs as `file` entries, make sure they don't contain secrets, credentials, API tokens, or other sensitive data. | ||
| ::: | ||
|
|
||
| ## Write a custom declarations script | ||
|
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do you think this is sufficient for a minimal pattern
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. spoke offline about this, but I think it's cool to just include the whole script |
||
|
|
||
| A custom script writes one JSON line per repository or file it wants Warp to snapshot, then exits. The minimal pattern looks like this: | ||
|
|
||
| ```bash title="snapshot-declarations.sh" | ||
| #!/bin/bash | ||
| set -euo pipefail | ||
|
|
||
| # Warp sets this to a per-run output path. Fail loudly if it's missing, | ||
| # so a misconfigured runner doesn't silently overwrite a shared file. | ||
| if [ -z "${OZ_SNAPSHOT_DECLARATIONS_FILE:-}" ]; then | ||
| echo "OZ_SNAPSHOT_DECLARATIONS_FILE must be set" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| mkdir -p "$(dirname "$OZ_SNAPSHOT_DECLARATIONS_FILE")" | ||
|
|
||
| # Find every git repo under the agent's workspace and emit a JSONL | ||
| # repo declaration for each one. Escape backslashes and double quotes in | ||
| # the path before writing so paths containing those characters don't | ||
| # produce malformed JSON (which Warp skips with a warning). | ||
| find "$PWD" -type d -name .git -prune -print 2>/dev/null \ | ||
| | while IFS= read -r git_dir; do | ||
| repo_root="$(cd "$(dirname "$git_dir")" && pwd)" | ||
| repo_json="$(printf '%s' "$repo_root" | sed 's/\\/\\\\/g; s/"/\\"/g')" | ||
| printf '{"version":1,"kind":"repo","path":"%s"}\n' "$repo_json" \ | ||
| >> "$OZ_SNAPSHOT_DECLARATIONS_FILE" | ||
| done | ||
| ``` | ||
|
|
||
| Then point Warp at it by exporting `OZ_SNAPSHOT_DECLARATIONS_SCRIPT` in the environment your cloud agent runs in: | ||
|
|
||
| ```bash | ||
| export OZ_SNAPSHOT_DECLARATIONS_SCRIPT=/path/to/snapshot-declarations.sh | ||
| ``` | ||
|
|
||
| For a managed [Direct backend](/agent-platform/cloud-agents/self-hosting/managed-direct/) worker, set it via the worker's `environment` config so it's present when the agent process starts. | ||
|
|
||
| ### The full bundled script | ||
|
|
||
| For reference, this is the canonical declarations script Warp invokes inside the bundled cloud agent image. It's a richer version of the minimal example above: it honors a colon-separated `OZ_SNAPSHOT_SCAN_ROOTS` override for operators who need to scan repos outside the default workspace, uses `jq` for canonical JSON encoding (which handles `"` and `\` escaping for you), and dedupes against repo declarations already written in the same run so repeated invocations stay additive. | ||
|
|
||
| :::note | ||
| The bundled script defaults `OZ_SNAPSHOT_JQ` to `/agent/tools/jq`, which only exists inside Warp's cloud agent image. If you adapt this script for a different image or for an unmanaged `oz agent run`, put `jq` on `PATH` or set `OZ_SNAPSHOT_JQ` to its absolute path. | ||
| ::: | ||
|
|
||
| ```bash title="snapshot-declarations.sh" | ||
| #!/bin/bash | ||
| # Generates the declarations file consumed by the end-of-run snapshot upload pipeline. | ||
| # | ||
| # Scans one or more roots for `.git` directories and appends one JSON object per | ||
| # newly-discovered repository to the declarations file. | ||
| # | ||
| # Scan root selection (in order of precedence): | ||
| # - OZ_SNAPSHOT_SCAN_ROOTS: colon-separated list of absolute paths. Operators can use this | ||
| # to target repos outside the default workspace. | ||
| # - $PWD: the Rust driver sets this to the agent's workspace via `Command::current_dir` | ||
| # before invoking the script, so the default scan root is always the assigned workspace. | ||
| # | ||
| # Output file: | ||
| # - OZ_SNAPSHOT_DECLARATIONS_FILE must be set. The Rust driver sets this to a per-run path | ||
| # so concurrent runs don't clobber each other. | ||
| # - The file is created if missing, and appended to if it already exists. Existing lines are | ||
| # never rewritten, so operator-authored declarations are preserved. | ||
| # - Repeated invocations within a single run skip already-emitted repo declarations so the | ||
| # upload pipeline can stay additive without duplicating identical generated entries. | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| SCAN_ROOTS_RAW="${OZ_SNAPSHOT_SCAN_ROOTS:-$PWD}" | ||
| IFS=':' read -r -a SCAN_ROOTS <<< "$SCAN_ROOTS_RAW" | ||
|
|
||
| if [ -z "${OZ_SNAPSHOT_DECLARATIONS_FILE:-}" ]; then | ||
| echo "OZ_SNAPSHOT_DECLARATIONS_FILE must be set" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| DECL_FILE="$OZ_SNAPSHOT_DECLARATIONS_FILE" | ||
| JQ="${OZ_SNAPSHOT_JQ:-/agent/tools/jq}" | ||
| mkdir -p "$(dirname "$DECL_FILE")" | ||
| touch "$DECL_FILE" | ||
|
|
||
| # Dedup set. Seed from repo declarations already emitted by this script so a repeated | ||
| # invocation within the same run doesn't re-emit repos it already discovered earlier. | ||
| # Keep the full canonical JSON line as the key; this lets us preserve unrelated JSONL entries | ||
| # verbatim while ensuring identical generated declarations are emitted at most once. | ||
| SEEN_FILE="$(mktemp)" | ||
| trap 'rm -f "$SEEN_FILE"' EXIT | ||
| "$JQ" --raw-input --compact-output ' | ||
| fromjson? | ||
| | select( | ||
| type == "object" | ||
| and (. | keys_unsorted | sort == ["kind", "path", "version"]) | ||
| and .version == 1 | ||
| and .kind == "repo" | ||
| and (.path | type == "string") | ||
| ) | ||
| | { version: 1, kind: "repo", path: .path } | ||
| ' "$DECL_FILE" > "$SEEN_FILE" | ||
|
|
||
| repo_declaration_for_path() { | ||
| "$JQ" --compact-output --null-input \ | ||
| --arg path "$1" \ | ||
| '{ version: 1, kind: "repo", path: $path }' | ||
| } | ||
|
|
||
| for root in "${SCAN_ROOTS[@]}"; do | ||
| [ -d "$root" ] || continue | ||
| while IFS= read -r git_dir; do | ||
| repo_root="$(cd "$(dirname "$git_dir")" && pwd)" | ||
| repo_declaration="$(repo_declaration_for_path "$repo_root")" | ||
| if grep -Fxq -- "$repo_declaration" "$SEEN_FILE"; then | ||
| continue | ||
| fi | ||
| printf '%s\n' "$repo_declaration" >> "$SEEN_FILE" | ||
| printf '%s\n' "$repo_declaration" >> "$DECL_FILE" | ||
| done < <(find "$root" -type d -name .git -prune -print 2>/dev/null) | ||
| done | ||
| ``` | ||
|
|
||
| ## Use a static declarations file | ||
|
|
||
| If the same set of repositories or files should be snapshotted on every run (for example, an unmanaged GitHub Actions job operating on a known checkout), you can skip the script entirely and pre-populate a JSONL file: | ||
|
|
||
| ```json title="/etc/oz/snapshot-declarations.jsonl" | ||
| {"version":1,"kind":"repo","path":"/workspace/my-repo"} | ||
| {"version":1,"kind":"repo","path":"/workspace/shared-libs"} | ||
| ``` | ||
|
|
||
| Then point Warp at the file directly: | ||
|
|
||
| ```bash | ||
| export OZ_SNAPSHOT_DECLARATIONS_FILE=/etc/oz/snapshot-declarations.jsonl | ||
| # Do not set OZ_SNAPSHOT_DECLARATIONS_SCRIPT in this mode. | ||
| ``` | ||
|
|
||
| Warp reads the file at end-of-run and uploads the listed repos and files. Because the file is the same on every run, this is best for environments where the workspace layout is fixed. | ||
|
|
||
| ## Disable snapshots | ||
|
|
||
| To opt out of snapshotting for a single run, pass `--no-snapshot` to the CLI: | ||
|
|
||
| ```bash | ||
| oz agent run-cloud --prompt "Fix the failing test" --no-snapshot | ||
| ``` | ||
|
|
||
| Snapshotting is also skipped automatically when cloud conversations are disabled for the team. | ||
|
|
||
| ## Related pages | ||
|
|
||
| * [Handoff from local to cloud](/agent-platform/cloud-agents/handoff/local-to-cloud/) - Promote a local conversation to a cloud run; the workspace snapshot is what carries your uncommitted changes across. | ||
| * [Handoff from cloud to cloud](/agent-platform/cloud-agents/handoff/cloud-to-cloud/) - Continue a finished cloud run; the prior session's workspace snapshot is what gets restored. | ||
| * [Self-hosting overview](/agent-platform/cloud-agents/self-hosting/) - Architecture decision guide for self-hosted workers, where customizing snapshots is most often needed. | ||
| * [Unmanaged architecture](/agent-platform/cloud-agents/self-hosting/unmanaged/) - Run `oz agent run` in CI, Kubernetes, or your dev environment outside the bundled image. | ||
| * [Oz CLI](/reference/cli/) - Full reference for `oz agent run` and `oz agent run-cloud`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💡 [SUGGESTION] [SECURITY] This section encourages declaring arbitrary files such as logs or scratch files, but it never warns that declared paths are uploaded to Warp. Add a caution to exclude secrets, credential stores, and sensitive logs before users copy this pattern.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this seems like a good call maybe