Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/logs/aeo_new_guide_recommendation_runs.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

New entries are prepended by each scheduled agent run. Most recent entry first.

This log tracks every bi-weekly run of the `aeo_new_guide_recommendations` skill — both runs that produced briefs and runs that wrote a no-brief or stale-snapshot report — so the team can detect recurring topic gaps and answer questions like "has this topic been flagged before?" without replaying individual Oz runs.
This log tracks every monthly run of the `aeo_new_guide_recommendations` skill — both runs that produced briefs and runs that wrote a no-brief report — so the team can detect recurring topic gaps and answer questions like "has this topic been flagged before?" without replaying individual Oz runs.

**Format**: see the `## Run log format` section in `.agents/skills/aeo_new_guide_recommendations/SKILL.md`.

Expand Down
56 changes: 33 additions & 23 deletions .agents/references/skill-authoring-guidelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,33 +45,37 @@ Agents often proceed past a failed file write without noticing. For any log upda
- After appending: `tail -5 <file>` and confirm the new entry appears.
- After push: `git log --oneline -1 origin/<branch>` and confirm the commit SHA matches the expected commit.

### Source data and freshness

**Cloud agents cannot call Peec MCP directly.** Peec requires OAuth authentication, which is not available in cloud agent environments. Any skill that needs Peec data must read from a pre-exported snapshot committed to the `buzz` repo.

If your skill uses external data (Peec, GSC, or any authenticated API) that is unavailable in cloud agents:

1. Build the data export into a separate **local-only skill** (e.g., `refresh-peec-aeo-snapshot`).
2. The cloud skill reads the committed snapshot, not the live API.
3. Add an explicit **freshness gate**: define a maximum age (e.g., 14 days), check `generated_at`, and exit with a stale-snapshot report if the threshold is exceeded. Never proceed with stale data.
4. Document the freshness constraint clearly at the top of the `## Source data` section: explain why a snapshot is used instead of a live call, so future editors don't remove the constraint thinking it is overly cautious.
### Source data from authenticated APIs

**Prefer live API calls with a token over pre-exported snapshots.** Cloud agents can authenticate with services that issue long-lived tokens. Peec, for example, is reached through its MCP server using a Personal Access Token stored as the `PEEC_PAT` Oz secret — cloud agents do not need OAuth, and they do not need a committed snapshot.

When your skill needs data from an authenticated API:

1. Store the credential as an Oz secret and reference it by name in the skill's `## Environment requirements` section. Never inline it.
2. For MCP-based sources, document the `mcp_servers` config the scheduled agent needs, so whoever creates the schedule knows the skill will not work without it:
```json
{
"peec-ai": {
"url": "https://api.peec.ai/mcp",
"headers": {
"Authorization": "Bearer ${PEEC_PAT}"
}
}
}
```
3. Define an explicit **unavailable path**: what the skill does when the token is missing or expired, the MCP server is not configured, or the call fails. Log the specific failure (never the token value), degrade to the remaining signals, and raise the confidence bar for any output produced without the primary signal.
4. Record availability in the run log (for example, `Source signals: Peec [available | unavailable]`) so the outer loop can distinguish a low-signal period from a broken credential.

Freshness gate pattern:
- Read `generated_at` from the snapshot metadata file.
- If the file is missing, `generated_at` is absent, or the age exceeds the threshold:
- Write a stale-snapshot report with the exact age (or error reason).
- Write a run log entry with a `No-run reason` of `snapshot stale — N days old`.
- Post a Slack alert.
- Exit. Do not proceed or open a PR.
**Use a committed snapshot only as a last resort** — when a source genuinely cannot be authenticated from a cloud agent. Snapshots introduce a freshness gate, a manual local refresh step, and a failure mode where the agent exits without doing work because nobody refreshed the data. If you do use one, define a maximum age, check it before use, and pair the skill with a scheduled refresh so the gate cannot silently starve the pipeline.

### Scope consistency

When you add a new topic area to a skill's scope, audit every section — especially `## Source data` — to confirm the source data actually covers the new topic. A common mistake: a skill lists four topic areas but the source data description names only three. The agent then produces lower-quality briefs for the fourth topic with no signal, or invents signals.

Checklist when expanding scope:
- Does the snapshot include data for the new topic? If not, update the snapshot refresh skill, or document the lower confidence explicitly.
- Does the source data include the new topic? If not, extend the tracked queries or prompts at the source, or document the lower confidence explicitly.
- Are all quality gates still valid for the new topic? (e.g., minimum brief count thresholds)
- Does the stale-snapshot report reflect the full scope?
- Do the no-action and unavailable-signal reports reflect the full scope?

### Scope contradictions in "Do not" lists

Expand All @@ -96,18 +100,24 @@ Never hard-code the Oz host in Slack messages or run output. The agent may run o

Always resolve the Oz run link at runtime:
```bash
oz-dev run get "<your run ID>" --output-format json | jq -r '.session_link'
oz run get "<your run ID>" --output-format json | jq -r '.session_link'
```

Use `oz`, not `oz-dev`. `oz-dev` is a local development build that ships with the Warp dev app; cloud sandboxes only have `oz`, so any skill instructing an agent to call `oz-dev` silently loses its run link.

If the command fails or returns an empty value, omit the `Oz run` line rather than posting a broken link.

### Secrets and environment variables

Always use `SLACK_BOT_TOKEN` and other secrets from environment variables — never inline them or print them to run output, logs, or Slack messages. If a required secret is unavailable, write the payload to the run output instead of posting to Slack. Do not crash the run on missing notification credentials. Include this in the skill as an explicit fallback, not just as an assumed environment guarantee.
Always read Slack tokens and other secrets from environment variables — never inline them or print them to run output, logs, or Slack messages. If a required secret is unavailable, write the payload to the run output instead of posting to Slack. Do not crash the run on missing notification credentials. Include this in the skill as an explicit fallback, not just as an assumed environment guarantee.

**Pick the token that matches the destination channel.** Several Slack bot tokens exist in the Oz secret store, and they authenticate as different bots with different channel memberships. A token that authenticates successfully still cannot post to a channel its bot has not joined. Name the expected bot in the skill's environment requirements (for example, `BUZZ_SLACK_TOKEN` posts as `buzz`, which is the account in `#growth-docs`) so a future editor does not swap in a token that authenticates but cannot deliver.

**A secret being present does not mean it works.** A token can authenticate while the paired channel ID is stale, or the bot may not be a member of the target channel — Slack returns `channel_not_found` in both cases. Skills that post to Slack should define what to do on a failed post (attempt a lookup by channel name, then fall back to run output) and must report the failure explicitly rather than logging the run as fully successful.

### Slack notifications

Post a Slack notification on every run, including no-action runs and stale-snapshot exits. A missing notification on a no-action run is indistinguishable from a run that silently failed. Use a simple text message (not Block Kit) that can be scanned in under 30 seconds.
Post a Slack notification on every run, including no-action runs and runs that exited early because a source signal was unavailable. A missing notification on a no-action run is indistinguishable from a run that silently failed. Use a simple text message (not Block Kit) that can be scanned in under 30 seconds.

---

Expand All @@ -117,7 +127,7 @@ Outer loop skills run less frequently (typically monthly) and read the inner loo

### Data minimum before the outer loop can run

The outer loop needs enough run log entries to identify real patterns, not noise. Require a minimum entry count before acting (the `improve-aeo-crosslink-skill` uses 8 entries ≈ 2 months; `improve-aeo-new-guide-rec-skill` should start after ~4 entries ≈ 6–8 weeks). If the minimum is not met, write a "too early to analyze" notice to run output and skip the PR.
The outer loop needs enough run log entries to identify real patterns, not noise. Require a minimum entry count before acting (the `improve-aeo-crosslink-skill` uses 8 entries ≈ 2 months at a weekly cadence; `improve-aeo-new-guide-rec-skill` should start after ~4 entries ≈ 4 months at a monthly cadence). If the minimum is not met, write a "too early to analyze" notice to run output and skip the PR.

This minimum must be stated explicitly in the skill's `## Schedule` section so the deployer knows when to start the agent.

Expand Down
18 changes: 12 additions & 6 deletions .agents/skills/aeo_crosslink_audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Do not:
The following environment secrets should be set in the Oz cloud agent environment:

- `PEEC_PAT` — Peec Personal Access Token for MCP authentication. Create one at **app.peec.ai → Company → API Keys → Personal Access Tokens**. If unavailable or expired, the audit proceeds with GSC and docs-only signals and logs "Peec: unavailable" in the run output.
- `SLACK_BOT_TOKEN` — Slack bot token for posting to `#growth-docs`. If unavailable, write the notification body to the run output instead and skip Slack posting.
- `BUZZ_SLACK_TOKEN` — Slack bot token for posting to `#growth-docs`, authenticating as the `buzz` bot. This is the account that posts to that channel; do not substitute another Slack token without confirming the bot is a member of the channel. If unavailable, write the notification body to the run output instead and skip Slack posting.
- `GROWTH_DOCS_SLACK_CHANNEL_ID` — Slack channel ID for `#growth-docs`. Find it in Slack by right-clicking the channel → Copy link (the ID begins with `C`). If unavailable, skip Slack posting.

The scheduled cloud agent must also include the Peec MCP server in its agent config (pass via `--mcp` flag or the agent config file `mcp_servers` key):
Expand All @@ -51,6 +51,8 @@ Do NOT print, log, commit, or include secret values in reports or Slack messages

Use the smallest reliable set of source data needed to justify link changes:
- **Peec** - Use the Peec MCP (configured in the agent with a Personal Access Token via the `PEEC_PAT` secret) to collect prompts, search queries, actions/recommendations, and source URLs for agents, cloud agents, and orchestration (last 30 days). Filter prompts and queries for relevance to the topic area. If the Peec MCP returns an error or is unavailable (missing `PEEC_PAT`, expired token, or connection failure), log "Peec: unavailable" in the run output and proceed with GSC and docs-only signals only.
- A cloud run may not expose `peec-ai` as a native tool even when it is in the agent config. If no tool appears, call `https://api.peec.ai/mcp` directly over JSON-RPC with `Authorization: Bearer $PEEC_PAT` (initialize, capture the `Mcp-Session-Id` header, then `tools/call`) rather than declaring Peec unavailable.
- Resolve the project with `list_projects` first; all other tools require `project_id`. `get_actions` needs `url_classification` for `scope=owned` and `scope=editorial` drill-downs, and `list_search_queries` returns `query_text` rather than `query`.
- **Google Search Console** - When available, use the environment's `GSC_SERVICE_ACCOUNT_CREDENTIALS_JSON` secret to inspect recent queries and pages related to agents, cloud agents, and orchestration. Never print, log, commit, or include the secret value in reports. If a GSC client requires a credentials file path, write the secret to a restricted temporary file, use it for the run, and remove it before finishing.
- **Docs repo** - Search existing pages under `src/content/docs/` for relevant source pages, link targets, and related terminology.

Expand Down Expand Up @@ -89,16 +91,16 @@ If Google Search Console data is unavailable, say what could not be verified and

This produces one perpetual, low-noise PR that accumulates every run's entry regardless of outcome. Reviewers merge it periodically (at minimum before each monthly `improve-aeo-crosslink-skill` run) so the log reaches `main`. If any git step fails, write the log entry to the run output instead and continue to step 8.

8. **Post Slack notification.** After writing the log entry, post the formatted message to `#growth-docs` using the Python snippet below. Python is preferred over curl because it reads `SLACK_BOT_TOKEN` from the environment (keeping the token out of process argv) and JSON-encodes the payload correctly regardless of newlines or special characters. If either secret is unavailable, write the notification body to the run output instead.
8. **Post Slack notification.** After writing the log entry, post the formatted message to `#growth-docs` using the Python snippet below. Python is preferred over curl because it reads `BUZZ_SLACK_TOKEN` from the environment (keeping the token out of process argv) and JSON-encodes the payload correctly regardless of newlines or special characters. If either secret is unavailable, write the notification body to the run output instead.

```bash
python3 - <<'SLACK_EOF'
import os, json, urllib.request, sys

token = os.environ.get("SLACK_BOT_TOKEN", "")
token = os.environ.get("BUZZ_SLACK_TOKEN", "")
channel = os.environ.get("GROWTH_DOCS_SLACK_CHANNEL_ID", "")
if not token or not channel:
print("SLACK_BOT_TOKEN or GROWTH_DOCS_SLACK_CHANNEL_ID not set — skipping Slack notification", file=sys.stderr)
print("BUZZ_SLACK_TOKEN or GROWTH_DOCS_SLACK_CHANNEL_ID not set — skipping Slack notification", file=sys.stderr)
sys.exit(0)

# Replace the triple-quoted string with the message from the Slack notification format section.
Expand All @@ -124,7 +126,11 @@ If Google Search Console data is unavailable, say what could not be verified and
SLACK_EOF
```

Replace `<message text here>` with the message from the appropriate format in the "Slack notification format" section. Do not print `SLACK_BOT_TOKEN` or `GROWTH_DOCS_SLACK_CHANNEL_ID` values in the run output or in any file.
Replace `<message text here>` with the message from the appropriate format in the "Slack notification format" section. Do not print `BUZZ_SLACK_TOKEN` or `GROWTH_DOCS_SLACK_CHANNEL_ID` values in the run output or in any file.

**If `chat.postMessage` returns `channel_not_found`**, the secrets being set is not sufficient — either the channel ID is stale or the bot is not a member of the channel. Do not treat this as a successful post. Instead:
1. Try resolving the channel by name: call `conversations.list` (types `public_channel,private_channel`) and look for `growth-docs`. If found, retry the post with that ID and report that the stored `GROWTH_DOCS_SLACK_CHANNEL_ID` is wrong so a human can correct the secret.
2. If the lookup also fails or returns `missing_scope`, the bot is not in the channel or lacks scope. Write the notification body to the run output, and state explicitly in the run output that the Slack post failed with `channel_not_found` — never imply it was delivered.

## Link quality rules

Expand Down Expand Up @@ -277,7 +283,7 @@ Oz run: [run URL]
Rules:
- Post on every run, including no-change runs.
- Never include raw secret values, personal access tokens, or credential file paths in the Slack message.
- Build the `Oz run` link at runtime — never hard-code the Oz host (for example `app.warp.dev` or `oz.warp.dev`). This agent may run on staging or production, and a hard-coded host resolves to the wrong environment (or a generic Runs page). Resolve the environment-correct link from your current run with `oz-dev run get "<your run ID>" --output-format json | jq -r '.session_link'`, substituting the run ID this agent is executing as.
- Build the `Oz run` link at runtime — never hard-code the Oz host (for example `app.warp.dev` or `oz.warp.dev`). This agent may run on staging or production, and a hard-coded host resolves to the wrong environment (or a generic Runs page). Resolve the environment-correct link from your current run with `oz run get "<your run ID>" --output-format json | jq -r '.session_link'`, substituting the run ID this agent is executing as. Cloud sandboxes ship the `oz` CLI; `oz-dev` is a local development build and is not present, so do not call it.
- If the Oz run URL is unavailable, omit that line rather than posting a broken link.

## Future expansion boundaries
Expand Down
Loading
Loading