Skip to content

docs(skills): use live Peec MCP in aeo_new_guide_recommendations - #470

Open
rachaelrenk wants to merge 6 commits into
mainfrom
rrenk/aeo-peec-mcp
Open

docs(skills): use live Peec MCP in aeo_new_guide_recommendations#470
rachaelrenk wants to merge 6 commits into
mainfrom
rrenk/aeo-peec-mcp

Conversation

@rachaelrenk

@rachaelrenk rachaelrenk commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

aeo_new_guide_recommendations was written when cloud agents could not authenticate with Peec, so it read a snapshot committed to the buzz repo and hard-exited when that snapshot was 14 days or older. That constraint no longer holds: aeo_crosslink_audit already calls Peec MCP directly using the PEEC_PAT secret, and its 2026-08-03 run logged "Peec available" and opened #453 while the committed snapshot was 40 days old.

This updates the new-guide skill to the same live-MCP pattern and switches its documented cadence from every two weeks to monthly.

Why this matters

The snapshot in buzz was last generated 2026-06-24. Scheduling the new-guide agent as written would have produced a stale-snapshot exit on every run, since nothing refreshes the snapshot on a cadence and the 14-day gate is shorter than a monthly interval.

Changes

.agents/skills/aeo_new_guide_recommendations/SKILL.md

  • Add PEEC_PAT and the peec-ai mcp_servers config block to environment requirements, mirroring aeo_crosslink_audit.
  • Replace the snapshot source-data section and 14-day freshness gate with live Peec MCP collection.
  • Replace the "Stale snapshot report" section with a "Peec unavailable" degrade path: log the specific failure, continue docs-only at a higher confidence bar, and fall back to the no-run report.
  • Update the run output, run log, and Slack formats to drop snapshot states in favor of Peec [available | unavailable].
  • Cadence changed to monthly in the frontmatter description, intro, and outer-loop timing estimate.
  • Mark refresh-peec-aeo-snapshot as legacy for this pipeline.

.agents/references/skill-authoring-guidelines.md

  • The "Source data and freshness" section taught that cloud agents cannot call Peec MCP and must use a committed snapshot with a freshness gate. That guidance is now inverted: prefer token-authenticated live calls, document the required mcp_servers config, define an explicit unavailable path, and treat snapshots as a last resort that needs a paired refresh schedule.
  • Corrected downstream references to stale-snapshot reports and the outer-loop entry-count timing.

.agents/logs/aeo_new_guide_recommendation_runs.md

  • Header updated from "bi-weekly" to monthly, and the stale-snapshot outcome removed.

Follow-up (not in this PR)

The schedule for this skill still needs to be created with the peec-ai MCP server wired to PEEC_PAT, and pointed directly at the skill spec rather than the Docs Agent profile (which would attach the unrelated release_updates skill).

Open questions for human review

  • Should aeo_crosslink_audit and this skill share the Peec query scope, or is per-skill divergence fine?
  • The monthly cadence is a judgment call. If the Docs team wants a tighter planning loop, the same config supports every two weeks with no skill change.

Update: findings from the validation run

Run 019fce3f executed the updated skill end to end. Peec MCP authenticated successfully with PEEC_PAT (HTTP 200, list_projects returned the Warp project) and produced 4 briefs, confirming the live-MCP approach. It also surfaced three real defects, now fixed here.

Peec MCP call contract was under-specified. The agent had no native peec-ai tool despite the MCP being in the agent config, so it called the JSON-RPC endpoint directly. The skill now documents that path plus the contract details it had to discover: list_projects must run first for project_id, get_actions drill-downs require url_classification, and list_search_queries returns query_text. It also notes that most EDITORIAL action rows are outreach pitches rather than docs gaps, so OWNED how-to/article rows are the real signal.

oz-dev does not exist in cloud sandboxes. It ships only in the local Warp dev app. Six occurrences across seven skill files instructed agents to call it to resolve their session link, so every scheduled agent hit command-not-found and silently dropped the run link from its Slack notification. Swept to oz across weekly-404-monitor, improve-drafting-skills, improve-aeo-crosslink-skill, and improve-404-monitor-skill, plus both AEO skills and the authoring guidelines.

Slack posting fails with channel_not_found. SLACK_BOT_TOKEN authenticates, but the stored GROWTH_DOCS_SLACK_CHANNEL_ID does not resolve to a channel either bot can see. Both AEO skills now attempt a channel-name lookup, then fall back to run output and report the failure explicitly instead of logging a clean run. This affects aeo_crosslink_audit too — its weekly #growth-docs notifications have not been landing.

Still needs a human

The GROWTH_DOCS_SLACK_CHANNEL_ID secret value is wrong and must be corrected outside this PR. Slack delivery remains unvalidated until then.

The scheduled cloud agent can authenticate with Peec via the PEEC_PAT
secret and the peec-ai MCP server, as aeo_crosslink_audit already does.
Replace the committed-snapshot workflow and its 14-day staleness gate
with live Peec MCP calls, and switch the documented cadence to monthly.

- Add PEEC_PAT and the peec-ai mcp_servers config to environment requirements
- Replace the snapshot source-data section and freshness gate with live MCP
- Replace the stale-snapshot report with a "Peec unavailable" degrade path
- Update run log, run output, and Slack formats to drop snapshot states
- Mark refresh-peec-aeo-snapshot as legacy for this pipeline
- Update skill-authoring-guidelines to prefer token-authenticated live
  calls over snapshots, since the old guidance taught the reverse

Co-Authored-By: Warp Agent <agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Aug 4, 2026
@vercel

vercel Bot commented Aug 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 4, 2026 8:32pm

Request Review

@oz-for-oss

oz-for-oss Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@rachaelrenk

I'm starting a first review of this pull request.

You can view the conversation on Warp.

I completed the review and no human review was requested for this pull request.

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

@oz-for-oss oz-for-oss Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

This PR updates the AEO new-guide recommendation skill and related guidance to use live Peec MCP data with a documented unavailable path instead of a committed snapshot freshness gate.

Concerns

  • The new Peec-unavailable degraded mode conflicts with the unchanged brief quality gate requiring every recommendation to be backed by a Peec signal, so docs-only low-confidence briefs cannot actually pass the skill's own final checks.

Verdict

Found: 0 critical, 1 important, 0 suggestions

Request changes

Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).

Powered by Oz

Before finalizing each brief:

- **Real signal** — Each recommendation is backed by a Peec snapshot signal, not generic SEO advice.
- **Real signal** — Each recommendation is backed by a Peec signal, not generic SEO advice.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [IMPORTANT] This still requires every recommendation to have a Peec signal, which contradicts the new Peec-unavailable path that can produce docs-only low-confidence briefs. Relax this rule for unavailable-mode briefs or the degraded path can never produce output.

@rachaelrenk rachaelrenk self-assigned this Aug 4, 2026
Validation run 019fce3f-6517-754e-8e97-7af8531e5d62 executed the updated
skill end to end and surfaced concrete gaps.

- Document the Peec MCP call contract: cloud runs may not surface peec-ai
  as a native tool, so call the JSON-RPC endpoint directly; resolve
  project_id via list_projects; get_actions drill-downs require
  url_classification; list_search_queries returns query_text
- Tell the agent to treat OWNED how-to/article rows as the docs signal,
  since many EDITORIAL rows are outreach rather than content gaps
- Add channel_not_found handling for Slack: attempt a lookup by channel
  name, then fall back to run output and report the failure explicitly
- Replace oz-dev with oz for run-link resolution. oz-dev is a local dev
  build and is absent from cloud sandboxes, so   build and is absent from cloud sandboxes, so   build and is absent from cloud sandboxes, so   build and is absent from cloud sandboxes, so   build and is hored-By: Warp Agent <agent@warp.dev>
oz-dev is a local development build shipped in the Warp dev app; cloud
sandboxes only have oz. Every scheduled skill calling oz-dev to resolve
its session link hit command-not-found and silently omitted the run link
from its Slack notification.

- weekly-404-monitor (run link + the schedule create deploy command)
- improve-drafting-skills (conversation fetch + run link)
- improve-aeo-crosslink-skill
- improve-404-monitor-skill

Also add the channel_not_found recovery path to aeo_crosslink_audit,
which shares the GROWTH_DOCS_SLACK_CHANNEL_ID secret and has been
failing to deliver its #growth-docs notifications for the same reason.

Co-Authored-By: Warp Agent <agent@warp.dev>
SLACK_BOT_TOKEN authenticates as the hubble bot, which is not a member
of #growth-docs and cannot be invited. buzz is the account that posts to
that channel, so both AEO skills now use BUZZ_SLACK_TOKEN.

Name the expected bot in each skill's environment requirements, and add
guidance to the authoring reference that Slack tokens authenticate as
different bots with different channel memberships, so a token that
authenticates is not necessarily a token that can deliver.

Co-Authored-By: Warp Agent <agent@warp.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant