Skip to content

AGENTS.md

Latest commit

 

History

History
151 lines (114 loc) · 7.26 KB

AGENTS.md

File metadata and controls

151 lines (114 loc) · 7.26 KB
Raw

TextBrowser — Agent Usage Guide

You are an AI agent. This document helps you use the TextBrowser extension effectively.

Available Tools

Tool Purpose When to use
browser_navigate Open a URL, return page context First visit to a site or explicit navigation
browser_click Click an element by CSS, XPath, or text Interact with links, buttons, form elements
browser_type Type text into an input field Fill forms, search boxes, login fields
browser_scroll Scroll the page or an element into view Explore long pages, find off-screen content
browser_screenshot Capture current page + OCR text Re-read page without changing anything
browser_read Get current page context (no action) Check if page state changed after an action
browser_evaluate Execute arbitrary JavaScript Escape hatch for shadow DOM, complex interactions

Dual-Mode Design: When to Use Visual vs Text-Only

⚠️ DEFAULT: Always use text-only mode first. Only switch to visual mode when the task explicitly requires seeing pixels.

Decision Framework

Ask yourself: "Does this task require seeing colors, positions, or images?"

✅ Text-Only Mode (visual: false or omit) — Use for:

Category Examples Why text works
Navigation Open a URL, find a link, click a button DOM elements + OCR reveal all interactive items
Form filling Login, search, type into inputs CSS selectors + text labels identify fields
Data extraction Scrape a table, read article text OCR + DOM textContent capture all visible text
Workflow automation Multi-step process (login → search → click) Text elements + OCR track state changes
API/Admin panels Gitea, Firebase Console, GitHub Structured UI labels are fully readable via text
Code review Read docs, explore repos Content is textual by nature

🖼️ Visual Mode (visual: true) — Use ONLY for:

Category Examples Why images are needed
Layout verification "Does the button align with the header?" "Is the sidebar overlapping?" Pixel positions matter
Color/theme checks "Is the dark mode working?" "Does this component match the brand color?" Colors are invisible in text
Visual regression "Compare before/after screenshots of this component" Diffing pixels
Image content "What does this dashboard chart show?" "Read the text in this logo" Images are non-text
UI design review "Does the spacing look right?" "Is the font readable?" Visual aesthetics
Debugging rendering "Why is this element invisible?" "Check if CSS is loading" CSS bugs invisible to OCR

🎯 Quick Reference

User says: "Open Gitea and explore repos"        → text-only ✅
User says: "Login to LinkedIn and post"           → text-only ✅
User says: "Check if the dark mode looks correct" → visual 🖼️
User says: "Is the button centered on the page?"  → visual 🖼️
User says: "Read the article content"             → text-only ✅
User says: "Compare this page to the mockup"      → visual 🖼️

Cost Comparison

Mode Avg Tokens Relative Cost When
Text-only ~200-400 1x Default — 90% of tasks
Visual ~1,500-3,000 5-15x Only when pixels matter

Escalation Path

If text-only mode provides insufficient information for the task, state why and escalate:

"I can see the login form (elements [22]-[33]) but need to verify the button color matches the brand palette. Let me switch to visual mode."

Don't silently switch to visual — explain the rationale. This helps users understand token costs and keeps you accountable.

Session Persistence

The browser maintains one Playwright Chromium instance per Pi session. All tools share the same page, cookies, and state. This means:

  • ✅ After browser_navigate, browser_click and browser_type work on the same page
  • ✅ Login sessions persist across tool calls
  • ⚠️ browser_navigate reloads the page — LinkedIn/GitHub may kill sessions

Best Practices

⚠️ CRITICAL: Prefer browser_read + browser_click over browser_navigate. Every browser_navigate triggers a full page load, destroying login sessions and current state. Once you're on a site, stay there. Use read/click/type/scroll to move around. Only use browser_navigate when you genuinely need a completely different URL domain.

⚠️ COST: Default to text-only mode. Visual mode burns 5-15× more tokens. Use visual: true ONLY for layout, colors, design review, or image-content tasks. If text-only is insufficient, explain why before switching.

Rule 1: Navigate once, then read & click

browser_navigate(url="https://example.com")   ← open the page
browser_read()                                  ← check what's there
browser_click(text="Sign In")                   ← interact
browser_type(selector="#email", text="user")    ← fill form

Avoid browser_navigate after logging in — it destroys the session.

Rule 2: Use headless: false for visible browsing

The user can see and interact with a visible browser window. After opening with headless: false, all subsequent tools inherit this setting — the window stays open.

Rule 3: Use element indices for precision

The page context returns elements with [index] markers. Refer to: "click element [15] (the Sign In button)"

Rule 4: browser_evaluate is your shadow DOM escape hatch

When elements live inside shadow DOM (LinkedIn, some React apps), standard clicks fail. Use browser_evaluate to query and interact through el.shadowRoot:

// Find and click a button inside shadow DOM
const el = document.querySelector('#host').shadowRoot.querySelector('button');
el.click();

Rule 5: Wait after actions

After clicking, the page may need time to load. Use browser_read to confirm the new state before the next action.

Common Patterns

Form login

browser_navigate(url="https://site.com/login", headless=false)
browser_type(selector="#username", text="user")
browser_type(selector="#password", text="pass")
browser_click(selector="button[type='submit']")
browser_read()  ← verify logged in

Explore with filters

browser_navigate(url="https://site.com/search")
browser_type(selector="input[placeholder='Search']", text="query")
browser_click(text="Search")
browser_scroll(direction="down", amount=500)
browser_read()

Extract data via JS

browser_evaluate(script="Array.from(document.querySelectorAll('.item')).map(el => el.textContent)")

Known Limitations

  • Shadow DOM: LinkedIn, Salesforce, and some SPA frameworks use shadow DOM that blocks standard clicks. Use browser_evaluate.
  • Login sessions: Some sites (LinkedIn) aggressively expire sessions when navigating. Log in once, then use read/click only.
  • Captchas / 2FA: Cannot bypass security checks. Ask the user to complete them in the visible browser window.
  • File downloads: Not supported. Use browser_evaluate to extract data as text.
  • OCR accuracy: Tesseract OCR depends on screenshot quality. Complex layouts, small fonts, or low-contrast text may produce garbled results. When OCR is unreliable, rely on DOM elements instead.