Sync browser agent features from ConnectOnion CLI

[wu-changxing]

Sync Browser Agent Features from ConnectOnion CLI Version

Summary

The standalone browser-agent and ConnectOnion's CLI browser agent (@connectonion/cli/browser-agent) have diverged significantly. This issue tracks features from the CLI version that should be adopted to improve UX and reliability.

Current State

• Standalone: Multi-agent architecture, deep research capabilities, advanced features
• CLI Version: Better UX, defensive error handling, platform optimizations
• Last Sync: Never systematically synced

Features to Adopt from CLI Version

1. Home Directory Profile Management

Priority: High

Current: Profile at .co/chrome_profile (project-specific)
Proposed: Profile at ~/.co/browser_profile (persistent across projects)

Benefits:

• Sessions persist across different projects
• User logs in once, cookies saved forever
• More intuitive behavior for users

Implementation:

# In WebAutomation.__init__()
if profile_path:
    self.chrome_profile_path = str(profile_path)
else:
    self.chrome_profile_path = str(Path.home() / ".co" / "browser_profile")

Files to modify:

• tools/web_automation.py (lines 40-43)

---

2. macOS Chrome Binary Detection

Priority: Medium

Problem: Playwright's bundled Chromium is unsigned and crashes on macOS in non-headless mode

Solution:

def open_browser(self, headless: bool = None) -> str:
    launch_kwargs = dict(
        headless=headless,
        args=['--disable-blink-features=AutomationControlled'],
        ignore_default_args=['--enable-automation'],
        timeout=120000,
    )

    # macOS fix: Use system Chrome for non-headless mode
    if not headless:
        import sys
        if sys.platform == 'darwin':
            chrome_path = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
            if os.path.exists(chrome_path):
                launch_kwargs['executable_path'] = chrome_path

    self.browser = self.playwright.chromium.launch_persistent_context(
        str(profile_dir),
        **launch_kwargs,
    )

Files to modify:

• tools/web_automation.py (open_browser method)

---

3. Smart URL Handling

Priority: Low

Enhancement: Auto-add https:// for partial URLs

Current:

def go_to(self, url: str) -> str:
    self.page.goto(url, wait_until="load")

Proposed:

def go_to(self, url: str) -> str:
    if not url.startswith(('http://', 'https://')):
        url = f'https://{url}' if '.' in url else f'http://{url}'

    self.page.goto(url, wait_until='domcontentloaded', timeout=30000)
    self.page.wait_for_timeout(2000)  # Wait for dynamic content
    self.current_url = self.page.url
    return f"Navigated to {self.current_url}"

Benefits:

• Users can type example.com instead of https://example.com
• More forgiving UX

Files to modify:

• tools/web_automation.py (go_to method, line 80-84)

---

4. Defensive Null Checking

Priority: Medium

Issue: Methods don't check if browser is open, leading to cryptic errors

Pattern to adopt:

def method_name(self, ...) -> str:
    if not self.page:
        return "Browser not open"

    # ... rest of implementation

Apply to all methods in:

• tools/web_automation.py: get_text, select_option, check_checkbox, wait_for_element, etc.

---

5. Additional Helper Methods

Priority: Low

Add convenience methods from CLI version:

def get_current_url(self) -> str:
    """Get the current page URL."""
    if not self.page:
        return "Browser not open"
    return self.page.url

def get_current_page_html(self) -> str:
    """Get the HTML content of the current page."""
    if not self.page:
        return "Browser not open"
    return self.page.content()

def get_urls(self, domain_filter: str = "") -> List[str]:
    """Extract all unique URLs from the current page.

    Args:
        domain_filter: Only return URLs containing this string
    """
    if not self.page:
        return []

    urls = self.page.evaluate("""
        (filter) => {
            const seen = new Set();
            const result = [];
            for (const a of document.querySelectorAll('a[href]')) {
                const href = a.href;
                if (href && !seen.has(href) && (!filter || href.includes(filter))) {
                    seen.add(href);
                    result.push(href);
                }
            }
            return result;
        }
    """, domain_filter)
    return urls or []

def set_viewport(self, width: int, height: int) -> str:
    """Set the browser viewport size."""
    if not self.page:
        return "Browser not open"
    self.page.set_viewport_size({"width": width, "height": height})
    return f"Viewport set to {width}x{height}"

def wait(self, seconds: float) -> str:
    """Wait for a specified number of seconds."""
    if not self.page:
        return "Browser not open"
    self.page.wait_for_timeout(seconds * 1000)
    return f"Waited for {seconds} seconds"

Files to modify:

• tools/web_automation.py (add after existing methods)

---

6. Enhanced Screenshot Method

Priority: Low

Current signature:

def take_screenshot(self, filename: str = None) -> str:

Proposed signature:

def take_screenshot(self, url: str = None, path: str = "",
                   width: int = 1920, height: int = 1080,
                   full_page: bool = False) -> str:

New features:

• Navigate to URL before screenshot
• Control viewport size
• Full-page capture option
• Auto-generate timestamped filenames

Files to modify:

• tools/web_automation.py (take_screenshot method)

---

7. Better Manual Login UX

Priority: Medium

Current:

def wait_for_manual_login(self, site_name: str = "the website") -> str:
    print(f"\n{'='*60}\n⏸️  MANUAL LOGIN REQUIRED\n{'='*60}")
    print(f"Please login to {site_name} in the browser window.")
    input("Press Enter to continue...")
    return f"User confirmed login to {site_name}"

Proposed:

def wait_for_manual_login(self, site_name: str = "the website") -> str:
    if not self.page:
        return "Browser not open"

    print(f"\n{'='*60}")
    print(f"  MANUAL LOGIN REQUIRED")
    print(f"{'='*60}")
    print(f"Please login to {site_name} in the browser window.")
    print(f"Once you're logged in and ready to continue:")
    print(f"  Type 'yes' or 'Y' and press Enter")
    print(f"{'='*60}\n")

    while True:
        response = input("Ready to continue? (yes/Y): ").strip().lower()
        if response in ['yes', 'y']:
            print("Continuing automation...\n")
            return f"User confirmed login to {site_name} - continuing"
        else:
            print("Please type 'yes' or 'Y' when ready.")

Files to modify:

• tools/web_automation.py (wait_for_manual_login method, lines 218-223)

---

Implementation Plan

Phase 1: Critical UX Improvements

• Home directory profile (Feature Incremental optimization #1)
• Defensive null checking (Feature Sync browser agent features from ConnectOnion CLI #4)
• Better manual login UX (Feature Update to use create_agent factory pattern #7)

Phase 2: Platform Support

• macOS Chrome binary detection (Feature Deep research feature: Added base deep-research functionality #2)

Phase 3: Nice-to-Have

• Smart URL handling (Feature updated to newest connectonion version #3)
• Helper methods (Feature Sync advanced browser agent features from standalone browser-agent #5)
• Enhanced screenshot (Feature Improve keyboard typing and screenshot verification #6)

Testing Checklist

After implementing:

• Test on macOS in non-headless mode
• Test profile persistence across different projects
• Test with browser not opened (defensive checks)
• Test manual login flow with invalid inputs
• Verify all helper methods work as expected

Related Issues

• This issue created as part of browser agent sync effort
• Related: ConnectOnion CLI browser agent issue (link TBD)

Files to Modify

• tools/web_automation.py - Main implementation file
• README.md - Update documentation for new features
• CLAUDE.md - Update development guidance

Migration Notes

These changes are backward compatible - existing code will continue to work. New features are additive.

[wu-changxing]

🤖 Night Runner - Implementation Plan

Issue #4: Sync browser agent features from ConnectOnion CLI

---

🔍 Summary

Sync 7 UX and reliability improvements from ConnectOnion CLI's browser agent into the standalone browser-agent. All changes are additive and backward-compatible. The main file to modify is tools/web_automation.py.

📊 Workflow

CURRENT STATE                          PROPOSED STATE
─────────────                          ──────────────

Profile per project:                   Profile global:
.co/chrome_profile/                    ~/.co/browser_profile/
  └── cookies lost on cd               └── cookies persist everywhere

open_browser() → Playwright Chromium   open_browser() → System Chrome (macOS)
                 ↑ unsigned, crashes                    ↑ signed, stable

go_to("google.com") → ❌ fails         go_to("google.com") → ✅ auto https://

method() → crashes if no browser       method() → "Browser not open" (friendly)

wait_for_manual_login():               wait_for_manual_login():
  input("Press Enter...")                requires explicit "yes"/"Y" confirmation

5 public methods                       10 public methods (+5 helpers)

📁 Files to Change

browser-agent/
└── tools/
    └── web_automation.py      ← ALL changes here
        │
        ├── __init__()         → [1] profile path: .co/ → ~/.co/
        ├── open_browser()     → [2] macOS Chrome detection
        ├── go_to()            → [3] smart URL (auto https://)
        ├── get_text()         → [4] add null check
        ├── select_option()    → [4] add null check
        ├── check_checkbox()   → [4] add null check
        ├── wait_for_element() → [4] add null check
        ├── wait_for_text()    → [4] add null check
        ├── take_screenshot()  → [6] add url/width/height/full_page params
        ├── wait_for_manual_login() → [7] better UX prompt
        │
        ├── + get_current_url()      [5] new helper
        ├── + get_current_page_html() [5] new helper
        ├── + get_urls()             [5] new helper
        ├── + set_viewport()         [5] new helper
        └── + wait()                 [5] new helper

🔄 Data Flow

Feature 1 - Profile Path:
  __init__(profile_path=None)
       │
       ▼
  profile_path provided? ──Yes──▶ use as-is
       │
       No
       ▼
  Path.home() / ".co" / "browser_profile"   ✅ ~/.co/browser_profile

─────────────────────────────────────────────────────

Feature 2 - macOS Chrome:
  open_browser(headless=False)
       │
       ▼
  headless == False?
       │
       Yes
       ▼
  sys.platform == 'darwin'?
       │
       Yes
       ▼
  /Applications/Google Chrome.app exists?
       │
       Yes                     No
       ▼                       ▼
  executable_path=Chrome    Playwright Chromium

─────────────────────────────────────────────────────

Feature 3 - Smart URL:
  go_to("google.com")
       │
       ▼
  starts with http:// or https://?
       │
       No
       ▼
  '.' in url?  ──Yes──▶  "https://google.com"
       │
       No
       ▼
  "http://google.com"
       │
       ▼
  page.goto(url, wait_until='domcontentloaded', timeout=30000)
  page.wait_for_timeout(2000)

📝 Implementation Steps

Phase 1: Critical UX (High Impact)

1. Home directory profile (__init__, line 43)

   • Change Path.cwd() / ".co" / "chrome_profile" → Path.home() / ".co" / "browser_profile"
   • Also update session_file path from Path.cwd() → Path.home()

2. Defensive null checks (methods: get_text, select_option, check_checkbox, wait_for_element, wait_for_text)

   • Add if not self.page: return "Browser not open" at top of each method

3. Better manual login UX (wait_for_manual_login, line 218)

   • Replace input("Press Enter...") with loop requiring "yes"/"y" confirmation

Phase 2: Platform Support (Reliability)

4. macOS Chrome detection (open_browser, line 45)
   • Add import sys at top of function
   • After headless check: if not headless and sys.platform == 'darwin', check for Chrome and set executable_path

Phase 3: Nice-to-Have (Usability)

5. Smart URL handling (go_to, line 80)

   • Add URL prefix check before page.goto()
   • Change wait_until="load" → wait_until='domcontentloaded' with timeout=30000
   • Add page.wait_for_timeout(2000) after navigation

6. Helper methods (add after wait_for_text)

   • get_current_url() → self.page.url
   • get_current_page_html() → self.page.content()
   • get_urls(domain_filter="") → JS evaluate to extract all <a href> links
   • set_viewport(width, height) → self.page.set_viewport_size()
   • wait(seconds) → self.page.wait_for_timeout(seconds * 1000)

7. Enhanced screenshot (take_screenshot, line 124)

   • Add params: url=None, width=1920, height=1080, full_page=False
   • If url provided, navigate first
   • If width/height provided, call set_viewport_size before capture
   • Pass full_page=full_page to page.screenshot()

⚠️ Risks & Considerations

• ~/.co/browser_profile migration: existing users with project-local profiles won't auto-migrate cookies — first run will be a fresh session
• macOS Chrome path is hardcoded; if user has Chrome in a custom location it won't be detected (acceptable for now per YAGNI)
• wait_for_timeout(2000) in go_to adds 2s to every navigation — keep an eye on test speed

🧪 Testing Strategy

• Open browser non-headless on macOS → confirm Chrome (not Chromium) launches
• Navigate to a site that requires login → confirm cookies persist when running from a different project directory
• Call get_text() / select_option() before open_browser() → confirm returns "Browser not open" instead of crashing
• Call go_to("google.com") → confirm navigates successfully
• Test wait_for_manual_login with invalid inputs → confirm loop repeats until yes/Y
• Call take_screenshot(url="google.com", full_page=True) → confirm navigates + captures full page

---

Reply with LGTM to approve this plan and create a PR.

🤖 Automated by Night Runner - OpenOnion's 24/7 code worker