Stagehand Reasoning Layer

An intelligent reasoning layer built on top of Stagehand that uses Gemini models to provide adaptive web automation with pattern learning.

Features

• Adaptive Automation: Automatically adapts to UI changes and recovers from failures
• Pattern Learning: Learns from successful trajectories to improve future automation
• Gemini Integration: Uses Gemini 2.0 Flash for planning, execution, and validation
• Visual Analysis: Screenshot-based validation and comparison
• Trajectory Recording: Comprehensive recording of automation sessions with visual snapshots

Quick Start

1. Prerequisites

   npm install

2. Environment Setup Create a .env file with:

   GOOGLE_API_KEY=your_gemini_api_key
   OPENAI_API_KEY=your_openai_key  # optional
   ANTHROPIC_API_KEY=your_anthropic_key  # optional
   

3. Start Demo Site

   npm run demo-site

4. Run Basic Automation

   npm run demo

5. Run Reasoning Demo

   npm run reasoning-demo

Architecture

Core Components

• Reasoning Orchestrator: Coordinates all reasoning components
• Pattern Library: JSON-based storage of automation patterns
• Session Manager: Maintains context across automation steps
• Gemini Models: Specialized model wrappers for different tasks
• Trajectory Recording: Captures and analyzes automation sessions

Key Files

• src/reasoning/ - Main reasoning layer implementation
• src/trajectory/ - Trajectory recording and analysis
• demo-site/ - Test application with v1/v2 versions
• patterns/library.json - Learned automation patterns

Usage Examples

Basic Automation

import { runSiteTest } from './src/index';
await runSiteTest('demo');

Reasoning-Powered Automation

import { ReasoningOrchestrator } from './src/reasoning';
import { createReasoningStagehand } from './src/reasoning/stagehand-integration';

const orchestrator = new ReasoningOrchestrator();
const { browserInterface } = await createReasoningStagehand({
  env: "LOCAL",
  modelName: "gemini-2.0-flash",
  modelClientOptions: { apiKey: process.env.GOOGLE_API_KEY }
});

const result = await orchestrator.executeTask(
  'Login and view account statement',
  browserInterface
);

Trajectory Comparison

import { compareTrajectories } from './src/compare-trajectories';
await compareTrajectories('trajectory1', 'trajectory2');

Scripts

• npm run demo - Run basic automation demo
• npm run reasoning-demo - Run reasoning layer demo
• npm run demo-site - Start demo banking application
• npm run compare - Compare trajectory recordings

Development

Built with TypeScript and Node.js. Uses Stagehand for browser automation and Gemini for AI reasoning capabilities.

The reasoning layer learns from successful automation patterns and adapts to UI changes, making it robust for real-world web automation scenarios.