feat(app): Browser Automations — smart connect + UI redesign (WIP)#3416
Draft
tofikwest wants to merge 67 commits into
Draft
feat(app): Browser Automations — smart connect + UI redesign (WIP)#3416tofikwest wants to merge 67 commits into
tofikwest wants to merge 67 commits into
Conversation
Adds the first step of the browser-automation connect wizard: a credentials form (username, password, optional authenticator setup key) shown before the live sign-in. On submit it resolves the auth profile and stores the login via the credentials endpoint so scheduled and manual runs can sign in on their own. Credential storage failures are non-fatal and fall back to manual sign-in. Part of the Browser Automations redesign (connect-a-login flow). Adds unit tests for the form (fields, validation, submit, password toggle).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Replaces the bare "enter a URL" prompt with the three-step explainer (01 Connect a login / 02 Describe what to capture / 03 Evidence on schedule), a single "Connect a vendor login" action, and the 1Password reassurance line. The URL is now collected in the connect credentials form rather than here. Part of the Browser Automations redesign (first-time setup).
Switches the first-time setup screen to the designer's 2c variant — a split card with the pitch and "Connect a Vendor Login" action on the left and a quiet "How it works" rail (01/02/03) on the right. Stacks to one column on mobile.
Adds a login-analysis backend: POST /v1/browserbase/analyze-login opens the vendor sign-in page in a throwaway cloud browser, detects the supported login methods (password / SSO / passkey, identifier type, extra fields), and returns a recommendation (ready / works-with-check-ins / manual). Reads a public page only — no credentials — and always degrades to a manual fallback if it can't be read. Foundation for the smart-connect flow. Unit tests for the recommendation logic and the analyzer service.
Replaces the connect flow with the "Browser-First Split" (1C) design: a stepper rail plus a browser stage that opens the vendor URL, runs AI login-method detection (/analyze-login), shows a recommendation (ready / works-with-check-ins / manual), opens the live sign-in, then captures the reusable credentials (username, password, authenticator setup key) after verification. Removes the superseded credentials-first form. Adds tests for the capture form. The SMS-only -> "enable an authenticator app" recommendation is deferred to the live sign-in step (2FA method isn't visible before sign-in).
tofikwest
changed the title
Rewrites the automations list to group automations under their vendor connection. Each connection shows its status (Connected / Needs reconnect / Needs your action / Not connected) with a one-click Reconnect for the ones that need a human. Adds a useBrowserProfiles hook to load connection statuses and wires reconnect through the existing live sign-in.
Extends credential capture beyond username/password/TOTP to any site-specific fields a login needs. The capture form gains an "add a field" control; the API stores extras as custom 1Password fields, resolves them at run time (via items.get, additive to the existing resolve path so the standard path is unchanged), and the sign-in agent fills each by label using variable substitution (only labels reach the model, never values).
…lable Move Browserbase session/context creation inside the try/catch so a missing BROWSERBASE_API_KEY (or any Browserbase outage) returns the manual-entry fallback instead of a 500 at the analyze step. Documents the required BROWSERBASE_API_KEY / BROWSERBASE_PROJECT_ID in .env.example.
…idence Below each connection's instructions, show a horizontal strip of recent run screenshots with pass/fail badges and timestamps. Clicking a tile opens a detail overlay with the full screenshot, the check verdict, any failure reason, timing, and re-run / open / download actions.
Add an ad-hoc test endpoint that runs a not-yet-saved instruction against a connection's live session and streams its progress, so the composer can show the AI working before the instruction is committed to the schedule. Reuses the same evidence runner scheduled runs use and persists nothing — no automation, no run record. The execution engine now emits each stage through an optional callback so the run's activity can be streamed live.
Replace the create/edit dialog with a two-pane instruction composer: write the instruction on the left, test it against the connection and watch the AI work live on the right, then save once it passes. Drops the separate name and schedule fields (name derives from the instruction; schedule lives on the task), and defaults the starting URL from the connection with an advanced override.
Since the composer no longer carries a schedule field, each instruction row now has an inline cadence picker (daily…yearly) that saves on change — so an instruction's schedule stays editable after the modal moved out. Remove the now-unused create/edit dialog the composer replaced.
Our panels are well under 1920px wide, so a larger native viewport lets the embedded live view (and evidence screenshots) downscale instead of upscale — upscaling was what looked soft, especially on HiDPI displays. Browserbase exposes no device-pixel-ratio setting, so viewport size is the only lever.
Instead of snapping from the live browser straight to the connected screen, hold a brief "Signed in" confirmation over the still-visible browser, then fade into the connected screen — a softer hand-off.
Surface the pass/fail check by default (drop the opt-in toggle and OPTIONAL label) since evidence instructions usually want a verdict; leave it blank to capture a screenshot only. Add clickable example chips under both the instruction and check fields — a short label inserts a full example — shown while the field is empty and hidden once the user starts typing.
…ctivity header The composer resolved the connection from a stale default host, so a freshly connected vendor could be tested against the wrong site's saved profile. Pass the connected URL up from the connect flow so instructions bind to the vendor the user actually connected. Also align the "Sign-in activity" header with the browser chrome bar (same 34px row).
After signing in, the run navigated back to the entered URL before checking the result — but some apps always show a login page at that URL, so a successful sign-in read as "did not complete" and forced a needless re-auth. Verify on the page the site lands on after login instead, and detect the signed-in state by the absence of a sign-in prompt rather than an avatar/profile marker (many apps have none), which was causing false negatives.
Give the live view a 16:9 frame matching the capture resolution so it fills the panel width instead of leaving wide side gaps, and stays sharp (downscaled).
The step fired whenever the first page looked like a sign-in prompt — including sites that always show login at the root, where nothing had expired. Reword it to state plainly that we're not signed in on this page and are signing in.
Show the connection as its hostname with an avatar and a status-driven dot (green verified / amber needs-reconnect / grey unverified) instead of the raw profile name with an always-green dot. Frame the check as "PASS when <your condition> / FAIL otherwise", so the pass/fail logic reads at a glance without a redundant second condition.
Sign-in alone can eat ~40s, so a 4-minute cap cut multi-step lookups off mid-run. Raise the budget and tell the agent to find its own way and read what's already on screen instead of over-navigating — the common cause of hitting the limit.
Line the instruction field up with the top of the live browser panel. Explain that the AI finds its own way (no exact steps needed) and what the advanced start-page override is for, and surface a clear, actionable message when a test run times out.
Keep navigation screenshot-based but make the computer-use model configurable via env (Claude or OpenAI, no per-site tuning) and default to the newest Claude our Stagehand supports. Give the agent more steps and a generous time budget so it can recover from a wrong turn on a complex site, and tell it to verify the page matches the request and fix a wrong item before finishing. The page-reading model is a separate env knob too.
Use openai/computer-use-preview for screenshot-based navigation by default — it's purpose-built for GUI agents — with a safe fallback to Claude when no OpenAI key is set. Page reading / verdicts stay on a general model.
If the primary computer-use model is unavailable mid-run (preview access, rate limits, upstream errors), retry navigation once on a Claude computer-use model instead of failing the run. Use claude-sonnet-4-6 for the fallback — sonnet-5 isn't a computer-use model Stagehand supports.
Switch the default computer-use model to openai/gpt-5.6-terra — the balanced tier of OpenAI's newest family, strong on browser automation at about half Sol's cost. Still env-swappable (e.g. gpt-5.6-sol for max accuracy) with the Claude fallback intact.
Two evidence-quality fixes: tell the agent to open the specific item an instruction names so the screenshot shows just that (not a whole list), and pass the instruction into the evaluator so a pass/fail check is judged about the intended target — not any matching value that happens to appear elsewhere on the page (which let an unrelated product's price satisfy the check).
The left field label and the right "Test run" header had different heights (the header carries a status pill), so the textarea and the browser view didn't line up. Fix both headers to the same height so they share one baseline.
The AI is meant to find its own way, so the composer no longer asks for a start page — instructions always run from the connection. Removes the advanced override and its field.
When automated sign-in succeeds, capture the page the app lands on and use it as the connection's URL. Runs then open the authenticated app directly instead of a login page, so the persisted session is reused and credentials aren't re-entered every time. Genuine expiry still redirects to login and re-authenticates.
Screenshots were viewport-height, so a short page left a tall band of empty viewport below the content — a wide gap above the audit overlay. Capture the full page instead: the overlay sits right under the real content and the evidence is complete.
After a successful sign-in, go straight into writing the first instruction instead of bouncing back to a "create" button. And make multiple connections first-class: each connection has its own "Add instruction", and a "Connect another vendor" button adds more vendors — any number of connections, each with any number of instructions.
Passkeys are device-bound and can't complete in Comp's cloud browser, so offering one was a dead end. Remove passkey from the method chooser. If a site only supports passkeys, show a plain "can't be connected here" message instead of a sign-in that can never finish.
Picking SSO no longer just dumps you in a blank browser. The AI opens the vendor, clicks through to your identity provider, and streams the steps — then hands the live browser over for you to finish the IdP login (which we can't hold or automate). If the saved session is still valid it signs in with no interaction at all. Reuses the live sign-in run via a mode flag.
Reconnect no longer dumps you into a fully manual browser. A password connection re-signs-in automatically with the stored credentials (no interaction); an SSO connection has the AI drive to the identity provider so you only finish there. Reuses the sign-in run and just refreshes the list when done.
…bility Sign-in, take-over, SSO and reconnect sessions now use a 1280x800 viewport so the page renders larger (and stays crisp) in the embedded live view — easier to read and fill. Unattended evidence runs keep the 1920x1080 viewport for sharp full-page screenshots. Viewport is now a parameter, defaulting to capture.
Add a glowing border around the live browser: a primary-color glow while the AI is acting, amber when it's the user's turn. Makes it obvious at a glance whether to wait or take over, on the connect sign-in, the instruction test run, and execution/auth views. Decorative and click-through, so take-over still works.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature is functionally code-complete end-to-end
POST /analyze-login: reads a vendor's login methods, returns a recommendation. Public page only, no creds, SSRF-guarded, manual fallback.Verification
Not done — needs you / infra (can't be done from code alone)
OP_SERVICE_ACCOUNT_TOKEN, PostHog flag on, a real Browserbase run, a deploy. This also confirms the 1Password WASM loads in the Trigger worker.Cosmetic (already covered, not separate components)
sm).