Author: Abdullah Alsayed

Computer Use Agent Session Manager

A scalable backend for managing concurrent Claude computer-use agent sessions, replacing the default Streamlit interface with a FastAPI backend, real-time WebSocket streaming, and a Next.js frontend.

Demo

• Full Demo with Usecase 1 & 2
• Multi-turn Chat Demo (2026-03-18)
• Overview: WebSocket streaming & architecture
• Live VNC desktop + WebSocket events side-by-side

Tech Stack

• Backend: FastAPI, SQLAlchemy 2.0 (async), PostgreSQL, Docker SDK
• Frontend: Next.js 16, React 19, TypeScript, TailwindCSS, SWR
• Agent: Anthropic computer-use-demo container with Claude Sonnet 4
• Infrastructure: Docker Compose, Alembic migrations

Prerequisites

• Docker Engine 24.0+
• Docker Compose 2.20+
• Anthropic API key with credits
• 4GB+ free RAM

Quick Start

git clone https://github.com/abd-als/test_Energent.git
cd test_Energent
cp docker/.env.example docker/.env
# Edit docker/.env and set ANTHROPIC_API_KEY
docker compose -f docker/docker-compose.yml up --build

• Frontend: http://localhost:3000
• API docs: http://localhost:8000/docs
• Health check: http://localhost:8000/health

Architecture

Sequence Diagram

Session Lifecycle

1. POST /sessions with task description
2. Backend allocates port atomically (PostgreSQL advisory locks)
3. energent-agent:latest container spawned (pre-built by docker-compose)
4. Health check polls noVNC until ready
5. relay.py started inside container via docker exec (FastAPI server on port 8080)
6. Health check polls relay /health until ready
7. WebSocket opened at /sessions/{id}/stream — backend sends history_snapshot
8. Backend POSTs /task to relay → relay calls Claude sampling_loop
9. Relay webhooks each event to POST /sessions/{id}/events — persisted to DB, broadcast via EventBus to WebSocket
10. On task completion, relay emits done event — frontend enables chat input
11. User sends follow-up via WebSocket → backend POSTs /message to relay → new turn begins

Concurrency

• Port allocation: PostgreSQL pg_try_advisory_xact_lock() prevents race conditions across concurrent requests
• Isolation: Each session gets its own Docker container with a dedicated noVNC port (range 5900-5999)
• Non-blocking: All Docker SDK calls wrapped in asyncio.to_thread() — container creation doesn't freeze the event loop
• Ordered events: Relay uses an ordered asyncio.Queue + sequential worker with retries — events arrive in exact emission order

Shutdown & Cleanup

• Startup: cleanup_zombie_containers() finds orphaned containers from previous crashes
• Shutdown: Backend lifespan hook calls stop_all_containers() with 60s grace period (stop_grace_period: 60s in docker-compose)
• Safety net: ./docker/down.sh stops agent containers externally before running docker compose down

API Reference

Method	Endpoint	Description
GET	/health	Health check
POST	/sessions	Create session ({"task": "..."}) → 201
GET	/sessions	List all sessions
GET	/sessions/{id}	Get session details (status, novnc_url)
POST	/sessions/{id}/stop	Stop container, preserve history
DELETE	/sessions/{id}	Stop container and delete session → 204
GET	/sessions/{id}/messages	Get persisted message history
POST	/sessions/{id}/messages	Send follow-up message via REST (alternative to WebSocket) → 202
POST	/sessions/{id}/events	Relay webhook receiver (internal — not for clients)
WS	/sessions/{id}/stream	Bidirectional: events down, user messages up

WebSocket Protocol

Server → Client (downstream events):

{"type": "history_snapshot", "messages": [...]}
{"type": "text_delta", "content": "Agent thinking..."}
{"type": "tool_use", "name": "computer", "input": {"action": "screenshot"}}
{"type": "screenshot", "image_base64": "iVBOR..."}
{"type": "error", "message": "API error: ..."}
{"type": "done", "reason": "completed"}

Client → Server (upstream messages):

{"type": "user_message", "content": "Now search for New York weather"}

Project Structure

.
├── backend/
│   ├── app/
│   │   ├── main.py                  # FastAPI app, CORS, lifespan (zombie cleanup + shutdown)
│   │   ├── config.py                # Pydantic settings (env-based config)
│   │   ├── database.py              # Async SQLAlchemy engine, session factory, get_session
│   │   ├── routers/
│   │   │   ├── sessions.py          # CRUD: create, list, get, stop, delete
│   │   │   ├── messages.py          # GET message history + POST follow-up (REST)
│   │   │   ├── webhook.py           # POST /events — relay webhook receiver → persist + EventBus
│   │   │   └── stream.py            # WebSocket: history_snapshot, EventBus forward, user_message
│   │   ├── services/
│   │   │   ├── session_manager.py   # 7-step session orchestration, stop, delete, zombie cleanup
│   │   │   ├── stream_manager.py    # Relay HTTP calls: POST /task, POST /message
│   │   │   ├── container.py         # Docker SDK wrapper (create, stop, start_relay, list)
│   │   │   ├── port_allocator.py    # Atomic port allocation with PostgreSQL advisory locks
│   │   │   ├── health_checker.py    # Poll noVNC + relay /health until ready
│   │   │   └── event_bus.py         # In-memory asyncio.Queue pub/sub (module singleton)
│   │   ├── models/
│   │   │   ├── session.py           # Session model (id, status, task, container_id, novnc_port)
│   │   │   └── message.py           # Message model (id, session_id, role, content JSONB)
│   │   ├── schemas/
│   │   │   ├── session.py           # SessionCreate, SessionResponse (computed novnc_url)
│   │   │   ├── message.py           # MessageCreate, MessageResponse
│   │   │   └── events.py            # Event type definitions
│   │   └── repositories/
│   │       ├── session.py           # Session DB queries (get, list, list_active, delete)
│   │       └── messages.py          # append_message, get_messages_for_session
│   ├── alembic/                     # Database migrations
│   ├── tests/                       # Unit + integration tests (pytest)
│   └── Dockerfile
├── frontend/
│   ├── app/
│   │   ├── layout.tsx               # Root layout with navigation header
│   │   ├── page.tsx                 # Home → redirects to /sessions
│   │   ├── sessions/page.tsx        # Session list with auto-refresh (SWR)
│   │   ├── sessions/new/page.tsx    # Create session form
│   │   └── sessions/[id]/page.tsx   # Split view: chat + VNC, multi-turn state
│   ├── components/
│   │   ├── EventStream.tsx          # Live event renderer (text, tools, screenshots, chat input)
│   │   ├── SessionCard.tsx          # Session list card with status badge
│   │   └── VNCIframe.tsx            # Embedded noVNC viewer (resize=scale, auto-fit)
│   ├── hooks/
│   │   ├── useSessions.ts           # SWR hooks + API helpers (create, stop, delete)
│   │   └── useWebSocket.ts          # Auto-reconnecting WebSocket with exponential backoff
│   ├── lib/api.ts                   # HTTP client (apiGet, apiPost, apiDelete)
│   └── Dockerfile                   # Multi-stage production build
├── docker/
│   ├── agent/
│   │   ├── Dockerfile               # Builds energent-agent:latest (Anthropic base + relay)
│   │   └── relay.py                 # Webhook relay: ordered queue, POST /task, POST /message
│   ├── docker-compose.yml           # postgres + backend + frontend + agent-image builder
│   ├── down.sh                      # Graceful shutdown (stops agent containers first)
│   └── .env.example                 # Template for ANTHROPIC_API_KEY + DB config
├── docs/diagrams/
│   ├── architecture.mmd             # Mermaid source — system architecture
│   ├── architecture.png             # Rendered architecture diagram
│   ├── sequence.mmd                 # Mermaid source — session lifecycle + streaming
│   └── sequence.png                 # Rendered sequence diagram
└── scripts/
    └── verify-stack.sh              # Automated health & concurrency test

Environment Variables

Variable	Required	Default	Description
ANTHROPIC_API_KEY	Yes	-	Anthropic API key
POSTGRES_USER	No	sessionmanager	PostgreSQL username
POSTGRES_PASSWORD	No	sessionmanager	PostgreSQL password
POSTGRES_DB	No	sessions	PostgreSQL database name
NEXT_PUBLIC_API_URL	No	http://localhost:8000	Backend URL for frontend

Local Development

Backend:

cd backend && uv sync && uv run uvicorn app.main:app --reload

Frontend:

cd frontend && npm install && npm run dev

PostgreSQL (standalone):

docker run -d --name postgres -p 5432:5432 \
  -e POSTGRES_USER=sessionmanager \
  -e POSTGRES_PASSWORD=sessionmanager \
  -e POSTGRES_DB=sessions \
  postgres:16-alpine

Troubleshooting

Issue	Fix
Port already in use	lsof -i :8000 to find the process
Container fails to start	docker compose -f docker/docker-compose.yml logs backend
Database errors	docker compose -f docker/docker-compose.yml down -v then restart
Agent returns "done" immediately	Check API key has credits at console.anthropic.com
Docker socket permission denied	Ensure /var/run/docker.sock is accessible
Network "energent_agent_net" still in use	Agent containers are still running — use ./docker/down.sh instead of docker compose down
[Errno -2] Name or service not known in logs	Session container was removed but frontend still reconnecting — session will auto-stop

License

MIT