From 7023639e2616bc895067da4fe9d12d44a347e3ce Mon Sep 17 00:00:00 2001 From: dimeloper Date: Tue, 5 Aug 2025 23:31:08 +0000 Subject: [PATCH] feat: improve local setup docs. --- DEVELOPMENT.md | 71 +++++++-- LOCAL_SETUP.md | 388 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 448 insertions(+), 11 deletions(-) create mode 100644 LOCAL_SETUP.md diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index baa47e2..f547f74 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -2,6 +2,12 @@ This guide covers setting up the Watchflow development environment for local development and testing. +## Quick Start + +🚀 **New to Watchflow?** Start with our [Local Development Setup Guide](./LOCAL_SETUP.md) for a complete end-to-end setup including GitHub App configuration and webhook testing. + +This document covers advanced development topics and workflow. + ## Prerequisites - Python 3.12 or higher @@ -22,6 +28,7 @@ cd watchflow ### 2. Create Virtual Environment Using uv (recommended): + ```bash # Create and activate virtual environment uv venv @@ -34,6 +41,7 @@ uv sync ``` Or using pip: + ```bash # Create virtual environment python -m venv .venv @@ -118,8 +126,10 @@ Create a GitHub App for development: - Issue comment - Pull request - Push + - Status 6. **Generate private key** and encode it: + ```bash cat /path/to/private-key.pem | base64 | tr -d '\n' ``` @@ -145,6 +155,7 @@ For debugging AI agents: 1. Create a [LangSmith](https://langsmith.com/) account 2. Get your API key 3. Add to environment: + ```bash LANGCHAIN_TRACING_V2=true LANGCHAIN_API_KEY=your-langsmith-api-key @@ -193,6 +204,41 @@ uv run ruff format src/ uv run mypy src/ ``` +### Pre-commit Hooks + +This project uses pre-commit hooks to ensure code quality and consistency. The hooks automatically run on every commit and include: + +- **Trailing whitespace removal** - Cleans up extra whitespace +- **End of file fixer** - Ensures files end with newlines +- **YAML/JSON validation** - Checks syntax +- **Ruff formatting and linting** - Formats code and sorts imports +- **Conventional commit validation** - Ensures commit messages follow conventional format + +#### Setup Pre-commit Hooks + +```bash +# Install pre-commit hooks (run once after cloning) +uv run pre-commit install + +# Also install commit message validation +uv run pre-commit install --hook-type commit-msg +``` + +#### Using Pre-commit Hooks + +```bash +# Hooks run automatically on commit, but you can run them manually: +uv run pre-commit run --all-files + +# Run on specific files +uv run pre-commit run --files src/main.py + +# Skip hooks for a commit (not recommended) +git commit --no-verify -m "commit message" +``` + +The hooks will prevent commits if any issues are found. Most formatting issues are automatically fixed, so you just need to stage the changes and commit again. + ### Testing The project includes comprehensive tests that run **without making real API calls** by default: @@ -212,7 +258,7 @@ pytest tests/integration/ ### Test Structure -``` +```txt tests/ ├── unit/ # ⚡ Fast unit tests (mocked OpenAI) │ └── test_feasibility_agent.py @@ -235,16 +281,6 @@ pytest tests/integration/ -m integration _Note: Real API tests make actual OpenAI calls and will cost money. They're disabled by default in CI/CD._ -### Pre-commit Hooks - -```bash -# Install pre-commit hooks -uv run pre-commit install - -# Run manually -uv run pre-commit run --all-files -``` - ## Testing AI Agents ### Rule Evaluation Testing @@ -282,6 +318,15 @@ rules: event_types: [pull_request] parameters: test_param: "test_value" + + - id: status-check-required + name: Status Check Required + description: All PRs must pass required status checks + enabled: true + severity: high + event_types: [pull_request] + parameters: + required_checks: ["ci/test", "lint"] ``` ## Debugging @@ -319,6 +364,7 @@ Test webhook delivery: ### GitHub App Permissions If webhooks aren't being received: + 1. Verify webhook URL is accessible 2. Check GitHub App permissions 3. Verify webhook secret matches @@ -326,6 +372,7 @@ If webhooks aren't being received: ### AI Agent Issues If agents aren't working: + 1. Verify OpenAI API key 2. Check LangSmith configuration 3. Review agent logs for errors @@ -333,6 +380,7 @@ If agents aren't working: ### Development Environment If dependencies aren't working: + 1. Ensure Python 3.12+ 2. Try recreating virtual environment 3. Check uv/pip installation @@ -352,6 +400,7 @@ locust -f load_test.py --host=http://localhost:8000 ### Agent Performance Monitor agent performance with LangSmith: + - Token usage per request - Response times - Error rates diff --git a/LOCAL_SETUP.md b/LOCAL_SETUP.md new file mode 100644 index 0000000..627e879 --- /dev/null +++ b/LOCAL_SETUP.md @@ -0,0 +1,388 @@ +# Local Development Setup Guide + +This guide covers setting up Watchflow for complete end-to-end local development, including GitHub App configuration, webhook setup, and API integration. + +## Why This Setup? + +**Personal GitHub App vs. Marketplace App**: For local development, creating your own GitHub App instead of using the production Watchflow app from the marketplace provides several critical advantages: + +- **Isolated development environment**: Your local testing won't interfere with production Watchflow instances +- **Full control over webhooks**: You can point webhooks to your local ngrok tunnel instead of production servers +- **Custom configuration**: You can modify permissions and settings without affecting the production app +- **Safe experimentation**: Test new features, rule changes, and integrations without risk to live systems +- **Independent debugging**: Monitor webhook deliveries and debug issues in isolation +- **No rate limiting conflicts**: Avoid hitting GitHub API rate limits that might affect production usage + +**ngrok for Local Development**: Since GitHub webhooks need to reach your local development server, and your localhost isn't accessible from the internet, ngrok creates a secure tunnel that: + +- **Exposes your local server** to GitHub's webhook delivery system +- **Provides HTTPS endpoints** required by GitHub for webhook URLs +- **Offers request inspection** tools to debug webhook payloads +- **Eliminates complex firewall configuration** or port forwarding + +## Prerequisites + +- Python 3.12 or higher +- [uv](https://docs.astral.sh/uv/) package manager (recommended) or pip +- [ngrok](https://ngrok.com/) installed for webhook tunneling +- GitHub organization or user account with admin access +- [OpenAI API key](https://platform.openai.com/api-keys) for AI agent functionality +- [LangSmith account](https://langsmith.com/) for AI agent debugging (optional) + +## Step 1: Create a GitHub App + +1. Navigate to your GitHub organization settings or [personal settings](https://github.com/settings/apps) +2. Go to **"Developer settings"** → **"GitHub Apps"** → **"New GitHub App"** +3. Fill in the basic app information: + - **App name**: `watchflow-dev` (or your preferred name) + - **Homepage URL**: `http://localhost:8000` + - **Webhook URL**: `https://placeholder.ngrok.io/webhooks/github` (we'll update this in Step 3) + - **Webhook secret**: Generate a secure random string and save it + - **Description**: "Local development instance of Watchflow" + +## Step 2: Configure GitHub App Permissions + +### Repository Permissions + +Set the following permissions for your GitHub App: + +- **Actions**: Read-only +- **Checks**: Read and write +- **Contents**: Read-only +- **Deployments**: Read and write +- **Environments**: Read-only +- **Issues**: Read and write +- **Metadata**: Read-only (mandatory) +- **Pull requests**: Read and write +- **Commit statuses**: Read and write + +### Organization Permissions + +- **Members**: Read-only + +### Subscribe to Events + +Check the following webhook events: + +- ✅ **Check run** +- ✅ **Commit comment** +- ✅ **Deployment** +- ✅ **Deployment protection rule** +- ✅ **Deployment review** +- ✅ **Deployment status** +- ✅ **Issue comment** +- ✅ **Issues** +- ✅ **Pull request** +- ✅ **Pull request review** +- ✅ **Pull request review comment** +- ✅ **Pull request review thread** +- ✅ **Push** +- ✅ **Status** +- ✅ **Workflow dispatch** +- ✅ **Workflow job** +- ✅ **Workflow run** + +### Generate and Download Private Key + +1. After creating the app, scroll down to **"Private keys"** +2. Click **"Generate a private key"** +3. Download the `.pem` file and save it securely +4. Note down your **App ID** from the app settings page + +## Step 3: Clone and Setup Watchflow + +### 3.1 Clone the Repository + +```bash +git clone https://github.com/watchflow/watchflow.git +cd watchflow +``` + +### 3.2 Create Virtual Environment + +Using uv (recommended): + +```bash +# Create and activate virtual environment +uv venv +source .venv/bin/activate # On macOS/Linux +# or +.venv\Scripts\activate # On Windows + +# Install dependencies +uv sync +``` + +Or using pip: + +```bash +# Create virtual environment +python -m venv .venv + +# Activate virtual environment +source .venv/bin/activate # On macOS/Linux +# or +.venv\Scripts\activate # On Windows + +# Install dependencies +pip install -e ".[dev]" +``` + +### 3.3 Install and Setup ngrok + +```bash +# Install ngrok +# On macOS with Homebrew: +brew install ngrok + +# On other systems, download from https://ngrok.com/download + +# Start ngrok tunnel to your local API +ngrok http 8000 +``` + +Copy the ngrok HTTPS URL from the terminal output (e.g., `https://abc123.ngrok.io`) + +### 3.4 Update GitHub App Webhook URL + +1. Go back to your GitHub App settings +2. Update the **Webhook URL** to: `https://your-ngrok-url.ngrok.io/webhooks/github` +3. Save the changes + +## Step 4: Configure Environment Variables + +### 4.1 Create Environment File + +```bash +cp .env.example .env +``` + +### 4.2 Configure Required Variables + +Edit your `.env` file with the following configuration: + +```bash +# GitHub App Configuration +APP_NAME_GITHUB=watchflow-dev +CLIENT_ID_GITHUB=your_app_id_from_github_app_settings +CLIENT_SECRET_GITHUB=your_client_secret_from_github_app_settings +PRIVATE_KEY_BASE64_GITHUB=your_base64_encoded_private_key +REDIRECT_URI_GITHUB=http://localhost:3000 + +# GitHub Webhook Configuration +WEBHOOK_SECRET_GITHUB=your_webhook_secret_from_step_1 + +# OpenAI API Configuration +OPENAI_API_KEY=your-openai-api-key + +# LangChain Configuration (Optional - for AI debugging) +LANGCHAIN_TRACING_V2=true +LANGCHAIN_ENDPOINT=https://api.smith.langchain.com +LANGCHAIN_API_KEY=your-langsmith-api-key +LANGCHAIN_PROJECT=watchflow-dev + +# Application Configuration +ENVIRONMENT=development + +# CORS Configuration +CORS_HEADERS=["*"] +CORS_ORIGINS='["http://localhost:3000", "http://127.0.0.1:3000"]' + +# AWS Configuration (if using AWS services) +AWS_ACCESS_KEY_ID=your_aws_access_key_id +AWS_SECRET_ACCESS_KEY=your_aws_secret_access_key +``` + +### 4.3 Encode Your Private Key + +Convert your GitHub App private key to base64: + +**Option 1: Using command line (recommended):** + +```bash +# Encode the private key file +cat /path/to/your-private-key.pem | base64 | tr -d '\n' +``` + +**Option 2: Using online tools:** + +If you prefer using a web interface, you can use online base64 encoding tools like [base64encode.org](https://www.base64encode.org): + +1. Open the `.pem` file in a text editor +2. Copy the entire content (including the BEGIN/END lines) +3. Paste it into the online encoder +4. Copy the base64 output + +Copy the base64 output and use it as the value for `PRIVATE_KEY_BASE64_GITHUB` in your `.env` file. + +**Security Note**: When using online tools, ensure you're using a reputable service and understand that your private key content will be processed by their servers. For maximum security, prefer the command-line method. + +## Step 5: Install GitHub App on Repositories + +### 5.1 Install the App + +1. In your GitHub App settings, click on **"Install App"** in the left sidebar +2. Choose to install on your **organization** or **personal account** +3. Select **"Selected repositories"** and choose the repositories you want to monitor +4. Alternatively, select **"All repositories"** for organization-wide installation +5. Click **"Install"** to complete the installation + +### 5.2 Verify Installation + +1. Go to the repository settings of an installed repository +2. Navigate to **"Integrations"** → **"GitHub Apps"** +3. Verify that your `watchflow-dev` app is listed and active + +## Step 6: Start Local Development Environment + +### 6.1 Start the Watchflow API + +```bash +# Using uv (recommended) +uv run uvicorn src.main:app --reload --host 0.0.0.0 --port 8000 + +# Using pip +uvicorn src.main:app --reload --host 0.0.0.0 --port 8000 +``` + +The API will be available at `http://localhost:8000` + +### 6.2 Verify API is Running + +Open your browser and navigate to: + +- `http://localhost:8000/docs` - Interactive API documentation +- `http://localhost:8000/health` - Health check endpoint + +## Step 7: Test the Setup + +### 7.1 Verify Webhook Connection + +1. Ensure your local API is running on port 8000 +2. Verify ngrok tunnel is active and forwarding to localhost:8000 +3. Check the ngrok dashboard at `http://localhost:4040` for incoming requests + +### 7.2 Test with Real Events + +1. Go to one of your monitored repositories +2. Create a new pull request or push a commit +3. Check your local API logs to confirm webhook events are being received +4. Monitor the ngrok dashboard for incoming webhook requests + +### 7.3 Test Rule Evaluation + +Create a test rule in a monitored repository by adding `.watchflow/rules.yaml`: + +```yaml +rules: + - id: test-rule + name: Test Rule for Local Development + description: Simple rule to test local setup + enabled: true + severity: medium + event_types: [pull_request] + parameters: + test_param: "local_test" + + - id: pr-approval-required + name: PR Approval Required + description: All pull requests must have at least 1 approval + enabled: true + severity: high + event_types: [pull_request] + parameters: + min_approvals: 1 +``` + +## Troubleshooting + +### Common Issues + +#### Webhook not receiving events + +- Verify ngrok is running and the URL is correct in GitHub App settings +- Check that the webhook secret matches your environment configuration +- Ensure your local API endpoint `/webhooks/github` is properly configured +- Check GitHub App webhook delivery logs in the app settings + +#### Permission errors + +- Double-check that all required permissions are granted to the GitHub App +- Verify the app is installed on the correct repositories/organization +- Ensure the private key is correctly encoded and configured + +#### ngrok tunnel expires + +- Free ngrok tunnels expire after 8 hours +- Restart ngrok and update the webhook URL in your GitHub App settings +- Consider upgrading to ngrok Pro for persistent URLs + +#### AI Agent not working + +- Verify your OpenAI API key is valid and has sufficient credits +- Check the AI model name in your configuration +- Review API logs for OpenAI-related errors + +### Logs and Debugging + +Monitor the following for debugging: + +1. **Local API server logs** - Check your terminal running uvicorn +2. **ngrok request logs** - Run `ngrok http 8000 --log=stdout` +3. **GitHub App webhook delivery logs** - Available in GitHub App settings +4. **LangSmith traces** - If configured, view at [LangSmith Dashboard](https://smith.langchain.com/) + +### Additional Debugging Commands + +```bash +# Check if API is responding +curl http://localhost:8000/health + +# Test rule evaluation endpoint +curl -X POST "http://localhost:8000/api/v1/rules/evaluate" \ + -H "Content-Type: application/json" \ + -d '{ + "rule_text": "All pull requests must have at least 2 approvals" + }' + +# View detailed logs +tail -f logs/watchflow.log +``` + +## Security Notes + +- **Never commit your private keys or webhook secrets to version control** +- Use environment variables or secure secret management for all credentials +- Rotate webhook secrets periodically +- Limit GitHub App installation scope to only necessary repositories during development +- Keep your ngrok tunnel URL private and don't share it publicly +- Use separate GitHub Apps for development and production environments + +## Next Steps + +Once your local setup is working: + +1. **Explore the API documentation** at `http://localhost:8000/docs` +2. **Create custom rules** in your test repositories +3. **Set up LangSmith** for AI agent debugging and monitoring +4. **Run the test suite** to verify everything is working: `pytest` +5. **Read the main development guide** in `DEVELOPMENT.md` for advanced topics + +## Development Workflow Integration + +After completing this setup, you can integrate with the standard development workflow: + +```bash +# Format and lint code +uv run ruff format src/ +uv run ruff check src/ + +# Run tests +pytest + +# Install pre-commit hooks for code quality +uv run pre-commit install +uv run pre-commit install --hook-type commit-msg +``` + +For more advanced development topics, testing strategies, and deployment options, refer to the main [DEVELOPMENT.md](./DEVELOPMENT.md) guide.