Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 192 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,197 @@
</a>
</p>

# covered
# Covered

Self-hosted coverage reports storage for GitHub repositories.
Self-hosted coverage report hosting for GitHub repositories — an alternative to Smokeshow, Codecov, and Coveralls for teams that prefer to keep their coverage data in-house and overcome limitations.

Docs are coming soon.
**Covered** has two parts:

- A **FastAPI backend** that stores HTML coverage reports, serves them over HTTP, and exposes an SVG badge.
- A **CLI** that runs in CI, uploads the report, posts a `covered` commit status to GitHub, and refreshes the badge cache.

## Architecture

Here is a [a bit simplified] diagram of how the components interact:

```mermaid
flowchart LR
S3[("AWS S3<br/>HTML reports")]

CI["Covered CLI"]
Backend["Covered backend"]
Redis[("[Optional] Redis cache")]
GH["GitHub API"]
Reader["Browser"]

CI -->|"Upload report files"| S3

CI -->|"Get S3 credentials for upload session"| Backend
CI -->|"Set commit status"| GH
Reader -->|"Get badge SVG<br />Redirect to report URL<br />Serve report files"| Backend
Backend -->|"Get latest status"| GH
Backend --> |"Get cached badge or report URL"| Redis
Backend -->|"Get report files"| S3
```

CLI requests temporary credentials from the backend to upload the report to S3, uploads the report, then sets a `covered` status on the commit.

When reader opens the page that contains badge, the badge is loaded from the backend, which looks up the latest `covered` status for the default branch commit, finds the corresponding coverage value, and serves the badge SVG. The badge links to the report URL, which is also served by the backend.

Optional Redis caching can be used to reduce latency and GitHub API calls for frequently accessed badges.

After uploading a report on the default branch, the CLI can also trigger a cache purge (if `-purge-cache` is specified) to ensure the badge reflects the new coverage value as soon as possible.

## Getting started

Setting up Covered takes two steps:

1. **Deploy the backend** — provision S3, Redis (optionally), and a GitHub token, then deploy to FastAPI Cloud. See [backend/README.md](backend/README.md).
2. **Wire up CI** — add the `covered` CLI to your test workflow. See the [GitHub Action setup](#github-action-setup) below; full CLI options are in [cli/README.md](cli/README.md).

## GitHub Action setup

In CI, coverage is uploaded by the [`covered` CLI](cli/README.md), which is published to PyPI. After your test job generates an HTML coverage report (typically `htmlcov/`), the CLI uploads it to your backend, posts a `covered` commit status on the commit, and refreshes the badge cache (if configured).

### Workflow setup

Before adding the workflows, make sure:

- **The backend is deployed and reachable** — see [backend/README.md](backend/README.md).
- **Your project uses `uv` with a committed `uv.lock`.** The workflows install dependencies via `uv sync --locked`.

Covered's CLI runs in a **separate workflow** that triggers after your test workflow finishes. This split is what makes coverage work for pull requests from forks: a fork's workflow can't read your `COVERED_API_KEY` secret and can't post commit statuses, but the second workflow runs in the base repository's context and can do both.

The first workflow runs your tests and uploads the HTML report as an artifact. The second downloads the artifact and invokes the `covered` CLI.

#### 1. Test workflow

```yaml
# .github/workflows/test.yml
name: Test

on:
push:
pull_request:

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: "0.11.7"
enable-cache: true
- run: uv sync --locked
- run: uv run coverage run -m pytest
- run: uv run coverage html # writes htmlcov/
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-html
path: htmlcov
```

#### 2. Coverage upload workflow

```yaml
# .github/workflows/coverage-upload.yml
name: Coverage upload

on:
workflow_run:
workflows: [Test]
types: [completed]

permissions:
contents: read
actions: read # download the artifact from the Test run
statuses: write # post the `covered` commit status

jobs:
upload:
runs-on: ubuntu-latest
if: github.event.workflow_run.conclusion == 'success'
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: coverage-html
path: htmlcov
github-token: ${{ secrets.GITHUB_TOKEN }}
run-id: ${{ github.event.workflow_run.id }}
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
version: "0.11.7"
enable-cache: true
- run: uv sync --locked
- name: Upload coverage to covered
run: uv run covered htmlcov
env:
COVERED_API_URL: ${{ secrets.COVERED_API_URL }}
COVERED_API_KEY: ${{ secrets.COVERED_API_KEY }}
COVERED_GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
COVERED_REPO_OWNER: ${{ github.repository_owner }}
COVERED_REPO_NAME: ${{ github.event.repository.name }}
COVERED_COMMIT_SHA: ${{ github.event.workflow_run.head_sha }}
COVERED_COVERAGE_THRESHOLD: 99
COVERED_PURGE_CACHE: ${{ github.event.workflow_run.head_branch == github.event.repository.default_branch }}
```

#### 3. Add dev dependencies

The workflows above expect `pytest`, `coverage`, and `covered` in the synced environment. Add them as dev dependencies:

```bash
uv add --dev pytest coverage covered
```

#### 4. Configure repository secrets

In your GitHub repository settings, add:

| Secret | Value |
|---|---|
| `COVERED_API_URL` | Base URL of your deployed backend, e.g. `https://covered.example.com`. |
| `COVERED_API_KEY` | The `API_KEY` value you configured on the backend. |

`COVERED_GH_TOKEN` does **not** need to be a custom PAT — the workflow-provided `${{ secrets.GITHUB_TOKEN }}` is sufficient, as long as the job has `statuses: write` permission.

---

A few non-obvious points worth knowing:

- `COVERED_COMMIT_SHA` must be `github.event.workflow_run.head_sha`, **not** `github.sha`. When a workflow is triggered by `workflow_run`, `github.sha` points at the default branch, not the commit that was actually tested.
- The same applies to `COVERED_PURGE_CACHE`: use `workflow_run.head_branch` to detect the default branch.
- `actions: read` is needed because the artifact lives on a different workflow run.

For a working pair of workflows, see this repo's [test.yml](.github/workflows/test.yml) and [coverage-upload.yml](.github/workflows/coverage-upload.yml). The full list of CLI options is in [cli/README.md](cli/README.md).

### Adding the badge to your README

Once the upload has run at least once on your default branch, add the badge to your repository's README:

```markdown
[![Coverage](https://covered.example.com/badge/OWNER/REPO.svg)](https://covered.example.com/badge/redirect/OWNER/REPO/)
```

Replace `covered.example.com` with your backend URL and `OWNER`/`REPO` with the repository's slug. The image points to the SVG badge endpoint. The link redirects to the latest stored report.

### Troubleshooting

| Symptom | Likely cause |
|---|---|
| Upload fails with 403 | `COVERED_API_KEY` doesn't match the backend's `API_KEY` setting. |
| Upload succeeds but no `covered` status on the commit | The job is missing `statuses: write`, or `COVERED_GH_TOKEN` cannot post statuses on the repository. |
| Coverage parsing is skipped | The directory passed to `covered` doesn't contain `index.html` — make sure `coverage html` ran successfully. |
| Badge shows `??` | No `covered` commit status has been posted on the default branch yet — re-run the workflow on `master`/`main`. |
| Badge doesn't refresh after a push to default branch | `COVERED_PURGE_CACHE` wasn't `true` for that run, or GitHub's image proxy (Camo) hasn't yet expired its cache (~5 minutes). |

## License

MIT. See [LICENSE](LICENSE).
66 changes: 66 additions & 0 deletions backend/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
<p align="center">
<img src="../docs/img/covered.png" alt="covered" width="200"><br />
<b>Make it green.</b>
</p>

# Covered backend

FastAPI app that stores HTML coverage reports, serves them over HTTP, and exposes an SVG badge endpoint reflecting each repository's latest coverage on its default branch.

This document is the operator manual — provisioning the external services, configuring the backend, deploying to [FastAPI Cloud](https://fastapicloud.com/), and verifying the install. For the project overview and the CI side of the picture, see the [top-level README](../README.md).

## Prerequisites

Before deploying the backend, you will need:

- **AWS infrastructure** — an S3 bucket, an IAM user, and an IAM role. See [aws.md](aws.md) for step-by-step setup with the exact policy JSON. The short version of what you'll end up with:
- An **S3 bucket** for HTML reports. The backend writes objects under the `sites/<site_id>/...` prefix and reads them back when serving reports. No public access or static website hosting — the backend serves files itself.
- An **IAM user** for the backend, with a long-lived access key. It needs `s3:PutObject` + `s3:GetObject` on `<bucket>/sites/*`, and `sts:AssumeRole` on the upload role.
- An **IAM role for uploads**, with a permissions policy granting `s3:PutObject` on `<bucket>/sites/*` and a trust policy allowing the IAM user to assume it. The backend assumes this role via STS to mint short-lived credentials for the CLI, further narrowed per upload by a session policy scoped to a single `site_id` — so the CLI never receives credentials that can write outside its own report directory.

- **A Redis instance** (optional) reachable from the backend. It is used as a short-lived cache for rendered badge SVGs.

- **A GitHub token** with read access to commit statuses on every repository whose coverage you want to display. A fine-grained PAT with `Commit statuses: Read-only` is sufficient; a classic PAT with `repo` (or `public_repo` for public repositories only) also works.

## Environment variables

The backend is configured via environment variables, loaded by `app.config.Settings` (pydantic-settings).

| Variable | Required | Default | Description |
|---|---|---|---|
| `API_KEY` | yes | — | Token the CLI uses to authenticate with `/coverage/create-site/` and `/coverage/invalidate-cache/*`. Treat as a secret. |
| `AWS_REGION` | no | `us-east-1` | Region of the S3 bucket. |
| `AWS_BUCKET` | no | `covered` | Name of the S3 bucket reports are stored in. |
| `AWS_ACCESS_KEY_ID` | yes | — | Access key of the backend's IAM user (the one with `s3:GetObject`/`PutObject` on `<bucket>/sites/*` and `sts:AssumeRole` on the upload role). |
| `AWS_SECRET_ACCESS_KEY` | yes | — | Secret access key for the above IAM user. |
| `AWS_UPLOAD_ROLE_ARN` | yes | — | ARN of the IAM role the backend assumes via STS to mint short-lived upload credentials for the CLI. |
| `REDIS_URL` | no | — | Connection URL for Redis. Used to cache rendered badge SVGs (60 s TTL). |
| `GITHUB_TOKEN` | yes | — | GitHub token used to read commit statuses. See [Prerequisites](#prerequisites) for the required scopes. |

## Deploying to FastAPI Cloud

Covered is designed to run on [FastAPI Cloud](https://fastapicloud.com/). The deploy flow:

1. **Sign up at [fastapicloud.com](https://fastapicloud.com/).** Access is waitlist-only at the moment — getting in typically takes a couple of days.

2. **Create an app** in the FastAPI Cloud dashboard.

3. **Configure the environment variables** on the app (the [table above](#environment-variables) lists all of them).

4. **Clone (or fork) this repository.** Fork if you plan to customize the backend.

5. **Install dependencies and deploy.** From the repo root:

```bash
uv sync --project backend
cd backend
fastapi deploy
```

6. **Authorize the CLI.** On the first deploy, the CLI opens your browser to authorize itself against your FastAPI Cloud account — grant the requested permissions.

7. **Pick your team and app** when prompted. Typically your personal team and the app you created in step 2; leave the rest at their defaults.

8. **Confirm and wait** for the deploy to report success.

9. **Verify** by triggering the [`Coverage upload` workflow](../README.md#workflow-setup) in a repository configured to use this backend. If anything fails, the FastAPI Cloud dashboard logs are the first place to look.
Loading