Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 0 additions & 9 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ version: 2.1

orbs:
node: circleci/node@7.1.1
docker: circleci/docker@1.5.0
codecov: codecov/codecov@5.0.3
gcp-cli: circleci/gcp-cli@3.3.2

Expand Down Expand Up @@ -140,8 +139,6 @@ jobs:
image: default
steps:
- checkout
- docker/install-docker-compose:
version: 1.29.2
- run:
name: Run tests and coverage within Docker container
command: |
Expand All @@ -155,8 +152,6 @@ jobs:
image: default
steps:
- checkout
- docker/install-docker-compose:
version: 1.29.2
- run:
name: Run tests and coverage within Docker container
command: |
Expand All @@ -170,8 +165,6 @@ jobs:
image: default
steps:
- checkout
- docker/install-docker-compose:
version: 1.29.2
- run:
name: Run tests and coverage within Docker container
command: |
Expand All @@ -185,8 +178,6 @@ jobs:
image: default
steps:
- checkout
- docker/install-docker-compose:
version: 1.29.2
- run:
name: Run tests and coverage within Docker container
command: |
Expand Down
6 changes: 3 additions & 3 deletions docs/backend_tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@
For more control, run each tool individually by first running:

```bash
docker-compose run backend bash
docker compose run backend bash
```

...which saves having to wait for docker-compose to spin up for every test run.
...which saves having to wait for `docker compose` to spin up for every test run.

`pnpm build` will generate a `.build` directory which will be seen within the `backend` container.
If you don't have `pnpm` working on your host you can run this instead `docker-compose run frontend sh -c "corepack enable && pnpm install && pnpm build"`
If you don't have `pnpm` working on your host you can run this instead `docker compose run frontend sh -c "corepack enable && pnpm install && pnpm build"`

Then run the individual tools within that shell, like so:

Expand Down
2 changes: 1 addition & 1 deletion docs/common_tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The docs will then be available at: <http://localhost:8000>
## Debugging Tools

You can use the Python Debugger ([pdb](https://docs.python.org/3.10/library/pdb.html)) in a Docker container.
After starting a local Treeherder instance using [docker-compose](installation.md#server-and-full-stack-development),
After starting a local Treeherder instance using [docker compose](installation.md#server-and-full-stack-development),
in a separate shell type `docker attach backend`. Then set a breakpoint in your file using either `import pdb; pdb.set_trace()`
or `breakpoint()`. The pdb debugger will start in that shell once the breakpoint has been triggered.
For example, it can be triggered via refreshing the browser (localhost) if the view you're on calls an API with a breakpoint on it.
2 changes: 1 addition & 1 deletion docs/frontend-backend-prod-db.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ th-db

```code
use-prod
docker-compose up -d
docker compose up -d
```

### ENV vars
Expand Down
32 changes: 16 additions & 16 deletions docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ production site. You do not need to set up the Docker environment unless making

To get started:

- Install Docker & docker-compose (both are installed if using Docker for Windows/Mac).
- Install [Docker] (Docker Compose is included with Docker Desktop for Windows/Mac; on Linux, install the Docker Compose plugin).

- If you just wish to [run the tests](backend_tasks.md#running-the-tests),
you can stop now without performing the remaining steps.
Expand All @@ -68,7 +68,7 @@ export those env variables in the shell first or inline with the command below.
- Open a shell, cd into the root of the Treeherder repository, and type:

```bash
docker-compose up --build
docker compose up --build
```

- Wait for the Docker images to be downloaded/built and container steps to complete.
Expand All @@ -85,13 +85,13 @@ If you would like to use the minified production version of the UI with the deve
- Run the build task:

```bash
docker-compose run frontend sh -c "corepack enable && pnpm install && pnpm build"
docker compose run frontend sh -c "corepack enable && pnpm install && pnpm build"
```

- Start Treeherder's backend:

```bash
docker-compose up --build
docker compose up --build
```

- Visit <http://localhost:8000> (NB: port 8000, unlike above)
Expand All @@ -115,10 +115,10 @@ DATABASE_URL=psql://user:password@hostname/treeherder
```

Alternatively, you can `export` that value in your terminal prior to executing
`docker-compose up` or just specify it on the command line as you execute:
`docker compose up` or just specify it on the command line as you execute:

```bash
DATABASE_URL=psql://user:password@hostname/treeherder SKIP_INGESTION=True docker-compose up
DATABASE_URL=psql://user:password@hostname/treeherder SKIP_INGESTION=True docker compose up
```

<!-- prettier-ignore -->
Expand All @@ -134,7 +134,7 @@ The Postgres database is kept locally and is not destroyed when the Docker conta
If you want to start from scratch type the following commands:

```bash
docker-compose down
docker compose down
docker volume rm treeherder_postgres_data
```

Expand All @@ -145,7 +145,7 @@ Celery tasks include storing of pushes, tasks and parsing logs (which provides f
Open a new shell tab. To run all the queues type:

```bash
docker-compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker --concurrency 1
docker compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker --concurrency 1
```

<!-- prettier-ignore -->
Expand All @@ -156,14 +156,14 @@ You can find a list of different celery queues in the the CELERY_TASK_QUEUES var
only store tasks and pushes and omit log parsing:

```bash
docker-compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker -Q store_pulse_tasks,store_pulse_pushes --concurrency 1
docker compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker -Q store_pulse_tasks,store_pulse_pushes --concurrency 1
```

### Manual ingestion

`NOTE`: You have to include `--root-url https://community-tc.services.mozilla.com` in order to ingest from the [Taskcluster Community instance](https://community-tc.services.mozilla.com), otherwise, it will default to the Firefox CI.

Open a terminal window and run `docker-compose up`. All following sections assume this step.
Open a terminal window and run `docker compose up`. All following sections assume this step.

#### Ingesting pushes

Expand All @@ -175,15 +175,15 @@ changesets from the web interface into subsequent commands to ingest all tasks.
Ingest a single Mercurial push or the last N pushes:

```console
docker-compose exec backend ./manage.py ingest push -p autoland -r 63f8a47cfdf5
docker-compose exec backend ./manage.py ingest push -p mozilla-central --last-n-pushes 100
docker compose exec backend ./manage.py ingest push -p autoland -r 63f8a47cfdf5
docker compose exec backend ./manage.py ingest push -p mozilla-central --last-n-pushes 100
```

Ingest a single Github push or the last 10:

```console
docker-compose exec backend ./manage.py ingest git-push -p servo-try -c 92fc94588f3b6987082923c0003012fd696b1a2d
docker-compose exec -e GITHUB_TOKEN=<foo> backend ./manage.py ingest git-pushes -p android-components
docker compose exec backend ./manage.py ingest git-push -p servo-try -c 92fc94588f3b6987082923c0003012fd696b1a2d
docker compose exec -e GITHUB_TOKEN=<foo> backend ./manage.py ingest git-pushes -p android-components
```

`NOTE`: You can ingest all tasks for a push. Check the help output for the script to determine the
Expand All @@ -198,7 +198,7 @@ to grant scopes for it.
`NOTE`: This will only ingest the commits if there's an active Github PRs project. It will only ingest the commits.

```bash
docker-compose exec backend ./manage.py ingest pr --pr-url https://github.com/mozilla-mobile/android-components/pull/4821
docker compose exec backend ./manage.py ingest pr --pr-url https://github.com/mozilla-mobile/android-components/pull/4821
```

#### Ingesting individual task
Expand All @@ -207,7 +207,7 @@ This will work if the push associated to the task exists in the database.

```bash
# Make sure to ingest 1bd9d4f431c4c9f93388bd04a6368cb07398f646 for autoland first
docker-compose exec backend ./manage.py ingest task --task-id KQ5h1BVYTBy_XT21wFpLog
docker compose exec backend ./manage.py ingest task --task-id KQ5h1BVYTBy_XT21wFpLog
```

## Learn more
Expand Down
2 changes: 1 addition & 1 deletion docs/pulseload.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Pulse Ingestion Configuration

By default, running the Docker container with `docker-compose up` will ingest data
By default, running the Docker container with `docker compose up` will ingest data
from the `autoland` and `try` repositories using a shared [Pulse Guardian] user. You can configure this the following ways:

1. Specify a custom set of repositories for which to ingest data
Expand Down
6 changes: 3 additions & 3 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,19 +77,19 @@ of specificity:
All tests:

```shell
docker-compose run backend pytest tests/
docker compose run backend pytest tests/
```

Just `/etl` tests

```shell
docker-compose run backend pytest tests/etl/
docker compose run backend pytest tests/etl/
```

Just the `test_ingest_pending_pulse_job` within the `/etl` tests

```shell
docker-compose run backend pytest tests/ -k test_ingest_pending_pulse_job
docker compose run backend pytest tests/ -k test_ingest_pending_pulse_job
```

### Updating backend python test data
Expand Down
22 changes: 11 additions & 11 deletions tox.ini
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ toxworkdir={toxinidir}/.tox
[testenv]
allowlist_externals =
sh
docker-compose
docker
commands_pre =
docker-compose up --detach postgres redis rabbitmq
docker compose up --detach postgres redis rabbitmq
pip install --no-deps -r {toxinidir}/requirements/dev.txt
pip install --no-deps -r {toxinidir}/requirements/common.txt
commands =
Expand All @@ -20,7 +20,7 @@ commands =
pytest tests/ --runslow
commands_post =
# This is to deal with running the containers with --detached
docker-compose down
docker compose down

[testenv:linters]
commands_pre =
Expand All @@ -44,30 +44,30 @@ commands_post =
[testenv:docker-general]
commands_pre =
allowlist_externals=
docker-compose
docker
commands =
docker-compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m 'not (perf or frontend)' --runslow -p no:unraisableexception"
docker compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m 'not (perf or frontend)' --runslow -p no:unraisableexception"

[testenv:docker-perf]
commands_pre =
allowlist_externals=
docker-compose
docker
commands =
docker-compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m 'perf and not telemetry_alerting' --runslow -p no:unraisableexception"
docker compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m 'perf and not telemetry_alerting' --runslow -p no:unraisableexception"

[testenv:docker-telemetry]
commands_pre =
allowlist_externals=
docker-compose
docker
commands =
docker-compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m telemetry_alerting --runslow -p no:unraisableexception"
docker compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m telemetry_alerting --runslow -p no:unraisableexception"

[testenv:docker-frontend]
commands_pre =
allowlist_externals=
docker-compose
docker
commands =
docker-compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m frontend --runslow -p no:unraisableexception"
docker compose run -e TREEHERDER_DEBUG=False backend bash -c "pytest --cov --cov-report=xml -m frontend --runslow -p no:unraisableexception"

[flake8]
per-file-ignores = treeherder/model/models.py:E402