diff --git a/docs/installation.md b/docs/installation.md index be35fe4cfc1..ca761e9266e 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -161,41 +161,108 @@ docker compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder w ### Manual ingestion -`NOTE`: You have to include `--root-url https://community-tc.services.mozilla.com` in order to ingest from the [Taskcluster Community instance](https://community-tc.services.mozilla.com), otherwise, it will default to the Firefox CI. +!!! note + You have to include `--root-url https://community-tc.services.mozilla.com` in order to ingest from the [Taskcluster Community instance](https://community-tc.services.mozilla.com), otherwise, it will default to the Firefox CI. Open a terminal window and run `docker compose up`. All following sections assume this step. #### Ingesting pushes -`NOTE`: Only the push information will be ingested. Tasks -associated with the pushes will not. This mode is useful to seed pushes so -they are visible on the web interface and so you can easily copy and paste -changesets from the web interface into subsequent commands to ingest all tasks. +!!! note + Only the push information will be ingested. Tasks + associated with the pushes will not. This mode is useful to seed pushes so + they are visible on the web interface and so you can easily copy and paste + changesets from the web interface into subsequent commands to ingest all tasks. -Ingest a single Mercurial push or the last N pushes: +These steps should help you set up everything you need to ingest data locally while working on perfherder. -```console -docker compose exec backend ./manage.py ingest push -p autoland -r 63f8a47cfdf5 -docker compose exec backend ./manage.py ingest push -p mozilla-central --last-n-pushes 100 +#### Create a pulse guardian account and run + +!!! note + This step is optional and used if you want to ingest from pulse messages. + +```bash +export PULSE_URL=amqp://USER:PASSWORD@pulse.mozilla.org:5671/?ssl=1 +pnpm install ``` -Ingest a single Github push or the last 10: +#### Run each of these commands in a seperate window -```console +```bash +docker compose up --build +docker compose run -e PROJECTS_TO_INGEST=autoland backend celery -A treeherder worker --concurrency 1 +``` + +--- + +#### Run the db viewer + +You can use any database viewer of your choice `(e.g. dbeaver-ce, mysql workbench, etc.)` + +**Connect to the following while `docker compose up --build` from the previous step is running:** + +`Serverhost: localhost` +`Port: 5432` +`Database: treeherder` +`Username: postgres` +`Password: mozilla1234` + +--- + +#### Run the following in a separate window while running above to do ingestion + + +!!! note + These commands perform fetches for the data, they are run sequentially in the same window. The first command makes the second one run faster. + +#### Ingest push + +```bash +docker compose exec backend ./manage.py ingest push -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9 +``` + +#### Ingest all the tasks, run celery to trigger the log parsing, and performance data ingestion + +> **Warning:** This command can take a long time to ingest and parse everything. + +```bash +docker compose exec backend ./manage.py ingest push -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9 -a --enable-eager-celery +``` + +`--enable-eager-celery` triggers the log parsing which is required to capture the `PERFHERDER_DATA` output. + +#### For ingesting multiple pushes + +```bash +docker compose exec backend ./manage.py ingest push -p autoland --last-n-pushes 100 +``` + +#### Ingest a single task + +```bash +docker compose exec backend ./manage.py ingest task -p autoland -r 1ee42a54a431acdd6cbe43b49de0237fe67eddd9 --task-id --enable-eager-celery +``` + +#### Ingest a single Github push or the last 10 + +```bash docker compose exec backend ./manage.py ingest git-push -p servo-try -c 92fc94588f3b6987082923c0003012fd696b1a2d docker compose exec -e GITHUB_TOKEN= backend ./manage.py ingest git-pushes -p android-components ``` -`NOTE`: You can ingest all tasks for a push. Check the help output for the script to determine the -parameters needed. +!!! note + You can ingest all tasks for a push. Check the help output for the script to determine the + parameters needed. -`NOTE`: If you make too many calls to the Github API you will start getting 403 messages because of the rate limit. -To avoid this visit [your settings](https://github.com/settings/tokens) and set up `GITHUB_TOKEN`. You don't need -to grant scopes for it. +!!! note + If you make too many calls to the Github API you will start getting 403 messages because of the rate limit. + To avoid this visit [your settings](https://github.com/settings/tokens) and set up `GITHUB_TOKEN`. You don't need + to grant scopes for it. #### Ingesting Github PRs -`NOTE`: This will only ingest the commits if there's an active Github PRs project. It will only ingest the commits. +!!! note + This will only ingest the commits if there's an active Github PRs project. It will only ingest the commits. ```bash docker compose exec backend ./manage.py ingest pr --pr-url https://github.com/mozilla-mobile/android-components/pull/4821 diff --git a/docs/pulseload.md b/docs/pulseload.md index 4be15fe2d9b..9bae88b7b23 100644 --- a/docs/pulseload.md +++ b/docs/pulseload.md @@ -19,7 +19,7 @@ export PROJECTS_TO_INGEST=autoland,try Visit [Pulse Guardian], sign in, and create a **Pulse User**. It will ask you to set a username and password. Remember these as you'll use them in the next step. This is recommended, because using the default value **MAY** cause you to miss some data, -if it was already ingested by another user Unfortunately, **Pulse** doesn't support creating +if it was already ingested by another user. Unfortunately, **Pulse** doesn't support creating queues with a guest account. If your **Pulse User** was username: `foo` and password: `bar`, your Pulse URL