Skip to content

Home

Jump to bottom
chandrakalahanchinal edited this page Jun 17, 2026 · 10 revisions

QueuePilot — Tool Documentation

Repository: https://github.com/chandrakalahanchinal/queuepilot

Problem Statement

The Magento 2.4-develop PR queue typically contains 3–8 PRs waiting to merge at any given time. Each PR runs functional tests across three editions — CE, EE, and B2B — producing separate Allure reports and Jenkins builds. PRs are often re-triggered multiple times as fixes land.

Manually triaging this means:

  • Opening 3–8 PRs individually on GitHub
  • Clicking into each CE / EE / B2B check run
  • Opening each Allure report and scrolling through failures
  • Cross-referencing Jira to find whether a failure is a known flake or a real regression
  • Repeating this for every previous re-run whose results are buried in PR comments
  • Doing all of this every time the queue changes — often multiple times a day

This takes 20–40 minutes per triage cycle, is error-prone, and produces inconsistent results across team members.

Solution

QueuePilot automates the entire triage workflow end-to-end. One Slack message triggers everything:

  1. Detects @qmbot dq 2.4-develop in #pr-queue-dashboard
  2. Waits for qmbot to reply with the PR list, then waits 5 minutes for test data to settle
  3. Fetches GitHub check runs for every PR in the queue
  4. Scrapes Allure failure data from the current run and all previous runs found in PR comments — no failure is missed across re-runs
  5. Looks up each unique failing test in Jira (ACQE project) to surface active tickets
  6. Posts a structured summary as a thread reply to qmbot's message — PR count, unique failing tests with occurrence counts and Jira links, per-PR CE/EE/B2B breakdown

The full triage that took 20–40 minutes now takes under 10 minutes of automated work and appears directly in the Slack thread where the team is already looking.

Table of Contents

How It Works

You               →  @qmbot dq 2.4-develop         (#pr-queue-dashboard)
qmbot             →  replies with PR list           (channel message)
Watcher (watch.py)→  detects reply, waits 5 min    (macOS LaunchAgent, always running)
QueuePilot        →  analyzes all PRs in parallel   (GitHub + Allure + Jira)
Slack Connect bot →  posts report in qmbot's thread (one reply, nothing at top level)

The 5-minute wait gives Jenkins and Allure time to finish writing test data before QueuePilot reads it.

Full Integration Flow

1. Slack → trigger detection (watch.py)

  • Polls #pr-queue-dashboard (C0B400Y1ZU2) every 30 seconds via conversations.history
  • When any user sends a message containing dq 2.4-develop, a 10-minute window opens
  • When qmbot (W015DAXESG0) replies within that window with GitHub PR URLs:
    • Records qmbot's message ts in .watcher_state.json as qmbot_reply_ts
    • Schedules the report to fire 5 minutes later
  • At fire time, calls queuepilot.py --reply-to-ts <qmbot_reply_ts>
  • After posting, clears all state and resets

2. GitHub → PR and check-run data

  • PR metadata (title, author, URL) via gh pr view
  • All check runs via gh api repos/{repo}/commits/{sha}/check-runs
  • Identifies CE, EE, B2B check runs by name prefix; reads conclusion and output.summary
  • No separate GitHub token needed — uses whatever gh auth is configured

3. Allure → test failure names (all runs)

When a check run has conclusion: failure, QueuePilot:

  1. Extracts Allure report URL from check-run summary (falls back to scraping Jenkins page)
  2. Fetches data/categories.json (fallback: suites.json → behaviors.json)
  3. Walks the tree collecting all leaf nodes with status failed or broken
  4. For each UID calls data/test-cases/{uid}.json to get the fully-qualified Java test method name
  5. Up to 10 test-case fetches in parallel per PR; up to 5 PRs in parallel

Retries up to --allure-attempts times (10-second pauses). Falls back to Prometheus stats if retries are exhausted.

Previous runs via PR body and comments: PRs are often re-triggered multiple times. Each run posts a bot comment containing Allure links; the PR description may also contain Allure URLs from earlier runs. QueuePilot scrapes both the PR body and all PR comments for Allure report URLs (get_pr_allure_urls), fetches failures from every previous run, then unions them with the current run — deduplicated by test method name. This ensures no failure is missed across re-runs.

Checks still running: When a PR's latest check is in_progress or queued, QueuePilot still fetches failures from all available previous-run Allure URLs and returns them, marking the edition with an ⏳ RUNNING · N prev fail badge so you can act on known failures while the latest run finishes.

4. Jira → ticket links

If JIRA_TOKEN is set, QueuePilot searches jira.corp.adobe.com for each unique failing test:

  • JQL: project = ACQE AND summary ~ "{test_method_name}" ORDER BY created DESC
  • Only non-Done, non-Cancelled tickets are surfaced; picks most recently created active ticket
  • Up to 5 Jira lookups in parallel
  • Results attached to each failure, shown in both the Slack summary and the HTML report

5. Slack → report delivery

  • Posted via chat.postMessage with thread_ts = qmbot's message ts
  • One bot reply in qmbot's thread — nothing posted at top level of the channel
  • The Slack Connect bot identity (the SLACK_TOKEN owner) is used
  • No file attachments — summary message only

6. HTML report → local file

A self-contained HTML dashboard is saved to reports/ in the project directory.

How the Slack Result Looks

The report is posted as a single thread reply inside qmbot's message. It has three parts:

Part 1 — Header

🐛 QueuePilot — `2.4-develop`
*3 PR(s)* in queue  ·  *5 unique failing test(s)*

Part 2 — Unique failures (sorted by total occurrence count)

Each line shows the test method name, how many times it failed in total across all PRs and all editions (CE + EE + B2B combined), and the linked Jira ticket if one exists.

• `AdminUserSetStatusForEachSourceItemTest`  *4x*  → ACQE-9629 _Tech Analysis_
• `StorefrontCheckTermsAndConditionIsPresentTest`  *3x*  → ACQE-10249 _Options Queue_
• `StockStatusChangedForConfigurableProductTest`  *2x*
• `CreateConfigurableProductWithTierPriceTest`  *1x*

4x means that test failed 4 times in total — for example CE+EE for PR1 and CE for PR2. It is not a PR count.

Part 3 — Per-PR breakdown

One block per PR showing the GitHub link, author, CE/EE/B2B badge, and a deduplicated list of failing test names (union across all editions):

*#10762* Fix cart total rounding issue
by *harshityadav90*  ·  CE: *2 FAIL*  EE: ✅ PASS  B2B: *3 FAIL*
  • `AdminUserSetStatusForEachSourceItemTest`
  • `StorefrontCheckTermsAndConditionIsPresentTest`
  • `StockStatusChangedForConfigurableProductTest`

*#10758* Support Tier-4 flowers bugfixes
by *thiaramus*  ·  CE: *1 FAIL*  EE: *4 FAIL*  B2B: ✅ PASS
  • `CreateConfigurableProductWithTierPriceTest`

Badge meanings:

Badge Meaning
N FAIL N tests failed in this edition
✅ PASS All checks passed
⏳ RUNNING Check still in progress, no previous failures available
⏳ RUNNING · N prev fail Check running; N failures found from previous runs
N/A Check has not run for this edition

HTML Dashboard

Saved to reports/ in the project directory. Contains:

Stats Bar

PR count and unique failing test count.

Queue Summary Table

One row per PR with CE / EE / B2B status badges.

All Failing Tests Table

Every unique failing test across all PRs and editions, sorted by total failure count, with Jira ticket links.

Per-PR Failure Cards

3-column layout (CE / EE / B2B). Each column shows full test method names with color-coded status (red = failed, amber = broken), Jira badge, Allure link, and Jenkins link.

Cost

All integrations are free — no LLM, no third-party service, no usage fees:

Integration API Cost
Slack read conversations.history Free (bot token)
Slack write chat.postMessage Free (bot token)
GitHub gh CLI + REST API Free (authenticated user)
Allure Static JSON from internal report URL Free
Jira REST API v2 search Free (internal PAT)

Setup

Prerequisites

Tool Install Verify
Python 3.10+ python.org python3 --version
GitHub CLI brew install gh gh --version

One-command install

git clone https://github.com/chandrakalahanchinal/queuepilot.git
cd queuepilot
bash setup.sh

setup.sh will:

  1. Install Python dependencies (pip install -r requirements.txt)
  2. Authenticate GitHub CLI if not already done
  3. Prompt for SLACK_TOKEN and optionally JIRA_TOKEN
  4. Install and start the watcher as a macOS LaunchAgent (auto-starts at every login)

Watcher commands

tail -f watcher.log                                                         # live logs
launchctl unload ~/Library/LaunchAgents/com.queuepilot.watcher.plist       # stop
bash setup.sh                                                               # start / update tokens

Tokens

Token Where to get it Required?
SLACK_TOKEN Slack app settings → OAuth tokens → Bot token (xoxb-...) Yes
JIRA_TOKEN Jira → Profile → Personal Access Tokens No — enables Jira links

Running Manually

# Auto mode — reads qmbot's latest message, analyzes, posts to Slack
export SLACK_TOKEN=xoxb-your-token-here
export JIRA_TOKEN=your-jira-token
python3 queuepilot.py 2.4-develop

# Pass PR numbers directly — skip Slack read
python3 queuepilot.py 2.4-develop --prs 10737 10740 10741

# HTML only — no Slack post
python3 queuepilot.py 2.4-develop --no-slack

# Reply into a specific qmbot thread (watcher sets this automatically)
python3 queuepilot.py 2.4-develop --reply-to-ts 1718000000.123456

CLI Reference

python3 queuepilot.py <branch> [options]

positional:
  branch                  Queue branch name, e.g. 2.4-develop

options:
  --output FILE           HTML output path (default: reports/queuepilot-{branch}-{date}-{n}.html)
  --no-slack              Generate HTML only, skip Slack post
  --prs N [N ...]         Analyze these PR numbers; skip reading from Slack
  --jira-token TOKEN      Jira PAT for ACQE ticket lookup (or set JIRA_TOKEN env var)
  --allure-attempts N     Allure retry count, default 2 (~10s each)
  --reply-to-ts TS        Slack message ts to post summary as thread reply
                          (set automatically by watcher — rarely needed manually)

Architecture

Files

File Purpose
queuepilot.py Main analysis + Slack reporting script
watch.py Background watcher daemon — triggers queuepilot.py
setup.sh One-command install and LaunchAgent setup
requirements.txt Python dependencies (requests)
reports/ Generated HTML dashboards
.watcher_state.json Watcher runtime state (last processed ts, fire time, qmbot reply ts)

watch.py — State machine

Constant Value Description
POLL_INTERVAL 30s Slack polling frequency
TRIGGER_WINDOW 600s Time allowed for qmbot to reply after trigger
WAIT_BEFORE_FIRE 300s Wait after qmbot replies before running analysis

Transitions:

  1. Any user sends dq 2.4-develop → saves pending_trigger_ts
  2. qmbot replies with PR URLs within 10 min → saves fire_after_ts + qmbot_reply_ts
  3. fire_after_ts reached → runs queuepilot.py --reply-to-ts <qmbot_reply_ts> → clears state

queuepilot.py — Code layers

Layer Function Purpose
Slack slack_history Fetch channel messages after a given timestamp
Slack parse_pr_list Extract repo + PR number pairs from qmbot's GitHub URLs
Slack slack_post_dashboard Post formatted summary blocks as thread reply
GitHub analyze_pr Fetch PR info and check runs; dispatch per-edition analysis
GitHub _analyze_edition Check one edition's conclusion; fetch Allure failures if failed
GitHub find_allure_url_from_jenkins Scrape Jenkins page as fallback for Allure URL
Allure get_failed_uids Walk categories/suites/behaviors JSON; collect failed/broken UIDs
Allure resolve_failures Resolve all UIDs → method names (10 parallel workers)
Allure get_prometheus_stats Fallback: read failure counts from prometheusData.txt
Jira search_jira_for_test JQL search in ACQE project for one test method name
Jira fetch_jira_tickets Parallel lookup for all unique failing tests (5 workers)
HTML render_all_failures_table Aggregated failures sorted by total occurrence count
HTML render_html Assembles complete self-contained HTML page

Parallelism

Level Workers
PR analysis 5 (PR_WORKER_MAX)
Test name resolution per PR 10 (ALLURE_WORKER_MAX)
Jira ticket lookup 5

Troubleshooting

Problem Fix
Watcher not firing tail -f watcher.log — common causes: SLACK_TOKEN expired (re-run bash setup.sh), token lacks channels:history scope, stuck state (delete .watcher_state.json)
Report at top level instead of thread Watcher sets --reply-to-ts automatically. For manual runs pass --reply-to-ts <ts>
ERROR: SLACK_TOKEN env var is required Use --prs: python3 queuepilot.py 2.4-develop --prs 10762 10758
CE/EE/B2B shows "test data fetch failed" Allure data not ready — build likely crashed before tests ran. Check Jenkins directly
Jira tickets not showing Verify: curl -H "Authorization: Bearer <token>" https://jira.corp.adobe.com/rest/api/2/myself
Test names show as UIDs test-cases/{uid}.json not yet uploaded. Wait a few minutes and re-run
gh: command not found brew install gh && gh auth login

Clone this wiki locally