AIP-97 POC: tag failure metrics with failure_kind (both emission sites)

[1fanwang]

POC for AIP-97, stacked on the Celery classifier (#15). This carries the classified cause into failure metrics, not only listeners and callbacks.

Why

ti_failures and operator_failures were emitted without failure_kind from two paths: the scheduler for an externally killed task, and the worker task runner for a failure caught by the worker. An evicted task and a ValueError therefore looked identical on the failure dashboards. AIP-97 keeps infra churn off the retry budget, but it also needs to keep it visible in metrics.

What

Tag both metrics with failure_kind at both emission sites. The scheduler side tags an externally killed task with the executor's kind, such as infra when classified. The worker side derives the kind from the caught exception: AirflowTaskTimeout becomes timeout; other caught exceptions become application. The tag is omitted when nothing classified the failure, so the unclassified metric shape is unchanged.

Testing

Live: statsd capture on a real Celery stack

Ran the Docker Celery stack (postgres, redis, api-server, scheduler, Dag processor, and celery worker) with statsd on and statsd_influxdb_enabled=True so tags render on the wire. A UDP sink captured two runs.

#	Scenario	Trigger	Tag on the wire
1	Infra kill	SIGKILL the billiard pool child mid-task, so Celery raises WorkerLost	failure_kind=infra
2	Application failure	task raises ValueError	failure_kind=application

Raw statsd packets

Infra, emitted scheduler-side:

airflow.ti_failures,dag_id=celery_infra_e2e,run_type=manual,task_id=sleeper,failure_kind=infra:1|c
airflow.operator_failures,dag_id=celery_infra_e2e,run_type=manual,task_id=sleeper,operator_name=PythonOperator,failure_kind=infra:1|c
airflow.operator_failures_PythonOperator,dag_id=celery_infra_e2e,run_type=manual,task_id=sleeper,failure_kind=infra:1|c

Application, emitted worker-side:

airflow.ti_failures,dag_id=celery_normalfail_e2e,task_id=boom,run_type=manual,failure_kind=application:1|c
airflow.operator_failures,dag_id=celery_normalfail_e2e,task_id=boom,run_type=manual,operator_name=PythonOperator,failure_kind=application:1|c
airflow.operator_failures_PythonOperator,dag_id=celery_normalfail_e2e,task_id=boom,run_type=manual,failure_kind=application:1|c

That capture also caught the gap fixed by the second commit. The first commit tagged only the scheduler emission, so scenario 2 emitted an untagged metric until the worker path was tagged too.

Live: OTel to Prometheus to Grafana backend, plus cardinality

The statsd capture proves the tag leaves both emission sites. This run carries it into a real backend and checks cardinality. Full writeup and reproduce steps: evidence branch.

Pipeline with no mocks in the path: Airflow OTel Stats client to OpenTelemetry Collector 0.116.1 to Prometheus v3.1.0 to Grafana 11.4.0. The collector's Prometheus exporter renames ti_failures to airflow_ti_failures_total and keeps the labels.

Cardinality: failure_kind is a bounded 4-value label on a metric that only fires on failure and is already keyed by dag_id and task_id, so it multiplies the series count by at most 4, usually less. A realistic matrix (5 Dags by 3 tasks, each failing a subset of kinds) produced 47 series from a base of 15, below the 60 worst case:

count by (failure_kind) (airflow_ti_failures_total)
  application 15   infra 15   manual 8   timeout 9   = 47

A 4-value enum is a normal metrics label. High cardinality would come from an unbounded label such as a run id or a free-text error string.

Prometheus, all 47 tagged series:

Series per kind (15 + 15 + 8 + 9 = 47):

Grafana, failures grouped by failure_kind with the series count. The bars separate infra churn (68 events) from application bugs (50), timeouts (21), and manual stops (8):

Airflow mocks Stats under pytest, so these backend numbers come from the real OTel Stats client driven with the POC's tag shape outside pytest. The handle_failure to stats.incr wiring is covered by the unit tests below.

Unit

Scheduler side: an infra failure tags both metrics; the unclassified path is unchanged. Worker side: an application failure tags both metrics in the legacy and influx forms, and successes stay untagged. All green.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ceb5d2fb-a8e8-4749-b818-818883f2dde0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch aip97-poc-metrics-kind

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}