Skip to content

[Serve][LLM] Fix malformed PromQL in Serve LLM Grafana dashboard#63893

Open
eicherseiji wants to merge 2 commits into
ray-project:masterfrom
eicherseiji:fix/serve-llm-grafana-promql
Open

[Serve][LLM] Fix malformed PromQL in Serve LLM Grafana dashboard#63893
eicherseiji wants to merge 2 commits into
ray-project:masterfrom
eicherseiji:fix/serve-llm-grafana-promql

Conversation

@eicherseiji
Copy link
Copy Markdown
Contributor

Why are these changes needed?

The Serve LLM Grafana dashboard (generate_serve_llm_grafana_dashboard) emits PromQL label matchers with a stray comma when the optional {global_filters} substitution is empty (the default). With no global filters set, the generated queries contain:

  • 32 leading-comma matchers, e.g. ray_serve_deployment_request_counter_total{, deployment=~"$deployment"}
  • 2 double-comma matchers, e.g. ... WorkerId=~"$workerid", , deployment=~"$deployment"

Prometheus rejects both:

bad_data: invalid parameter "query": parse error: unexpected "," in label matching, expected identifier or "}"

Root cause: _WORKER_JOIN and _VLLM_DEPLOYMENT_FILTER placed {global_filters} in a non-trailing position, so an empty substitution left a leading or double comma. The other fragments (_VLLM_FILTER, _WORKERID_FILTER) already keep {global_filters} trailing, where an empty substitution yields only a trailing comma (which Prometheus tolerates).

This moves {global_filters} to the trailing position in the two offending fragments to match that convention.

Fixes #63219

Related issue number

Closes #63219

Checks

  • I've signed off every commit (-s) per the DCO requirement.
  • ruff/black pass via pre-commit on the changed file.

The Serve LLM Grafana dashboard generated label matchers with a leading
comma (`{, deployment=~"$deployment"}`) or a double comma when the
optional `{global_filters}` substitution was empty, which Prometheus
rejects as a parse error.

Move `{global_filters}` to the trailing position in `_WORKER_JOIN` and
`_VLLM_DEPLOYMENT_FILTER` so an empty substitution leaves a tolerated
trailing comma, matching the convention already used by `_VLLM_FILTER`
and `_WORKERID_FILTER`.

Fixes ray-project#63219

Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji eicherseiji added the go add ONLY when ready to merge, run all tests label Jun 5, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request repositions the {global_filters} placeholder to the end of the Prometheus metric filter lists in serve_llm_dashboard_panels.py to prevent syntax errors (like leading or double commas) when the substitution is empty. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@eicherseiji eicherseiji marked this pull request as ready for review June 5, 2026 21:58
@eicherseiji eicherseiji requested a review from a team as a code owner June 5, 2026 21:58
@ray-gardener ray-gardener Bot added serve Ray Serve Related Issue observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling labels Jun 6, 2026
@eicherseiji eicherseiji requested a review from a team as a code owner June 8, 2026 18:30
@eicherseiji eicherseiji force-pushed the fix/serve-llm-grafana-promql branch from bfbb039 to 7599c94 Compare June 8, 2026 18:32
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji eicherseiji force-pushed the fix/serve-llm-grafana-promql branch from 7599c94 to bb9653f Compare June 8, 2026 18:34
@eicherseiji eicherseiji removed the request for review from a team June 8, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling serve Ray Serve Related Issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Serve][LLM][Grafana] Ray Serve LLM Grafana dashboard ships with malformed PromQL

2 participants