[Serve][LLM] Fix malformed PromQL in Serve LLM Grafana dashboard#63893
Open
eicherseiji wants to merge 2 commits into
Open
[Serve][LLM] Fix malformed PromQL in Serve LLM Grafana dashboard#63893eicherseiji wants to merge 2 commits into
eicherseiji wants to merge 2 commits into
Conversation
The Serve LLM Grafana dashboard generated label matchers with a leading
comma (`{, deployment=~"$deployment"}`) or a double comma when the
optional `{global_filters}` substitution was empty, which Prometheus
rejects as a parse error.
Move `{global_filters}` to the trailing position in `_WORKER_JOIN` and
`_VLLM_DEPLOYMENT_FILTER` so an empty substitution leaves a tolerated
trailing comma, matching the convention already used by `_VLLM_FILTER`
and `_WORKERID_FILTER`.
Fixes ray-project#63219
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
Contributor
There was a problem hiding this comment.
Code Review
This pull request repositions the {global_filters} placeholder to the end of the Prometheus metric filter lists in serve_llm_dashboard_panels.py to prevent syntax errors (like leading or double commas) when the substitution is empty. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
bfbb039 to
7599c94
Compare
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
7599c94 to
bb9653f
Compare
jeffreywang-anyscale
approved these changes
Jun 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
The Serve LLM Grafana dashboard (
generate_serve_llm_grafana_dashboard) emits PromQL label matchers with a stray comma when the optional{global_filters}substitution is empty (the default). With no global filters set, the generated queries contain:ray_serve_deployment_request_counter_total{, deployment=~"$deployment"}... WorkerId=~"$workerid", , deployment=~"$deployment"Prometheus rejects both:
Root cause:
_WORKER_JOINand_VLLM_DEPLOYMENT_FILTERplaced{global_filters}in a non-trailing position, so an empty substitution left a leading or double comma. The other fragments (_VLLM_FILTER,_WORKERID_FILTER) already keep{global_filters}trailing, where an empty substitution yields only a trailing comma (which Prometheus tolerates).This moves
{global_filters}to the trailing position in the two offending fragments to match that convention.Fixes #63219
Related issue number
Closes #63219
Checks
-s) per the DCO requirement.ruff/blackpass via pre-commit on the changed file.