Skip to content

Fix token usage- total tokens #45202

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
YoYoJa wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Fix token usage- total tokens #45202

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
207d722
Fix token usage- total tokens
YoYoJa Feb 15, 2026
2770d0b
run black
YoYoJa Feb 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions ...luation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,23 +299,29 @@ def _parse_eval_result(self, eval_result) -> Dict[str, T]:

# Extract token counts from metrics
metrics = properties.get("metrics", {})
prompt_tokens = metrics.get("promptTokens", "")
completion_tokens = metrics.get("completionTokens", "")
prompt_tokens = int(metrics.get("promptTokens", 0)) if metrics.get("promptTokens") else 0
completion_tokens = (
int(metrics.get("completionTokens", 0)) if metrics.get("completionTokens") else 0
)

# Calculate total tokens
try:
total_tokens = (
str(int(prompt_tokens) + int(completion_tokens))
int(prompt_tokens) + int(completion_tokens)
if prompt_tokens and completion_tokens
else ""
else 0
)
Comment on lines 309 to 313
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The condition on line 309 checks if both prompt_tokens and completion_tokens are truthy, which means if either value is 0, total_tokens will be set to 0 instead of the correct sum. Since prompt_tokens and completion_tokens are already guaranteed to be integers (defaulting to 0), this condition is incorrect. The calculation should be unconditional:

total_tokens = prompt_tokens + completion_tokens

This bug causes total_tokens to be 0 when either prompt_tokens or completion_tokens is 0, even though the other might have a non-zero value.

Copilot uses AI. Check for mistakes.
except (ValueError, TypeError):
total_tokens = ""
total_tokens = 0

# Add token metadata (matching old format)
parsed_result[f"{self._eval_metric. value}_total_tokens"] = total_tokens
parsed_result[f"{self._eval_metric.value}_prompt_tokens"] = prompt_tokens
parsed_result[f"{self._eval_metric.value}_completion_tokens"] = completion_tokens
parsed_result[f"{self._eval_metric.value}_prompt_tokens"] = (
int(prompt_tokens) if prompt_tokens else 0
)
parsed_result[f"{self._eval_metric.value}_completion_tokens"] = (
int(completion_tokens) if completion_tokens else 0
)

# Add empty placeholders for fields that sync_evals doesn't provide
parsed_result[f"{self._eval_metric.value}_finish_reason"] = ""
Expand All @@ -334,17 +340,17 @@ def _parse_eval_result(self, eval_result) -> Dict[str, T]:

# Extract token counts
metrics = properties.get("metrics", {})
prompt_tokens = metrics.get("promptTokens", "")
completion_tokens = metrics.get("completionTokens", "")
prompt_tokens = int(metrics.get("promptTokens", 0)) if metrics.get("promptTokens") else 0
completion_tokens = (
int(metrics.get("completionTokens", 0)) if metrics.get("completionTokens") else 0
)

try:
total_tokens = (
str(int(prompt_tokens) + int(completion_tokens))
if prompt_tokens and completion_tokens
else ""
int(prompt_tokens) + int(completion_tokens) if prompt_tokens and completion_tokens else 0
)
Comment on lines 349 to 351
Copy link

Copilot AI Feb 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same bug as lines 307-311. The condition on line 343 checks if both tokens are truthy, which incorrectly returns 0 when either value is 0. Since the variables are already integers, the calculation should be unconditional:

total_tokens = prompt_tokens + completion_tokens

Copilot uses AI. Check for mistakes.
except (ValueError, TypeError):
total_tokens = ""
total_tokens = 0

# Return in the expected format matching parse_response output
return {
Expand Down
Loading