Skip to content

fix: serialize missing init params in to_dict (transformers, huggingface_api, ragas) - #3808

Open
pcbeingused333 wants to merge 3 commits into
deepset-ai:mainfrom
pcbeingused333:fix/zero-shot-text-router-multi-label-serde
Open

fix: serialize missing init params in to_dict (transformers, huggingface_api, ragas)#3808
pcbeingused333 wants to merge 3 commits into
deepset-ai:mainfrom
pcbeingused333:fix/zero-shot-text-router-multi-label-serde

Conversation

@pcbeingused333

@pcbeingused333 pcbeingused333 commented Aug 17, 2026

Copy link
Copy Markdown

Related Issues

  • No issue; found while auditing to_dict against __init__ across the integrations.

Proposed Changes:

Three components accept an __init__ parameter, store it, and use it at run time, but leave it out of to_dict. In each case the value silently falls back to its default after a to_dict / from_dict round trip, and nothing in the serialized dict shows that the setting was ever there. from_dict needed no change in any of them — default_from_dict passes init_parameters straight through to __init__.

TransformersZeroShotTextRouter.multi_label — passed to the Hugging Face pipeline in run:

self.pipeline([text], candidate_labels=self.labels, multi_label=self.multi_label)

The flag decides whether label scores are normalized to sum to 1 across labels, or whether each label is scored independently against its own contradiction score. The router picks its output branch from those scores, so the same text can route to a different branch after a pipeline is saved and reloaded. model and device are absent from to_dict too, but correctly so: _resolve_hf_pipeline_kwargs folds them into huggingface_pipeline_kwargs, which is serialized. multi_label is the only one that goes nowhere. The sibling component in this same integration, TransformersZeroShotDocumentClassifier, has the same shape and already serializes it — which is what suggests an oversight rather than a decision.

HuggingFaceTEIRanker.raw_scores — goes directly into the request payload, on both the sync and async paths:

payload: dict[str, Any] = {"query": query, "texts": texts, "raw_scores": self.raw_scores}

After a reload the ranker silently stops asking the TEI endpoint for raw relevance scores: the request body changes with nothing to show why. Everything else __init__ takes (url, top_k, timeout, token, max_retries, retry_status_codes) is already serialized; raw_scores is the only one missing.

RagasEvaluator.concurrency_limit — sizes the semaphore bounding how many metrics are evaluated at once in run_async:

sem = Semaphore(max(1, self.concurrency_limit))

to_dict serialized only ragas_metrics, so an evaluator built with concurrency_limit=16 comes back at the default of 4. Every one of those concurrent evaluations is an LLM call, so the value decides whether an evaluation run finishes in a quarter of the time or hits the judge model's rate limit.

How did you test it?

One new round-trip regression test per component, each asserting the value survives to_dictfrom_dict:

Component Test Suite
TransformersZeroShotTextRouter test_multi_label_survives_a_serialization_round_trip hatch run test:pytest tests/test_zero_shot_text_router.py — 9 passed
HuggingFaceTEIRanker test_raw_scores_survives_a_serialization_round_trip hatch run test:pytest tests/test_ranker.py — 16 passed
RagasEvaluator test_concurrency_limit_survives_a_serialization_round_trip hatch run test:pytest tests/test_evaluator.py — 27 passed, 8 skipped (integration tests needing an API key)

hatch run test:types and hatch run fmt-check are clean in all three integrations.

I checked each regression test is not vacuous — with the production file reverted to main, each fails for the reason the bug describes rather than on an incidental assertion mismatch:

  • router: assert False is True on the restored component's multi_label
  • ranker: assert False is True on the restored component's raw_scores
  • evaluator: assert 4 == 16 — the round trip falling back to the default

test_to_dict in the transformers and ragas suites compares the whole dict, so both needed the new key added in the same commit. The huggingface_api test_to_dict asserts key by key, which is also why that gap survived: a full-dict assertion would have flagged the missing key when raw_scores was added.

Notes for the reviewer

This supersedes #3809 (raw_scores) and #3810 (concurrency_limit), which I have closed in favour of this one. Those PRs said I would keep the fixes separate because each integration ships on its own; I have grouped them here instead, since it is one class of bug found by one sweep and it reads better reviewed as a whole than as three near-identical PRs. Releases are unaffected — CI_pypi_release.yml generates each changelog with git-cliff --include-path "<integration>/**/*", so the entry is partitioned by file path, not by the commit scope in the title.

The three commits are kept separate and per-integration, so the change can still be split back apart if you would rather take them one at a time.

concurrency_limit is documented as only affecting run_async, so that round trip is silent on the sync path — part of why it is easy to miss.

I used an AI assistant while writing this change. I have reviewed it, reproduced the behaviour, and run the tests.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title.
  • I have documented my code.
  • I have run pre-commit hooks and fixed any issue.

…outer

to_dict left multi_label out, so a router built with multi_label=True came
back from from_dict with it set to False. The flag is passed straight to the
zero-shot pipeline and decides whether label scores are normalized to sum to
1 across labels or treated independently, so a pipeline saved and reloaded
routes differently than the one that was built.

The sibling TransformersZeroShotDocumentClassifier, same integration and same
shape, already serializes it.
@pcbeingused333
pcbeingused333 requested a review from a team as a code owner August 17, 2026 23:40
@pcbeingused333
pcbeingused333 requested review from bogdankostic and removed request for a team August 17, 2026 23:40
@github-actions github-actions Bot added integration:transformers type:documentation Improvements or additions to documentation labels Aug 17, 2026
@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Heads-up for maintainers

This PR is from a fork and touches integrations whose integration tests require API keys.
Those tests are skipped in CI because fork PRs don't have access to repo secrets for security reasons.

Affected integrations:

  • huggingface_api
  • ragas
  • transformers

Please run the integration tests locally (hatch run test:integration inside each folder) before approving.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage report (transformers)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/transformers/src/haystack_integrations/components/routers/transformers
  zero_shot_text_router.py
Project Total  

This report was generated by python-coverage-comment-action

to_dict left raw_scores out, so a ranker built with raw_scores=True came back
from from_dict with it at False. The flag goes straight into the request
payload sent to the TEI endpoint, on both the sync and the async path, so a
pipeline saved and reloaded stops asking the service for raw relevance
scores.
to_dict serialized only ragas_metrics, so an evaluator built with a
concurrency_limit came back from from_dict at the default of 4. The value
sizes the semaphore that caps how many metric evaluations run at once in
run_async, and every one of those is an LLM call, so a saved and reloaded
evaluation pipeline runs at a different concurrency than the one that was
configured.
@pcbeingused333 pcbeingused333 changed the title fix(transformers): serialize multi_label in TransformersZeroShotTextRouter fix: serialize missing init params in to_dict (transformers, huggingface_api, ragas) Aug 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report (huggingface_api)

This PR does not seem to contain any modification to coverable code.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage report (ragas)

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  integrations/ragas/src/haystack_integrations/components/evaluators/ragas
  evaluator.py
Project Total  

This report was generated by python-coverage-comment-action

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant