fix: serialize missing init params in to_dict (transformers, huggingface_api, ragas) - #3808
Open
pcbeingused333 wants to merge 3 commits into
Open
Conversation
…outer to_dict left multi_label out, so a router built with multi_label=True came back from from_dict with it set to False. The flag is passed straight to the zero-shot pipeline and decides whether label scores are normalized to sum to 1 across labels or treated independently, so a pipeline saved and reloaded routes differently than the one that was built. The sibling TransformersZeroShotDocumentClassifier, same integration and same shape, already serializes it.
pcbeingused333
requested review from
bogdankostic
and removed request for
a team
August 17, 2026 23:40
Contributor
|
Heads-up for maintainers This PR is from a fork and touches integrations whose integration tests require API keys. Affected integrations:
Please run the integration tests locally ( |
Contributor
Coverage report (transformers)Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
This was referenced Aug 18, 2026
to_dict left raw_scores out, so a ranker built with raw_scores=True came back from from_dict with it at False. The flag goes straight into the request payload sent to the TEI endpoint, on both the sync and the async path, so a pipeline saved and reloaded stops asking the service for raw relevance scores.
to_dict serialized only ragas_metrics, so an evaluator built with a concurrency_limit came back from from_dict at the default of 4. The value sizes the semaphore that caps how many metric evaluations run at once in run_async, and every one of those is an LLM call, so a saved and reloaded evaluation pipeline runs at a different concurrency than the one that was configured.
Contributor
Contributor
Coverage report (ragas)Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
to_dictagainst__init__across the integrations.Proposed Changes:
Three components accept an
__init__parameter, store it, and use it at run time, but leave it out ofto_dict. In each case the value silently falls back to its default after ato_dict/from_dictround trip, and nothing in the serialized dict shows that the setting was ever there.from_dictneeded no change in any of them —default_from_dictpassesinit_parametersstraight through to__init__.TransformersZeroShotTextRouter.multi_label— passed to the Hugging Face pipeline inrun:The flag decides whether label scores are normalized to sum to 1 across labels, or whether each label is scored independently against its own contradiction score. The router picks its output branch from those scores, so the same text can route to a different branch after a pipeline is saved and reloaded.
modelanddeviceare absent fromto_dicttoo, but correctly so:_resolve_hf_pipeline_kwargsfolds them intohuggingface_pipeline_kwargs, which is serialized.multi_labelis the only one that goes nowhere. The sibling component in this same integration,TransformersZeroShotDocumentClassifier, has the same shape and already serializes it — which is what suggests an oversight rather than a decision.HuggingFaceTEIRanker.raw_scores— goes directly into the request payload, on both the sync and async paths:After a reload the ranker silently stops asking the TEI endpoint for raw relevance scores: the request body changes with nothing to show why. Everything else
__init__takes (url,top_k,timeout,token,max_retries,retry_status_codes) is already serialized;raw_scoresis the only one missing.RagasEvaluator.concurrency_limit— sizes the semaphore bounding how many metrics are evaluated at once inrun_async:to_dictserialized onlyragas_metrics, so an evaluator built withconcurrency_limit=16comes back at the default of4. Every one of those concurrent evaluations is an LLM call, so the value decides whether an evaluation run finishes in a quarter of the time or hits the judge model's rate limit.How did you test it?
One new round-trip regression test per component, each asserting the value survives
to_dict→from_dict:TransformersZeroShotTextRoutertest_multi_label_survives_a_serialization_round_triphatch run test:pytest tests/test_zero_shot_text_router.py— 9 passedHuggingFaceTEIRankertest_raw_scores_survives_a_serialization_round_triphatch run test:pytest tests/test_ranker.py— 16 passedRagasEvaluatortest_concurrency_limit_survives_a_serialization_round_triphatch run test:pytest tests/test_evaluator.py— 27 passed, 8 skipped (integration tests needing an API key)hatch run test:typesandhatch run fmt-checkare clean in all three integrations.I checked each regression test is not vacuous — with the production file reverted to
main, each fails for the reason the bug describes rather than on an incidental assertion mismatch:assert False is Trueon the restored component'smulti_labelassert False is Trueon the restored component'sraw_scoresassert 4 == 16— the round trip falling back to the defaulttest_to_dictin the transformers and ragas suites compares the whole dict, so both needed the new key added in the same commit. The huggingface_apitest_to_dictasserts key by key, which is also why that gap survived: a full-dict assertion would have flagged the missing key whenraw_scoreswas added.Notes for the reviewer
This supersedes #3809 (
raw_scores) and #3810 (concurrency_limit), which I have closed in favour of this one. Those PRs said I would keep the fixes separate because each integration ships on its own; I have grouped them here instead, since it is one class of bug found by one sweep and it reads better reviewed as a whole than as three near-identical PRs. Releases are unaffected —CI_pypi_release.ymlgenerates each changelog withgit-cliff --include-path "<integration>/**/*", so the entry is partitioned by file path, not by the commit scope in the title.The three commits are kept separate and per-integration, so the change can still be split back apart if you would rather take them one at a time.
concurrency_limitis documented as only affectingrun_async, so that round trip is silent on the sync path — part of why it is easy to miss.I used an AI assistant while writing this change. I have reviewed it, reproduced the behaviour, and run the tests.
Checklist