Fix compatible issue with transformers 5.0+ #2328
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # Apply patches for DynamicCache / past_key_values compatibility | ||
| if version.parse(transformers.__version__) >= version.parse("5.0"): | ||
| # transformers >= 5.0: DynamicCache refactored to use DynamicLayer | ||
| from transformers.integrations.executorch import register_dynamic_cache_export_support |
There was a problem hiding this comment.
register_dynamic_cache_export_support Does not have the right ordering for kvcaches. I would just use the same patch regardless of the transformers version
There was a problem hiding this comment.
Do you mean _patch_model_if_necessary? transformers 5.0 updated DynamicCache and i got error "AttributeError: 'DynamicCache' object has no attribute 'to_legacy_cache'"
There was a problem hiding this comment.
We can update the patch code (_patch_model_if_necessary) so that it works universally. There is no need to call to_legacy_cache. The executorch integration is not reliable for our usages.
There was a problem hiding this comment.
@titaiwangms suggestions on what the patch logic should be?
| # transformers >= 5.0: DynamicCache refactored to use DynamicLayer | ||
| from transformers.integrations.executorch import register_dynamic_cache_export_support | ||
|
|
||
| register_dynamic_cache_export_support() |
There was a problem hiding this comment.
@xadupre Do you have any suggestion to avoid using executorch function? From Justin's reminder, it provides a wrong kv cache order?
| logger.debug("Patched DynamicLayer.lazy_initialization for torch.export compatibility.") | ||
|
|
||
|
|
||
| def _convert_past_key_values_to_dynamic_cache(dummy_kwargs: dict) -> dict: |
There was a problem hiding this comment.
With transformers 5+, the update mechanism is not defined by the class DynamicCache but by the class of each layer. This code only works for a DynamicCache using DynamicLayer only. That won't work for a DynamicCache mixing DynamicLayer and DynamicSlidingWindowLayer. The code is fine but it is better to keep that in mind for other models using sliding windows.
| soundfile | ||
| tabulate | ||
| torchvision | ||
| # Remove version pin when the tests are fixed |
There was a problem hiding this comment.
Should you add transformers>=5 unless there are some tests checking multiple versions of transformers.
4 participants
Describe your changes
This pull request introduces compatibility updates for Hugging Face Transformers 5.0 and improves handling of dynamic cache and input formats in Olive's ONNX conversion and training utilities. It also updates tests and requirements to reflect these changes and ensure robust model export and training workflows.
Transformers 5.0 Compatibility
DynamicLayer.lazy_initialization,past_key_values, and dynamic shapes to support the new DynamicCache format in Transformers >= 5.0. This ensures models using dynamic cache export correctly withtorch.export._export_pytorch_modellogic to apply the new patches and conversions only for Transformers >= 5.0, while maintaining legacy support for older versions.Training Argument Handling
create_training_argsto remove fields not valid for Transformers 5.0 and excludeNonevalues, allowing Transformers to use its own defaults.Test Suite Updates
trust_remote_codeparameter and update expected file counts and tokenizer types for Transformers 5.0. [1] [2] [3] [4]floatbefore comparison, ensuring consistency across dtypes.Requirements Adjustments
onnxscriptversion to<0.6.1and removed the Transformers version pin, reflecting confidence in test suite compatibility with Transformers 5.0. [1] [2]Checklist before requesting a review
lintrunner -a(Optional) Issue link