Upgrade transformers v5

[ethanlee928]

Related to #383.

Hi colpali team,

I found some time to work on upgrading transformers to 5.0.0, which requires several code changes. This PR contains the current state of that work.

@QuentinJGMace, sorry for jumping in on this upgrade. If you don't mind, could you help take a look and review the change when you have time? Also please feel free to take over. Thank you very much!

Summary

peft version

• Updated minimum version requirement. Refer to this HF Issue.

paligemma changes

• Referred to https://github.com/linkedin/Liger-Kernel/pull/966/changes to add extra model. layer
• Removed self.model.lm_head = torch.nn.Identity() as it caused AttributeError: Identity has no attribute weight.

others

• Updated __init__ to explicitly accept image_processor and tokenizer arguments

Testing Environment

• Python version: 3.12.11
• OS Version: Ubuntu 22.04.5 LTS
• CUDA Version: 13.0
• NVIDIA Driver Version: 580.95.05
• GPU: NVIDIA GeForce RTX 3090

Testing Results

I ran the test suite with both versions.

Most failures appear to be unrelated to the transformers upgrade itself, but there are some new/different errors in this branch.

main branch:

transformers==4.57.6

================================================= short test summary info =================================================
FAILED tests/models/gemma3/bigemma3/test_modeling_bigemma3.py::TestBiGemma3_ModelIntegration::test_retrieval_integration - AssertionError: assert tensor(False, device='cuda:0')
FAILED tests/models/paligemma/colpali/test_modeling_colpali.py::TestColPaliModelIntegration::test_retrieval_integration - AssertionError: Expected scores tensor([[16.5000,  7.5938, 15.6875],
ERROR tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3Model::test_load_model_from_pretrained - AttributeError: 'NoneType' object has no attribute 'get'
ERROR tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3ModelIntegration::test_forward_images_integration - AttributeError: 'NoneType' object has no attribute 'get'
ERROR tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3ModelIntegration::test_forward_queries_integration - AttributeError: 'NoneType' object has no attribute 'get'
ERROR tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3ModelIntegration::test_retrieval_integration - AttributeError: 'NoneType' object has no attribute 'get'
=================================== 2 failed, 161 passed, 4 errors in 152.67s (0:02:32) ===================================

This PR:

transformers==5.0.0

============================================================================ short test summary info =============================================================================
FAILED tests/models/gemma3/bigemma3/test_modeling_bigemma3.py::TestBiGemma3_ModelIntegration::test_retrieval_integration - AssertionError: assert tensor(False, device='cuda:0')
FAILED tests/models/paligemma/colpali/test_modeling_colpali.py::TestColPaliModelIntegration::test_retrieval_integration - AssertionError: Expected scores tensor([[16.5000,  7.5938, 15.6875],
FAILED tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3ModelIntegration::test_forward_images_integration - ValueError: Image features and image tokens do not match, tokens: 128, features: 128
FAILED tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3ModelIntegration::test_forward_queries_integration - RuntimeError: mat1 and mat2 shapes cannot be multiplied (46x4096 and 1024x320)
FAILED tests/models/qwen3/colqwen3/test_modeling_colqwen3.py::TestColQwen3ModelIntegration::test_retrieval_integration - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 66.00 MiB. GPU 0 has a total capacity of 23.56 GiB of which 38.50 MiB is free. Including non-PyTorch memory, th...
=================================================================== 5 failed, 162 passed in 233.52s (0:03:53) ====================================================================

[QuentinJGMace]

Hey @ethanlee928 thanks for looking into this ! Will check thoroughly that everything is consistent with previous codebase.

[ManuelFay]

Please check out #389 - I tried to update all that needed to be done ! I'd be glad to have a second look from someone that worked on this - and have some real testing on your usecases to assert everything is nominal.