Skip to content

Add aya_vision#4720

Open
Minestar6 wants to merge 1 commit into
PaddlePaddle:developfrom
Minestar6:feature/add_ayavision
Open

Add aya_vision#4720
Minestar6 wants to merge 1 commit into
PaddlePaddle:developfrom
Minestar6:feature/add_ayavision

Conversation

@Minestar6

Copy link
Copy Markdown
Contributor

Add aya_vision

添加了aya_vision模型的PaddlePaddle 实现,包含模型代码、配置文件、单元测试。

对齐验证

精度对齐

前 20 个 token 的 logits 对齐结果如下:

  • mean diff: 1.119111e-05
  • max diff: 1.258850e-04

训练loss对齐

训练数据

文本数据

  • GSM8K SFT 数据
  • 训练步数:300 steps

多模态数据

训练结果

文本结果
2_ayavision
2_ayavision_diff

多模态结果
3_ayavision
3_ayavision_diff

@Minestar6
Minestar6 force-pushed the feature/add_ayavision branch from 6019e2f to a43b286 Compare July 19, 2026 09:01

@risemeup1111 risemeup1111 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已复查当前提交,CI 通过,但仍有会影响 AyaVision 生成路径和权重绑定语义的阻塞问题;另外有一个 processor 的低优先级问题需要作者回复。具体细节已放在行级评论中,请先按评论处理后再继续推进。

Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.

**kwargs,
)
if is_first_iteration or not kwargs.get("use_cache", True) or past_key_values is None:
model_inputs["pixel_values"] = pixel_values

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级:P1
处理要求:请针对该评论修复并提交新的 commit。

这里在首个 generation step 会继续传入 pixel_values,但当前类没有覆盖 PaddleFormers 实际调用的 expand_inputs_for_generation。默认实现只会展开 input_ids/attention_mask/position_ids 等文本 batch 张量,不会展开 pixel_values;当 num_beams>1num_return_sequences>1 时,input_ids 中的图像 token 数会被放大,而视觉特征仍是原 batch 数量,随后 _merge_image_features() 会因为 n_image_tokens != flat_image_features.shape[0] 抛出不匹配错误。这个 PR 新增的 beam 生成测试路径也会触发该问题。

请为 AyaVision 增加与当前生成入口匹配的视觉输入展开逻辑;如果继续支持 crop_to_patches=True,需要按样本的 tile 数切分后重复,而不是直接把整批 pixel_values 当作文本 batch gather:

def expand_inputs_for_generation(self, input_ids, expand_size, attention_mask=None, **model_kwargs):
    source_input_ids = input_ids
    input_ids, model_kwargs = super().expand_inputs_for_generation(
        input_ids, expand_size=expand_size, attention_mask=attention_mask, **model_kwargs
    )
    pixel_values = model_kwargs.get("pixel_values")
    if expand_size > 1 and pixel_values is not None:
        # 根据 source_input_ids 中每个样本的图像 token 数和每个 tile 的 token 数计算切片长度,
        # 然后把每个样本对应的 pixel_values 切片 repeat expand_size 次。
        model_kwargs["pixel_values"] = repeat_visual_samples(pixel_values, lengths, expand_size)
    return input_ids, model_kwargs

if hasattr(config, attr):
setattr(config.text_config, attr, getattr(config, attr))
self.lm_head = GeneralLMHead(config.text_config)
self.criterion = AyaVisionCriterionLayer(config)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 优先级:P1
处理要求:请针对该评论修复并提交新的 commit。

AyaVisionConfig/Cohere2Config 默认 tie_word_embeddings=True,并且本类声明了 _tied_weights_keys,但普通构造路径只创建 lm_head 后没有调用 tie_weights();目前只有 from_pretrained() 后置调用会绑定。这样 AyaVisionForConditionalGeneration(config) 用于从配置训练、单测或保存前推理时会保持未绑定输出头,和配置/检查点语义不一致。建议在初始化完 lm_head/criterion 后按配置绑定:

Suggested change
self.criterion = AyaVisionCriterionLayer(config)
self.criterion = AyaVisionCriterionLayer(config)
if config.tie_word_embeddings:
self.tie_weights()

Comment on lines +75 to +77
self.image_ids = tokenizer.convert_tokens_to_ids(
[img_patch_token, tile_token, tile_global_token, start_of_img_token, end_of_img_token]
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 优先级:P2
处理要求:请针对该评论进行回复(同意并已修改请回复 Done,不同意请说明理由)。

这里把完整的图像特殊 token 列表保存到了 image_ids,但仓库内没有读取这个属性;下面 return_mm_token_type_ids=True 调用的 create_mm_token_type_ids() 依赖的是 image_token_ids 属性。结果是 TILE_GLOBAL、起止图像 token 等不会按这里的列表参与 mm_token_type_ids 标记,开启该选项时多模态 token 类型会漏标。请改成 helper 实际使用的属性名,并补一个覆盖 return_mm_token_type_ids=True 的 processor 测试。

Suggested change
self.image_ids = tokenizer.convert_tokens_to_ids(
[img_patch_token, tile_token, tile_global_token, start_of_img_token, end_of_img_token]
)
self.image_token_ids = tokenizer.convert_tokens_to_ids(
[img_patch_token, tile_token, tile_global_token, start_of_img_token, end_of_img_token]
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants