Correct parameter setup for LLaVA-NeXT results (640, 320, 160 tokens)

[danzel-crazy]

Description

I am currently working on reproducing the LLaVA-NeXT (v1.6) results for the visual token budgets of 640, 320, and 160 tokens (average). While I have successfully reproduced the results for LLaVA-1.5, I am encountering configuration issues specifically with LLaVA-NeXT's AnyRes (multi-patch) architecture.

The Problem

When attempting to run the evaluation script with the following flag combinations:

• --reduction_ratio: 0.778
• --max_num_trunction: 640 (or 128 / 320)
• --retain_token_num_for_llava_next: (tried various values)

I frequently encounter errors such as:

• RuntimeError: selected index k out of range (IndexError)
• Error messages indicating that the requested token count exceeds the original image token length.

Since LLaVA-NeXT produces approximately 2,880 visual tokens (576 tokens × 5 patches + newline tokens), I am unclear on the intended interaction between retain_token_num_for_llava_next and max_num_trunction.

Key Questions

1. Scope of Flags: Should retain_token_num_for_llava_next be set as a global budget (across all patches) or as a per-patch limit?
2. AnyRes Logic: How does the pruning logic account for the concatenated patches and newline tokens in v1.6?
3. Reproduction Config: To achieve the 640, 320, and 160 token average reported in the paper for LLaVA-NeXT-7B, could you provide the exact mapping of reduction_ratio, max_num_trunction, and retain_token_num_for_llava_next?