[Better engineering]Make vllm commit consistent across benchmark tests

[yangw-dev]

Overview

due to the machine queue time, vllm benchmark jobs run benchmark based on the latest vllm can be dependent on different vllm commit,
we want this to be consistent

Details

sometimes the latest commit from vllm does not fully generate all docker imgs for all hardwares, so in order to not make tests fail, before run benchmark, we want to find the commit that has docker img binaries for all hardwares:

1.If user provides vllm_commit input → Use that commit directly (user's responsibility to ensure images exist)
2. autoL Iterate through the last 100 commits and find the first one that has Docker images available for all platforms being benchmarked

Test run:
https://github.com/pytorch/pytorch-integration-testing/actions/runs/20861018484

[huydhn]

This ALL_IMAGES_EXIST might not work in practice because I remember the Docker image for different platforms are maintained differently:

1. If HPU (not that important when comparing to CUDA) build fail, it might take a long time to get fix. And if I read the code here correctly, this would leave us stuck on an older commit (the last commit HPU was built successfully)
2. They might be build at different schedule. HPU and arm64 CPU build for example were in this bucket before where they were build daily or something (IIRC). Again, the loop needs to go back to much older commit to find a common commit.

What do you think about returning the list of commit for each of these platform independent? For example, 1 for CUDA, 1 for ROCm, etc.. IMO, this should be a good middle ground because the perf results are used per platform anyway, i.e. no ROCm vs CUDA comparison. With this, you can fold the vllm_commit into the output of generate_vllm_benchmark_matrix.py, and potentially move the Docker check logic here into the Python script too, which is usually easier to maintain than CI bash

[yangw-dev]

i see, we can def do thatm let me modify this