Pod always failed

[zhchen-zhchen]

Hi, yesterday I upgraded from 1.5 to 1.6.0. I found status of pod of runner always shows Error status like following

predator.a3733d0d-5a00-443d-a956-f99be870c5fa-9tpns               0/1     Error 

the logs showed

{"name":"predator-runner","hostname":"predator.a3733d0d-5a00-443d-a956-f99be870c5fa-9tpns","pid":1,"level":30,"msg":"Runner exceeded test duration, sending DONE status and existing","time":"2021-02-17T04:26:48.467Z","v":0}

Besides, I can saw the results of the test during the test but after the test finished. The test status turned to Failed so that I cannot see the result anymore.

[enudler]

hi @zhchen-zhchen
can you share the test duration and arrival rate you used?

[zhchen-zhchen]

@enudler
hi, I tried multiple times with following setting.
test duration => 1 minute
qps => 50-100
parallelism => 2-3

sometimes it showed partially finished

[roychri]

I've fixed that by creating my own predator-runner docker image with the value of RUNNER_TIMEOUT_GRACE_MS changed from 300 to 90000. I did:

1. Cloned this git repo (git clone)
2. Changed RUNNER_TIMEOUT_GRACE_MS value using my preferred editor.
3. Commit (git commit) (Optional)
4. Build the docker image (docker build . -t some/image:tag
5. Push the image (docker push some/image:tag)
6. In the predator ui, go to Settings and change "Docker image name" to your image and save

I propose that this value be made dynamic so we can change it. Maybe as part of an environment variable? doesn't have to be editable from the UI I think... Just something to avoid one having to build their own image.