Encrypted computation time reported by the harness is much less than the server reported times

[code-perspective]

See https://github.com/fhe-benchmarking/BERT/blob/main/measurements/small/results-1.json

[code-perspective]

@desilo-doyoung - If you have time, please take a look

[desilo-doyoung]

In my setup, four jobs were running the workload in parallel, so the wall-clock time for encrypted computation was roughly one-quarter of the total compute time across all jobs(#40 ). However, the server currently adds together the computation time from all four jobs and reports that accumulated value. Should I change the server to report wall-clock time instead, or adjust the other values to use accumulated compute time for consistency?

[code-perspective]

It would be good to record wall-clock time. Please update the code accordingly. we want our benchmarks to be as close to real-world metrics as possible.

(Optionally you can also record sequential time in the server_reported_steps under a different name.)

[desilo-doyoung]

For the server-side report, would it be possible to divide the accumulated computation/IO time by the worker count? With batched execution, there is no straightforward way to separate computation time from IO time in the wall-clock measurement.

Edit: In fhe-benchmarking/fhe-benchmarking.github.io#11, the reference reports a total runtime of 72.15 hours, while the GPU version reports 3.317 hours. Assuming the reference measurement did not use parallel workers—roughly 40+ minutes per sample × 100 samples ÷ 60 minutes ≈ 72 hours—the actual speedup could have been around 11× or even 5×, rather than approximately 22×, depending on the machine specifications. I am concerned that reporting wall-clock time this way may be more confusing or misleading than reporting the aggregate compute time.

[andreea-alexandru]

IMO, dividing the total time by the number of workers is not a good measure, since it's only real when there is zero idle time and perfectly balanced work. Reporting several metrics as part of server reported steps might be the way to go.

[code-perspective]

For the server-side report, would it be possible to divide the accumulated computation/IO time by the worker count? With batched execution, there is no straightforward way to separate computation time from IO time in the wall-clock measurement.

I would assume the wall-clock time for CPU would be close to the (total time / worker count) since server_reported_compute_time = sum of individual worker times.

Edit: In fhe-benchmarking/fhe-benchmarking.github.io#11, the reference reports a total runtime of 72.15 hours, while the GPU version reports 3.317 hours.
I believe the reference time will be close to 18 hours as reported in the harness timing.

Lets find a resolution to this. Here are two options I can think of (Please suggest alternatives):

1. Measure wall-clock time for the batch and report that for a single run to unblock the next steps. You can submit a more optimized batched implementation in the future.
2. Keep only the single inference table for the initial release. (Since we have confirmed from the small run that the quality is > 80%)

@desilo-doyoung - what do you think?

[desilo-doyoung]

I am not sure If I am understanding this correctly, but does option 1 mean that we should measure a separate computation time in https://github.com/fhe-benchmarking/BERT/blob/main/submissions/server_encrypted_compute.py (possibly from Line 36 to Line 93) and use that as the server reported values(for both Encrypted computation and Total)? Or does wall-clock time for the batch refer to the value currently recorded as Timing.Encrypted computation in the measurements?

[code-perspective]

I am not sure If I am understanding this correctly, but does option 1 mean that we should measure a separate computation time in https://github.com/fhe-benchmarking/BERT/blob/main/submissions/server_encrypted_compute.py (possibly from Line 36 to Line 93) and use that as the server reported values(for both Encrypted computation and Total)?

Yes option 1 suggestion is to measure the wall clock time from line 36 to 93 as report that as Server_Reported.Total and potentially Server_Reported.Encrypted_computation unless there is a better way. I suggest also reporting Server_Reported.Total_paused, etc (even if they are not displayed in tables) so you can derive any additional timing if needed.

Or does wall-clock time for the batch refer to the value currently recorded as Timing.Encrypted computation in the measurements?

Yes, my comment referring to wall-clock time refers to the value currently recorded as Timing.Encrypted computation in the measurements (which is measured by the harness). This measures even the IO time for loading keys and ciphertexts. The Server_Reported.* times will only measure the time encrypted computation and separate out IO time for loading weights where possible.

[desilo-doyoung]

In that case, it would be easy for me to update the code, but I assume we would need to run another small batch of measurements. Would that be a problem on your end, @code-perspective?

Also, if we go with that approach, would it be acceptable for the measurements to report only the wall-clock time without showing the number of parallel workers used? Or do we still need to include the additional metrics, even though they are not currently used in the reports on the website, @andreea-alexandru?

If it is difficult to decide right now, I am happy to proceed with Option 2.

[code-perspective]

In short: lets go with option 2 but update the code so I can run the small batch and update the tables after.

In that case, it would be easy for me to update the code, but I assume we would need to run another small batch of measurements. Would that be a problem on your end, @code-perspective?

Please update the code. I can run the small batch and update the tables in a couple of days.

Also, if we go with that approach, would it be acceptable for the measurements to report only the wall-clock time without showing the number of parallel workers used? Or do we still need to include the additional metrics, even though they are not currently used in the reports on the website, @andreea-alexandru?

Just wall-clock time is fine without additional metrics.

[code-perspective]

I will leave this issue open until the batch measurements are updated. this might take a week and should not be blocking any timelines. let me know otherwise