Verification failed for the ‘run_generated with SMC’ test case

[mengllm]

@and-ivanov @benrothen

Hi,

The verification succeeds in the ‘test_generated with SMC’ test case, but it always fails in the ‘run_generated with SMC’ test case. My test hardware environment includes an A100 GPU and an Intel 6342 CPU, with a software environment of Ubuntu 22.04, CUDA 11.8, and Python 3.10.

makefile:
test_generated: checksum_launcher
cuda-memcheck ./checksum_launcher -c cuda_src_ptx.cubin -k checksum_kernel_from_data -b checksum_function_generated.bin -g 3 -t 4 -v

run_generated: checksum_launcher
./checksum_launcher -c cuda_src_ptx.cubin -k checksum_kernel_from_data -b checksum_function_generated.bin -v

test cmd:
make clean && make CAP=80 GENARGS="--with_self_modification --num_iters 1000 --with_inner_loop --num_inner_iters 5000 --num_shifts 100" run_generated

log:
./checksum_launcher -c cuda_src_ptx.cubin -k checksum_kernel_from_data -b checksum_function_generated.bin -v
checksum function loaded took 3.329e-05 s
init_kernel took 0.000902028 s
State: C=0x114ff980 S0=0x9ec7d529 S1=0x25eeb106 S2=0x714076ca S3=0xf695e42d S4=0x1127edbb S5=0x72424053 S6=0xf0934262 S7=0x0a5bf7c9 S8=0x3fe4a81f S9=0x6fcd2b6b S10=0x90c02ce1 S11=0x62fb928f S12=0x27cb6f6e S13=0xbf9cf582 S14=0xee6c2d5f S15=0xb37f402e S16=0xe40b4299 S17=0x251721ca S18=0xd5893e62 S19=0xb95b7f33 S20=0x21de2e74
Configuartion: grid_size=216 block_size=1024 binary_name=checksum_function_generated.bin cubin_name=cuda_src_ptx.cubin kernel_name=checksum_kernel_from_data
checksum_code size:153488
HtoD took 0.238831 s size:113770592
Number of SMs: 108
GPU clock rate: 1410000 kHz (unreliable on consumer GPUs: varying clock boost)
base address 0x7fe47c000000
Data pointer alignment is good!
checksum Runtime: 12.432 s
result DtoH took 0.000166865 s size:104
GPU clocks: 17448643087
Optimal clocks 17428320000
Observed 100 % of peak performance
Computing checksum on host... 100%
Verification on host took 10507.7 s
verification FAILED! dev: bc57a5cb host: 5a81f4cb

Could you provide some suggestions?

[and-ivanov]

Hi @mengllm

This test may fail sometimes as explained in the warning here https://github.com/spcl/sage/blob/main/checksum/main/readme.md#large-implementation-with-self-modifying-code-and-inner-loop-to-hide-its-impact.

Self-modifying code is relying on undocumented instruction cache invalidations and sometimes can fail the test, e.g., when checksum code aligned differently in memory. In this case, try increasing num_shifts to make the test pass.

Did you try increasing the num_shifts parameter as described? This can be seen as trading runtime overhead to minimize false positives.

[mengllm]

@and-ivanov
Yes.

Despite incrementally increasing the num_shifts parameter from 100 to 350, the ‘run_generated with SMC’ test case continues to fail.

Moreover,if the num_shifts parameter is greater than 352, the system will raise an exception "Exception: Specified memory size is too small to hold checksum code". It seems 80GB GPU memory isn't sufficient for additional checksum instructions.