Gpu-enabled cpu runs are slow

[oir]

If I build mnist.cu.cpp with -DGINN_ENABLE_GPU=0 and run, single epoch takes ~3.5s. If I build it with -DGINN_ENABLE_GPU=1 and run using the same docker instance (no gpu, falling back to cpu), single epoch takes ~30s.

Could be:

• Optimizer flags are not properly set / sufficient
  • Although, I verified with verbose build that pxtas and compiler options are set to at least O3, is there anything else missing?
• nvcc is doing a poor job somehow
  • Maybe test using cuda > 11.1

[oir]

Adding missing -march=native -mtune=native into Gpu flags reduced it to ~24s. Adding linkage to Threads reduced it to slightly under 24s but it might not be significant.

Other things I tried:

• Ensure same host compiler used for both
• Ensure same optimization flags used for both (switched from 03 to 0fast for host side in Gpu)
• Use a recent nvidia/cuda docker image with cuda 11.8.