Build CI CD for gpu.

[hunghaoti]

Currently, we only test cpu version for every commit. We may need to add for gpu version (maybe running test on local machine.)

[kaihsin]

There is a way to do GPU testing for CI but I don't think we have bandwidth to do that

[yingjerkao]

We are bringing in some help.

[jysh1214]

Hi, I would like to contribute this part.

The related PR will use nvidia/cuda docker image.
What's the CUDA version we intent to use in CI?

[kaihsin]

Let's start with 11.8? ideally we want to upgrade to 12 fully

…

On Mon, Dec 16, 2024, 08:30 Alex Chiang ***@***.***> wrote: Hi, I would like to contribute this part. The related PR will use nvidia/cuda docker image. What's the CUDA version we intent to use in CI? — Reply to this email directly, view it on GitHub <#538 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCX3SKPU3JO4DRXSVUUUXD2F3IXRAVCNFSM6AAAAABTTIAKWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBVGYZTSMJTHE> . You are receiving this because you commented.Message ID: ***@***.***>

[jysh1214]

I think we are using ubuntu 20 or 22, right?

[kaihsin]

22 would be nice. Also some DevOps help on setting up GitHub CI actions of that would be great.

…

On Mon, Dec 16, 2024, 08:44 Alex Chiang ***@***.***> wrote: I think we are using ubuntu 20 or 22, right? — Reply to this email directly, view it on GitHub <#538 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFCX3SNFHNE73EBYCEKWJW32F3KKTAVCNFSM6AAAAABTTIAKWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBVGY3DSMZVGI> . You are receiving this because you commented.Message ID: ***@***.***>

[jysh1214]

@kaihsin
I try to build and test in the nvidia container (nvidia/cuda:11.8.0-devel-ubuntu22.04).

Here is my config:

cmake -B $CYTNX_BUILD -G Ninja \
  -D CMAKE_BUILD_TYPE=Debug \
  -D Python_EXECUTABLE="$(which python3.11)" \
  -D Python_INTERPRETER="$(which python3.11)" \
  -D Python3_EXECUTABLE="$(which python3.11)" \
  -D Python3_INTERPRETER="$(which python3.11)" \
  -D BLAS_LIBRARIES="$OPENBLAS_BLAS_LIB" \
  -D LAPACK_LIBRARIES="$OPENBLAS_LAPACKE_LIB" \
  -D USE_MKL=OFF \
  -D USE_CUDA=ON \
  -D USE_CUTENSOR=OFF \
  -D USE_CUQUANTUM=OFF \
  -D USE_CUTT=OFF \
  -D USE_MAGMA=OFF \
  -D RUN_TESTS=ON

The 43 tests failed. Is It expected?
Can the tests pass with only USE_CUDA enabled?

95% tests passed, 43 tests failed out of 891

Total Test time (real) = 413.69 sec

The following tests FAILED:
        640 - NetworkTest.gpu_Network_dense_no_order (Failed)
        641 - NetworkTest.gpu_Network_dense_find_optimal (Failed)
        642 - NetworkTest.gpu_Network_dense_order_line (Failed)
        643 - NetworkTest.gpu_Network_dense_specified_order (Failed)
        644 - NetworkTest.gpu_Network_dense_reuse (Failed)
        646 - NconTest.gpu_ncon_default_order (Failed)
        647 - NconTest.gpu_ncon_specified_order (Failed)
        648 - NconTest.gpu_ncon_optimal_order (Failed)
        649 - ContractTest.gpu_Contract_denseUt_optimal_order (Failed)
        650 - ContractTest.gpu_Contract_denseUt_default_order (Failed)
        651 - ContractTest.gpu_Contract_denseUt_specified_order (Failed)
        652 - ContractTest.gpu_Contract_denseUt_optimal_specified_order (Failed)
        653 - ContractTest.gpu_Contracts_denseUt_optimal_order (Failed)
        654 - ContractTest.gpu_Contracts_denseUt_default_order (Failed)
        655 - ContractTest.gpu_Contracts_denseUt_specified_order (Failed)
        656 - ContractTest.gpu_Contracts_denseUt_optimal_specified_order (Failed)
        657 - BlockUniTensorTest.gpu_Trace (Failed)
        706 - BlockUniTensorTest.gpu_group_basis (SEGFAULT)
        710 - DenseUniTensorTest.gpu_Trace (Failed)
        775 - Directsum.gpu_one_elem_tens (SEGFAULT)
        779 - Directsum.gpu_shared_axis_contains_all_tens_one_elem (SEGFAULT)
        789 - ExpM.gpu_ExpM_test (Failed)
        791 - Lanczos_Gnd.gpu_Bk_Lanczos_Gnd_test (Failed)
        792 - Arnoldi.gpu_which_LM_test (Failed)
        793 - Arnoldi.gpu_which_LR_test (Failed)
        794 - Arnoldi.gpu_which_LI_test (Failed)
        795 - Arnoldi.gpu_which_SM_test (Failed)
        796 - Arnoldi.gpu_which_SR_test (Failed)
        797 - Arnoldi.gpu_which_SI_test (Failed)
        798 - Arnoldi.gpu_mat_type_real_test (Failed)
        799 - Arnoldi.gpu_k1_test (Failed)
        800 - Arnoldi.gpu_k_max (Failed)
        801 - Arnoldi.gpu_smallest_dim (Failed)
        807 - Svd.gpu_dense_one_elem (Failed)
        808 - Svd.gpu_dense_nondiag_test (Failed)
        813 - Svd.gpu_U1_zeros_test (Failed)
        820 - Gesvd.gpu_dense_one_elem (Failed)
        821 - Gesvd.gpu_dense_nondiag_test (Failed)
        835 - linalg_Test.gpu_BkUt_Svd_truncate3 (Failed)
        836 - linalg_Test.gpu_BkUt_Qr1 (Failed)
        838 - linalg_Test.gpu_BkUt_expM (Failed)
        839 - linalg_Test.gpu_DenseUt_Gesvd_truncate (Failed)
        840 - linalg_Test.gpu_DenseUt_Svd_truncate (Failed)

BTW, all tests passed when USE_CUDA=OFF.

[IvanaGyro]

If you built on the master branch, it's expected. If you build on the dev-master branch there are fewer falling tests but still a few.

[jysh1214]

I got it. I think we should list all the failed tests and plan to fix them in the future. The list will be included in the PR.
Does that make sense?

[jysh1214]

Which branch should I submit my pr to? master or dev-master?

BTW, I cannot build dev-master successfully. Is It expected?
We can discuss this issues on another thread.

[hunghaoti]

I think dev-master is fine. Could you try to use cuda:12.2 to build?

[IvanaGyro]

These tests are expected to fail on the develop branch.

linalg_Test.gpu_BkUt_Svd_truncate3
linalg_Test.gpu_BkUt_expM
Svd.gpu_U1_zeros_test
Lanczos_Gnd.gpu_Bk_Lanczos_Gnd_test