I follow this notebook in colab with T4 GPU.
There are several problems:
- Now the version of cuda is 12.2 instead of 11.8.
mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 should be replaced by mlc-ai-nightly-cu122 mlc-chat-nightly-cu122
- The prebuilt_libs does not work for colab T4 GPU (Get a error CUDA_ERROR_NO_BINARY_FOR_GPU). I compiled the model lib on colab with command
!mlc_chat compile ./dist/Llama-2-7b-chat-hf-q4f32_1-MLC/mlc-chat-config.json --device cuda -o dist/Llama-2-7b-chat-hf-q4f32_1-cuda.so and it works to me. Maybe consider update the prebuild_libs.
I follow this notebook in colab with T4 GPU.
There are several problems:
mlc-ai-nightly-cu118 mlc-chat-nightly-cu118should be replaced bymlc-ai-nightly-cu122 mlc-chat-nightly-cu122!mlc_chat compile ./dist/Llama-2-7b-chat-hf-q4f32_1-MLC/mlc-chat-config.json --device cuda -o dist/Llama-2-7b-chat-hf-q4f32_1-cuda.soand it works to me. Maybe consider update the prebuild_libs.