Skip to content

Tutorial_compile_llama2_with_mlc_llm.ipynb dose not work #32

@chosen-ox

Description

@chosen-ox

I follow this notebook in colab with T4 GPU.
There are several problems:

  • Now the version of cuda is 12.2 instead of 11.8. mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 should be replaced by mlc-ai-nightly-cu122 mlc-chat-nightly-cu122
  • The prebuilt_libs does not work for colab T4 GPU (Get a error CUDA_ERROR_NO_BINARY_FOR_GPU). I compiled the model lib on colab with command !mlc_chat compile ./dist/Llama-2-7b-chat-hf-q4f32_1-MLC/mlc-chat-config.json --device cuda -o dist/Llama-2-7b-chat-hf-q4f32_1-cuda.so and it works to me. Maybe consider update the prebuild_libs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions