Errors when running ` python -m fadtk.test`

[lmxue]

After installation, and then running python -m fadtk.test without GPUs, the errors in the below figures occurred:

Here is another error when running python -m fadtk.teston a GPU

[hykilpikonna]

Hmm, a segmentation fault is rare for a python-based program... Can you provide more details on what device you're trying to run this on? (is it an ARM mac?) and what GPU are you using?

[lmxue]

Thanks for your reply. I run the script on the Linux system with NVIDIA H800.

[hykilpikonna]

Thanks for your reply. I run the script on the Linux system with NVIDIA H800.

I tried to replicate this issue on a Linux system with four NVIDIA H100 cards (which is the closest configuration I can find) and did not encounter this issue.

Logs:

My machine specifications:

In my test run, I used the following commands to set up the environment and run the tests:

# Install miniconda per official documentation
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh
source ~/miniconda3/bin/activate

# Install python 3.11
conda install python=3.11

# Install FADTK
pip install fadtk

# Run test
python -m fadtk.test

Since this did not replicate your issue, please provide more details on your system environment (one convenient way to gather system information would be using hyfetch or fastfetch). Please also provide more details about your python environment (specifically, the outputs of python -VVV and python -m pip freeze)

[simonrouard]

Hi @lmxue I have been facing the same issues and I think that I found a solution to the problem of the imaginary parts.

It seems that the numpy.linalg.eigvals function has bugs for numpy==1.23.5 that are solved when I upgrade to numpy==1.26.2

Here is a code snippet to spot the bug:

import numpy as np

cov = np.random.randn(1024, 20)  # Covariance matrix for the first Gaussian
cov = cov @ cov.T  # Make it positive semi-definite

eigenvalues = np.linalg.eigvals(cov)
print(eigenvalues)

With numpy==1.23.5 I get negative eigenvalues or even eigenvalues with non-zero imaginary part. With numpy==1.26.2 I get non-negative eigenvalues.

As well there is a bug for MERT models (see huggingface/transformers#36134) and switching to transformers==4.47.1 did solve the problem for me

Hope it will help!