All values in the dataframe are "NAN" after running pycombat

[fredsamhaak]

Hello @abdelkaderbehdenna,
After running pycombat according to the pyComBat’s documentation, I got a dataframe with all "NAN" values.
Seems like everything is ok:

Would you please help me with this?
Thank you very much and have a nice day!

[julienrichardalbert]

Hello, exact same behaviour here.
While I get the pickle file example in the docs to work, any other dataset results in this error and a NaN table.
Thank you!

EDIT: After much troubleshooting, I found that altering my batches (reducing the number) changes pyCombat's behaviour. So one of my batches is probably the problem, I don't know which or why.

Will edit again with updates.

[aryoepigene]

Hi @fredsamhaak @julienrichardalbert , thank you very much for reaching out to us. We appreciate you using our tool! 😄

Unfortunately, we cannot help much without the data. Would you mind providing us a small dummy data to try to reproduce the error? Likewise, for @julienrichardalbert's comment. Altered behavior due to changes in batches is not a desirable outcome. If you can provide us with a dummy data, we can try to debug it internally.

In the meantime, if you find a solution from your end, feel free to also open a pull request with the fix. Here's the contribution guideline to help you start with the process.

[rbf22]

To reproduce the error using the documentation example:

import libraries

from combat.pycombat import pycombat
import pandas as pd
import matplotlib.pyplot as plt

prepare data

the datasets are dataframes where:

# the indexes correspond to the gene names
# the column names correspond to the sample names

Any number (>=2) of datasets can be treated

dataset_1 = pd.read_pickle("test_data/GSE18520.pickle") # datasets can also be stored in csv, tsv, etc files
dataset_2 = pd.read_pickle("test_data/GSE66957.pickle")
dataset_3 = pd.read_pickle("test_data/GSE69428.pickle")

we merge all the datasets into one, by keeping the common genes only

df_expression = pd.concat([dataset_1,dataset_2,dataset_3],join="inner",axis=1)

df_expression.at[:1, df_expression.columns] = 5.0

we generate the list of batches

batch = []
datasets = [dataset_1,dataset_2,dataset_3]
for j in range(len(datasets)):
batch.extend([j for _ in range(len(datasets[j].columns))])

run pyComBat

df_corrected = pycombat(df_expression,batch)

df_corrected.isnull().values.all()

[VEZcoding]

Hey is there any updates on this?

[rbf22]

you can compensate for this bug by adding a very small amount of random noise to your input expression array.

[britneyting]

Hi, I'm also running into this issue; all my values are NaNs. Can it be fixed without introducing noise?

i'm getting this error that might be part of the problem:
RuntimeWarning: invalid value encountered in divide s_data = np.divide((dat - stand_mean), \

[anfoss]

I encountered this issue too but the solution is a pretty easy fix.
Basically in s_data = standardise_data(dat, stand_mean, var_pooled, n_array) I was normalizing using an housekeeping gene and this led to one column with NAN variance. Not sure why but then the NAN gets propagated to everything downstream. Removing genes with 0 variance before batch correction fixed this.

[axelgschwind]

Hi,

I had exactly the same problem. It occurs if one or more rows of your input gene expression matrix are all zeroes. After I removed all of these rows, Pycombat worked well.

Cheers,
Axel