is it possible to set required spacing

[jakubMitura14]

Hello thanks for sharing fantastic algorithm I see that one can add slices in one axis - Hovewer is it possible to use it to achieve fixed predefined spacing ?

I have set of prostate MRI-s with various spacings values - and I want to interpolate them into uniform spacing - can I use your tool for it?

[kevin-vmi]

Thanks for the comment.

Yes, you can. The primary purposes of the algorithm include interpolating MRI sequences to make them uniform.

In practice, you can go through your prostate MRIs in a loop. For each MRI (suppose it has some original spacing deviated from the predefined value), calculate the sampling factor by "factor = original spacing / predefined spacing." Then, you can call the SOUP_GAN function "new_MRI = SOUP_GAN (MRI, factor)."

Completion of the loop will result in a bunch of new MRIs with predefined uniform spacing.

[jakubMitura14]

Fantastic ! Thank You !

[jakubMitura14]

Ok just to be sure first it seems that only upsampling in z direction ?
and secondly Is there only upsampling possible, I mean can one achieve downsampling?

For resampling in one direction using itk - becouse my data is in MHA

spacingX_a= 0.234375
spacingX_b= 0.78125

spacingY_a= 0.234375
spacingY_b= 0.78125

spacingZ_a= 2.200000060372773
spacingZ_b= 5.000000066297424

imgB = sitk.ReadImage(path) # low resolution image

thicks_ori = sitk.GetArrayFromImage(imgA)
Z_FAC = spacingZ_b/spacingZ_a

# Call the SR interpolation tool from KevinSR
thins_gen = SOUP_GAN(thicks_ori, Z_FAC, 0)

Now let's say I would like to achieve spacing that would be mean between those two spacing's - so up sample lower resolution and down sample upper. Also I would like to make the x and y spacing uniform not only z spacing - is it possible?

Thank you for time reading it - and no matter whether it is possible or not what I try to achieve tool is fantastic and thank you for sharing it!

[kevin-vmi]

Please find my comments below:

--Ok just to be sure first it seems that only upsampling in z direction ?

z can be any direction you selected, i.e., axial, coronal, or sagittal. To achieve the tri-axial upsampling, you may want to perform the operation in the three directions, respectively.

--and secondly Is there only upsampling possible, I mean can one achieve downsampling?

For downsampling, as it is not an inverse problem, I think interpolation is sufficient. You can play with the "zoom" function in scipy.ndimage.

[jakubMitura14]

Thank you ! Hovewer I do not see how to choose direction ?

[kevin-vmi]

Sure. You can switch the dimensions of the numpy array by np.moveaxis(), for example, to move your targeted direction to the third dimension.

[jakubMitura14]

ok now I get It - Thank you !!!

[jakubMitura14]

Hello for reference if somebody would try to use the library as I described I give the code - it seems to work :)
Hovewer I regularly get the out of memory error - and out of memory on normal RAM not GPU - I suppose that amount of ram is proportional to both size of the image and to amount of cores that are working.

How can I reduce number of CPU cores used in order to reduce memory footprint?

thanks!

import pandas as pd
import SimpleITK as sitk
from KevinSR import mask_interpolation, SOUP_GAN
import os
import numpy as np
from scipy import ndimage, interpolate
from scipy.ndimage import zoom
from KevinSR import SOUP_GAN
import SimpleITK as sitk
from copy import deepcopy
import tensorflow as tf

def copyDirAndOrigin(imageOrig,spacing,data):
    image1 = sitk.GetImageFromArray(data)
    image1.SetSpacing(spacing) #updating spacing
    image1.SetOrigin(imageOrig.GetOrigin())
    image1.SetDirection(imageOrig.GetDirection()) 
    #print(image1.GetSize())
    return image1




def resample_with_GAN(path, targetSpac):
    imageOrig = sitk.ReadImage(path)
    origSize= imageOrig.GetSize()
    orig_spacing=imageOrig.GetSpacing()
    currentSpacing = list(orig_spacing)
    print(f"origSize {origSize}")
    #new size of the image after changed spacing
    new_size = tuple([int(origSize[0]*(orig_spacing[0]/targetSpac[0])),
                    int(origSize[1]*(orig_spacing[1]/targetSpac[1])),
                    int(origSize[2]*(orig_spacing[2]/targetSpac[2]) )  ]  )
    print(f"new_size {new_size}")
    anySuperSampled = False
    data=sitk.GetArrayFromImage(imageOrig)
    #supersampling if needed
    for axis in [0,1,2]:    
        if(new_size[axis]>origSize[axis]):
            anySuperSampled=True
            currentSpacing[axis]=targetSpac[axis]
            pre_slices = origSize[axis]
            post_slices = new_size[axis]
            Z_FAC = post_slices/pre_slices # Sampling factor in Z direction
            if(axis==1):
                data = np.moveaxis(data, 1, 2)
            if(axis==2):
                data = np.moveaxis(data, 0, 2)
            #Call the SR interpolation tool from KevinSR
            #print(f"thicks_ori shape {data.shape} ")

            data = SOUP_GAN(data, Z_FAC,1)
            #print(f"thins_gen shape {data.shape} ")
            if(axis==1):
                data = np.moveaxis(data, 2, 1)
            if(axis==2):
                data = np.moveaxis(data, 2, 0)            
            

    #we need to recreate itk image object only if some supersampling was performed
    if(anySuperSampled):
        image=copyDirAndOrigin(imageOrig,tuple(currentSpacing),data)
    else:
        image=imageOrig
    #copmpleting resampling given some subsampling needs to be performed
    resample = sitk.ResampleImageFilter()
    resample.SetOutputSpacing(targetSpac)
    resample.SetOutputDirection(image.GetDirection())
    resample.SetOutputOrigin(image.GetOrigin())
    resample.SetTransform(sitk.Transform())
    resample.SetDefaultPixelValue(image.GetPixelIDValue())
    resample.SetInterpolator(sitk.sitkBSpline)
    resample.SetSize(new_size)
    return resample.Execute(image)

[kevin-vmi]

Thanks for your input!

Since the model is trained on 3D data, as you can see it occupies memories. Can you please forward the error? With more details, we could try identifying possible ways to reduce memory.

[jakubMitura14]

thanks!

2022-06-28 10:17:47.872534: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-06-28 10:17:47.872729: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
1/1 [==============================] - 22s 22s/step
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
2022-06-28 10:18:20.632518: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 3623878656 exceeds 10% of free system memory.
2022-06-28 10:18:20.632562: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 3623878656 exceeds 10% of free system memory.
2022-06-28 10:18:20.632602: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 3623878656 exceeds 10% of free system memory.
2022-06-28 10:18:20.632652: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 3623878656 exceeds 10% of free system memory.
2022-06-28 10:18:20.632699: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 3623878656 exceeds 10% of free system memory.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
error: [/home/sliceruser/Slicer/bin/./python-real] exit abnormally - Report the problem.

[jakubMitura14]

error occurs in case of big diffrences in spacing ; is partially resolved by reducing number of available cores (I am executing it in docker container)

[jakubMitura14]

I made it work in let's say ad hoc manner solution is not perfect - but it is based on idea on iteratively apply your algorithm in case the difference in amount of slices is big, do you have better solution?

def useGan(data,post_slices,pre_slices):
    diffPrim = post_slices-pre_slices# always positive number if too big we will run out of memoory
    diffDone=0
    stepSize=100 # how big diffrence in number of slices it can handle at one go
    if(diffPrim<stepSize):
            return SOUP_GAN(data, post_slices/pre_slices,1)
    else:        
        for st in range(stepSize,diffPrim,stepSize):
            Z_FAC = (pre_slices+st)/(pre_slices+st-stepSize)
            diffDone=st
            data = SOUP_GAN(data, Z_FAC,1)
            
    return SOUP_GAN(data, post_slices/(pre_slices+diffDone),1)
 




#pathT2w,pathHbv,pathADC,patht2wLabel
def resample_with_GAN(path, targetSpac):
    imageOrig = sitk.ReadImage(path)
    origSize= imageOrig.GetSize()
    orig_spacing=imageOrig.GetSpacing()
    currentSpacing = list(orig_spacing)
    print(f"origSize {origSize}")
    #new size of the image after changed spacing
    new_size = tuple([int(origSize[0]*(orig_spacing[0]/targetSpac[0])),
                    int(origSize[1]*(orig_spacing[1]/targetSpac[1])),
                    int(origSize[2]*(orig_spacing[2]/targetSpac[2]) )  ]  )
    print(f"new_size {new_size}")
    anySuperSampled = False
    data=sitk.GetArrayFromImage(imageOrig)
    #supersampling if needed

    for axis in [0,1,2]:   
        if(new_size[axis]>origSize[axis]):
            anySuperSampled=True
            #in some cases the GPU memory is not cleared enough
            device = cuda.get_current_device()
            device.reset()
            currentSpacing[axis]=targetSpac[axis]
            pre_slices = origSize[axis]
            post_slices = new_size[axis]
            Z_FAC = post_slices/pre_slices # Sampling factor in Z direction
            if(axis==1):
                data = np.moveaxis(data, 1, 2)
            if(axis==2):
                data = np.moveaxis(data, 0, 2)
            #Call the SR interpolation tool from KevinSR
            #print(f"thicks_ori shape {data.shape} ")
            data =useGan(data,post_slices,pre_slices)
            #data = SOUP_GAN(data, Z_FAC,1)
            #print(f"thins_gen shape {data.shape} ")
            if(axis==1):
                data = np.moveaxis(data, 2, 1)
            if(axis==2):
                data = np.moveaxis(data, 2, 0)            
            

    #we need to recreate itk image object only if some supersampling was performed
    if(anySuperSampled):
        image=copyDirAndOrigin(imageOrig,tuple(currentSpacing),data)
        print(f"sssssupersampled {image.GetSize()}")
    else:
        image=imageOrig
    #copmpleting resampling given some subsampling needs to be performed
    resample = sitk.ResampleImageFilter()
    resample.SetOutputSpacing(targetSpac)
    resample.SetOutputDirection(image.GetDirection())
    resample.SetOutputOrigin(image.GetOrigin())
    resample.SetTransform(sitk.Transform())
    resample.SetDefaultPixelValue(image.GetPixelIDValue())
    resample.SetInterpolator(sitk.sitkBSpline)
    resample.SetSize(new_size)
    return resample.Execute(image)

[kevin-vmi]

Sorry for my late response. I was occupied by other tasks.

The approach you tried looks interesting to me. It seems you alleviate the "burden" by progressively up-sampling the data. However, I couldn't see any reason that it works in this way but fails in the original one-step setting. I am willing to see more details at the OOM time point (I assume it always happens when executing "return SOUP_GAN(data, post_slices/pre_slices,1)", is it correct?). For example, we can print the dimensions before and after, examine if Z_FAC is outside the range, and identify the specific "data" that fails so we can play with it.

[jakubMitura14]

Yes it is always about this line , I have couple busy days but I will make an experiment as you described with progressively increasing z factor until the error happens and give you back the result

Thanks!

[jakubMitura14]

Basically as I experimented the bigger the imege and the bigger the step size (in above code it was set to 100 - line 3 function use gan) the higher the probability that erroor will occur.
What is intresting it will call always the error that it tried to allocate 10% above available memory - not less no more in all cases

full error below

2022-07-07 06:35:19.206347: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:975] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-07-07 06:35:19.207768: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/cli-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Python/lib:/home/sliceruser/Slicer/bin/../lib:/home/sliceruser/Slicer/bin/../lib/Teem-1.12.0:/home/sliceruser/Slicer/bin/../lib/PythonQt:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/core:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/lib:/home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Python/lib:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-07-07 06:35:19.208024: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/cli-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Python/lib:/home/sliceruser/Slicer/bin/../lib:/home/sliceruser/Slicer/bin/../lib/Teem-1.12.0:/home/sliceruser/Slicer/bin/../lib/PythonQt:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/core:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/lib:/home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Python/lib:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-07-07 06:35:19.208303: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/cli-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Python/lib:/home/sliceruser/Slicer/bin/../lib:/home/sliceruser/Slicer/bin/../lib/Teem-1.12.0:/home/sliceruser/Slicer/bin/../lib/PythonQt:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/core:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/lib:/home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Python/lib:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-07-07 06:35:19.208550: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/cli-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0/qt-loadable-modules:/home/sliceruser/Slicer/bin/../lib/Python/lib:/home/sliceruser/Slicer/bin/../lib:/home/sliceruser/Slicer/bin/../lib/Teem-1.12.0:/home/sliceruser/Slicer/bin/../lib/PythonQt:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/core:/home/sliceruser/Slicer/bin/../lib/Python/lib/python3.9/site-packages/numpy/lib:/home/sliceruser/Slicer/bin/../bin:/home/sliceruser/Slicer/bin/../lib/Slicer-5.0:/home/sliceruser/Slicer/bin/../lib/Python/lib:/usr/local/cuda-11.0/targets/x86_64-linux/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-07-07 06:35:19.208909: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-07-07 06:35:19.209435: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
2022-07-07 06:35:22.702584: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 2189426688 exceeds 10% of free system memory.
2022-07-07 06:35:22.702662: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 2189426688 exceeds 10% of free system memory.
2022-07-07 06:35:22.702702: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 2189426688 exceeds 10% of free system memory.
2022-07-07 06:35:22.702775: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 2189426688 exceeds 10% of free system memory.
2022-07-07 06:35:22.702831: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 2189426688 exceeds 10% of free system memory.
terminate called after throwing an instance of 'std::bad_alloc'

I did tests on two machines Hovewer in both cases in docker container, dockerfile - https://github.com/jakubMitura14/forPicaiDocker

data taken from https://zenodo.org/record/6624726#.Ysk8JNJBxH4 I resample t2w, adc, hbv to the same spacing