N4BiasFieldCorrection step repeatedly fails for certain subjects during bold preprocessing

[bcb225]

Hi, I'm encountering a recurring problem when running deepprep. For some subjects, the pipeline stops partway through, and the logs consistently show a crash at the n4_correct step during bold preprocessing.
For example, in the case of sub-GA00940, the process fails with this error:
FileNotFoundError: No such file or directory '/output/WorkDir/.../n4_correct/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_sbref_corrected.nii.gz' for output 'output_image'
I haven't tried manually cleaning up the work directory, but this issue seems to happen intermittently across subjects, always at the same stage. Any idea what might be causing this or how to prevent it?
Thanks in advance.

Below is the error log.

crashfile: /output/WorkDir/bold_preprocess/log/crash-20250619-140104-root-n4_correct-66ccdc7a-3667-4e41-a76e-cd9ba69458f6.txt
250619-14:01:04,961 nipype.workflow INFO:
***********************************
Traceback (most recent call last):
File "/opt/DeepPrep/deepprep/nextflow/bin/bold_preprocess.py", line 316, in
result = workflow.run()
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/pipeline/plugins/linear.py", line 82, in run
raise error from cause
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/pipeline/plugins/linear.py", line 47, in run
node.run(updatehash=updatehash)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node n4_correct.

Cmdline:
N4BiasFieldCorrection --bspline-fitting [ 200 ] -d 3 --input-image /input/sub-GA00940/ses-20230428/func/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_sbref.nii.gz --output sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_sbref_corrected.nii.gz -r --weight-image /output/WorkDir/bold_preprocess/sub-GA00940_wf/rest_wf/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_wf/bold_wf/bold_fit_wf/enhance_and_skullstrip_bold_wf/fix_header/tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_fixhdr.nii.gz
Stdout:

Stderr:

Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
setattr(outputs, key, val)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'output_image' trait of a _FixN4BiasFieldCorrectionOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/output/WorkDir/bold_preprocess/sub-GA00940_wf/rest_wf/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_wf/bold_wf/bold_fit_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_sbref_corrected.nii.gz' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 398, in run
runtime = self._post_run_hook(runtime)
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/interfaces/mixins/fixheader.py", line 127, in _post_run_hook
outputs = self.aggregate_outputs(runtime=runtime).get_traitsfree()
File "/opt/conda/envs/deepprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/output/WorkDir/bold_preprocess/sub-GA00940_wf/rest_wf/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_wf/bold_wf/bold_fit_wf/enhance_and_skullstrip_bold_wf/n4_correct/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_sbref_corrected.nii.gz' for output 'output_image' of a FixN4BiasFieldCorrection interface

Work dir:
/output/WorkDir/nextflow/9d/7aff89c4b59175d347d6013f252504

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out

[NingAnMe]

First, You can check if this file is normal: /output/WorkDir/bold_preprocess/sub-GA00940_wf/rest_wf/sub-GA00940_ses-20230428_task-rest_dir-PA_run-1_wf/bold_wf/bold_fit_wf/enhance_and_skullstrip_bold_wf/fix_header/tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_fixhdr.nii.gz
This may be caused by an error in the process of generating the mask.

[bcb225]

Thank you very much for the quick and helpful response.

I checked the file you mentioned (tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_fixhdr.nii.gz)—it doesn’t appear to be corrupted or missing. However, upon visual inspection, I noticed that the mask covers only about 80% of the cerebellum in the T1 reference, with the lower portion left out. I’m wondering whether this partial coverage might be contributing to the N4BiasFieldCorrection failure.

Currently, I’m running the pipeline with the --ignore_error option to allow processing to continue for the remaining subjects. That said, I do see some failed cases—for example:
[a7/fea873] bold_wf:bold_pre_process (sub-BI05874_ses-20230525_task-rest_dir-PA_run-1) [ 39%] 22 of 56, cached: 3, failed: 3

I’m not entirely sure whether these failures are occurring at the same step, but they seem to be consistent for a few subjects.

In such cases, is there a recommended way to re-run only the failed subjects later? Or do these failures typically indicate deeper issues with the specific input data that are unlikely to be resolved through reprocessing?

I’d really appreciate any advice or best practices for handling these situations. Thanks again!

[NingAnMe]

After the run is completed, you can check the report.html file in the QC directory. It will show which sub's data had errors during the run, and also which step had errors.
I've only encountered this problem once before. The main reason is that the mask file generated by the AI_mask program has structural missing and misalignment. AI_mask is an integrated ANTs program, so there is currently no way to solve it.
I apologize for this.

[bcb225]

Thank you very much for your kind and detailed explanation. Thanks to your clarification, I now understand this issue as a limitation of the current preprocessing pipeline. Based on your guidance, I will consider excluding the affected subjects from further analysis.

At the moment, I am proceeding with the --ignore_error option. May I kindly confirm whether, in this mode, the remaining subjects will be processed without issue?

Additionally, I have a separate question.
Our current setup includes 64 CPUs, 252 GB of RAM, and four RTX 3090 GPUs. Given this environment, could you please provide a rough estimate of how long it might take to preprocess approximately 4,000 fMRI runs from 1,000 participants?
From what I have observed during processing, GPU usage appears to be minimal, and CPU performance seems to be the main bottleneck. Would you agree with this assessment?

Thank you again for your support and valuable insights.

[bcb225]

Hi again,

I'm currently re-running the pipeline, but I'm encountering a notably high failure rate. For example, the current log shows:

[a2/e46cfc] bold_wf:bold_pre_process (sub-LA08975_ses-20230102_task-rest_dir-PA_run-1) [ 45%] 401 of 876, failed: 84

Following your previous advice, I reviewed the problematic mask file (tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_fixhdr.nii.gz) and confirmed that the issue indeed relates to incorrectly generated masks.

Could you please clarify if these mask-generation issues originate from the skull-stripping and bias-field correction step during structural MRI preprocessing? If so, is it possible that providing fieldmaps might inadvertently contribute to these issues? To clarify, our dataset structure looks like this:

├── anat
│   ├── sub-CA06239_ses-20230712_run-1_T1w.nii.gz
│   └── sub-CA06239_ses-20230712_run-1_T2w.nii.gz
├── fmap
│   ├── sub-CA06239_ses-20230712_dir-AP_run-1_epi.nii.gz (json IntendedFor T1, T2)
│   ├── sub-CA06239_ses-20230712_dir-AP_run-2_epi.nii.gz (json IntendedFor run-1 AP bold)
│   └── sub-CA06239_ses-20230712_dir-AP_run-3_epi.nii.gz (json IntendedFor run-2 AP bold)
└── func
    ├── sub-CA06239_ses-20230712_task-rest_dir-AP_run-1_bold.nii.gz
    └── sub-CA06239_ses-20230712_task-rest_dir-AP_run-2_bold.nii.gz

Could inconsistencies in the IntendedFor metadata and file-naming conventions potentially cause these preprocessing errors, or might the presence of fieldmaps themselves complicate the process?

Given these challenges, would you advise potentially omitting fieldmaps entirely from preprocessing? If we choose to omit fieldmaps, can we still expect robust preprocessing results for the BOLD signals?

Additionally, could you confirm whether the pipeline relies strictly on the IntendedFor field in JSON metadata, or if file-naming conventions also influence how fieldmaps are utilized?

Thank you very much for your continued support and valuable guidance. I greatly appreciate your insights on these points!

[NingAnMe]

This issue is not affected by the field map. It is mainly caused by the stability of the AI_mask program provided by ANTs when processing the original BOLD data.

[bcb225]

Thank you very much for your clear and detailed explanation.

Initially, I suspected that the issue might be due to the file itself. To verify this, I processed the problematic subject’s images using fMRIPrep, and found that the file tpl-MNI152NLin2009cAsym_res-01_label-brain_probseg_trans_fixhdr.nii.gz was generated correctly without any issues.

Based on this observation, I believe there may be some differences between DeepPrep and fMRIPrep, particularly in how the pipelines handle mask generation during the BOLD image preprocessing step. For other users encountering similar issues with DeepPrep, utilizing fMRIPrep as an alternative or complementary tool might be a viable solution.

Although I am not fully familiar with all the details of the pipeline, from my brief review, apart from the core AI modules, DeepPrep and fMRIPrep appear quite similar in structure. Nevertheless, it’s interesting to note that such differences can occur.

I sincerely appreciate your efforts in developing such a helpful tool, and I’m also very grateful for your prompt and thoughtful responses to user inquiries.

Thank you again!