Incorrect Version Files Pulled When Running in Batch on ifarm

[rmbrunner]

As of version 2.11.0, when running using the script for the ifarm (e.g. Using$MCWRAPPER_CENTRAL/gluex_MC.py /path/to/config 42550 10000 batch=2) the script seems to try pull v2.9.0 files which no longer exist and thus slurm/swif2 fails. See attached image for proof.

This issue has been reproduced by Alex Austregesilo using my config here /work/halld/home/rbrunner/omegaKpLambda/MC/swif2/MC.config and I have heard of similar issues from at least one other.

[s6pepaul]

Thanks for reporting this. I will take a look.

[T-Britton]

your ENV file contains a direct reference to So MCwrapper attempts to pull those files. Please update:
ENVIRONMENT_FILE=/group/halld/www/halldweb/html/halld_versions/recon-2018_01-ver02_32.xml

I recommend making a custom environment file based on the one above and swap the MCwrapper version. Reference this custom file in your MC.config and you should be good to go

[rmbrunner]

I will make the custon env for now, but should we not update all the provided env files to pull v2.11.0 given it is likely people would be using them if they are using data from GlueX-1 (and thus want MC that "matches" it)?

[T-Britton]

That would be up to those managing build scripts. The problem with swapping out other version sets is it would no longer represent the exact versions in use at that time (maybe an MCwrapper bug in 2.9.0). Better might be to ensure that the versions of MCwrapper are kept, but there are still challenges with that.

MCwrapper is a bit of a stranger beast as it sit above the other packages and there is little reason to want to use an older version of MCwrapper. Hence, it is included as a sign post to guide users and experts as to the totality of software at the time.

[rmbrunner]

I see. But how does this explain it working fine when running interactively? Alex was using the same config and saw no issue when not running it as a swif2 job (i.e. it was pulling the correct files in interactive mode).

[aaust]

With the recent changes to MCWrapper, we will no longer need to update the "recon-compatible" version sets to pull in the latest changes to halld_sim and hdgeant4. I suggest, we treat MCWrapper the same way.

[s6pepaul]

While looking at this again I noticed that you use a version set that was built on CentOS7 but not Alma9. So the issue isn't so much that you are pulling old MCWrapper files. It is more the issue that you ask for a genr8,mcsmear,... build that does not exist.
@aaust did you try this MC.config in a container?

The solution should be to appropriately make use of the new OS flags and set them all to Centos7. C.f. the MC.config file in the example folder.
I can't try it out right now but I am hopeful that this might fix the problem. I will try it tomorrow morning (my time) if I don't hear from you before that.

[aaust]

@s6pepaul When testing this MC.config file in batch jobs, it uses the container and the specified xml file. The interactive test used the current default environment.

We can't build these recon-compatible version sets (e.g. recon-2018_01-ver02_32.xml) on Alma9, and I don't believe @rmbrunner intends to do use a dependency on an old MCWrapper release. Going forward, we should use the xml file only for halld_recon which is run in the CentOS7 container, while all other packages (hdgent4, halld_sim and mcwrapper) should be taken from the latest release and run in the alma9 container.

[s6pepaul]

@aaust So you suggest to use the old recon version xml in the reconstruction step and the latest version.xml for all others? That makes sense I think. But then we should also have a way to change that if needed. Need a few more options for this. Especially on the webform.

Regarding the issue at hand. When I add the OS options and specify CentOS7 (which was used to build the requested version sets), the errors that were reported disappear. That being said, the job still does not run due to some issue in genr8. I am currently following up on this.

[acschick]

I have the same exact problem. I tried @T-Britton 's solution of copying recon-2018_01-ver02_32.xml and making a custom env file with MCWrapper v2.11.0 swapped in to no avail.

=======SOFTWARE USED=======
MCwrapper version v2.11.0
MCwrapper location /group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/gluex_MCwrapper/gluex_MCwrapper-v2.11.0
Streaming via xrootd? 0 Event Count: 441934
BC /usr/bin/bc
python /usr/bin/python
file:/work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/1801_0DEG_DBLRAD_newHDDM.hddm: Command not found.
hdgeant4: Command not found.
mcsmear: Command not found.

The MCwrapper location now matches the version used, but you still end up with the same result.
I also attempted to use RECON_OS=centos7 and ANA_OS=centos7 also with the same end result.

@rmbrunner did making a custom environment file work for you?

[T-Britton]

so the file:/work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/1801_0DEG_DBLRAD_newHDDM.hddm: Command not found. might be spurious or an MCWrapper bug hdgeant4: Command not found. mcsmear: Command not found. however indicates that the versions of hdsim are not actually there. This is probably a bigger problem. Can you post the version xml and where build_scripts is trying to find these packages? Thomas Britton Staff Scientist in Scientific Computing Jefferson Lab x7624

…

________________________________ From: Andrew Schick ***@***.***> Sent: Tuesday, August 5, 2025 10:32 AM To: JeffersonLab/gluex_MCwrapper ***@***.***> Cc: Thomas Britton ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [JeffersonLab/gluex_MCwrapper] Incorrect Version Files Pulled When Running in Batch on ifarm (Issue #147) [https://avatars.githubusercontent.com/u/8531932?s=20&v=4]acschick left a comment (JeffersonLab/gluex_MCwrapper#147)<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JeffersonLab_gluex-5FMCwrapper_issues_147-23issuecomment-2D3155481575&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=0Csgt7subKaoCAILwWcWrFwWtGVqdiMR-10XVe7LU_iHRp-n9Rvxb34n9pi5pc3J&s=U0jRAk-rAxdM6CIAfAOGmi_eMhF--Sa7H9tKO7Ymw0U&e=> I have the same exact problem. I tried @T-Britton<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_T-2DBritton&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=0Csgt7subKaoCAILwWcWrFwWtGVqdiMR-10XVe7LU_iHRp-n9Rvxb34n9pi5pc3J&s=Xo1GfCNDTGYWnGIX6KJFRHqYBiijGZfauuNj-NFpOj8&e=> 's solution of copying recon-2018_01-ver02_32.xml and making a custom env file with MCWrapper v2.11.0 swapped in to no avail. =======SOFTWARE USED======= MCwrapper version v2.11.0 MCwrapper location /group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/gluex_MCwrapper/gluex_MCwrapper-v2.11.0 Streaming via xrootd? 0 Event Count: 441934 BC /usr/bin/bc python /usr/bin/python file:/work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/1801_0DEG_DBLRAD_newHDDM.hddm: Command not found. hdgeant4: Command not found. mcsmear: Command not found. The MCwrapper location now matches the version used, but you still end up with the same result. I also attempted to use RECON_OS=centos7 and ANA_OS=centos7 also with the same end result. @rmbrunner<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rmbrunner&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=0Csgt7subKaoCAILwWcWrFwWtGVqdiMR-10XVe7LU_iHRp-n9Rvxb34n9pi5pc3J&s=7s_-hEC3kxEcFtlT33hd-4y_9zcNpqh2SAJFkD909rE&e=> did making a custom environment file work for you? — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JeffersonLab_gluex-5FMCwrapper_issues_147-23issuecomment-2D3155481575&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=0Csgt7subKaoCAILwWcWrFwWtGVqdiMR-10XVe7LU_iHRp-n9Rvxb34n9pi5pc3J&s=U0jRAk-rAxdM6CIAfAOGmi_eMhF--Sa7H9tKO7Ymw0U&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFRO2BBJE2DXFWQAKGWTDK33MC6ARAVCNFSM6AAAAACAPW7P5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNJVGQ4DCNJXGU&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=0Csgt7subKaoCAILwWcWrFwWtGVqdiMR-10XVe7LU_iHRp-n9Rvxb34n9pi5pc3J&s=S478ciG83ZP8w4ynrWepChLolLcwJoUtxmk4Fva_YUA&e=>. You are receiving this because you were mentioned.

[acschick]

My copied version xml = /work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/MCWrapper/recon-2018_01-ver02_32.xml

singularity exec --bind /u,/group,/w,/work,/scigroup,/volatile,/scratch,/u/home,/usr/local,/farm_out /cvmfs/singularity.opensciencegrid.org/jeffersonlab/gluex_prod:v1 tcsh

source /group/halld/Software/build_scripts/gluex_env_jlab.csh /work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/MCWrapper/recon-2018_01-ver02_32.xml

which hdgeant4
/group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/hdgeant4/hdgeant4-2.26.0/bin/Linux-g++/hdgeant4

which mcsmear
/group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/halld_sim/halld_sim-4.31.0^at0121/Linux_CentOS7-x86_64-gcc4.8.5-cntr/bin/mcsmear

[T-Britton]

to be really clear: which hdgeant4 /group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/hdgeant4/hdgeant4-2.26.0/bin/Linux-g++/hdgeant4 which mcsmear /group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/halld_sim/halld_sim-4.31.0^at0121/Linux_CentOS7-x86_64-gcc4.8.5-cntr/bin/mcsmear seem to be contradicted in the logs. Is this the case? Thomas Britton Staff Scientist in Scientific Computing Jefferson Lab x7624

…

________________________________ From: Andrew Schick ***@***.***> Sent: Tuesday, August 5, 2025 10:48 AM To: JeffersonLab/gluex_MCwrapper ***@***.***> Cc: Thomas Britton ***@***.***>; Mention ***@***.***> Subject: [EXTERNAL] Re: [JeffersonLab/gluex_MCwrapper] Incorrect Version Files Pulled When Running in Batch on ifarm (Issue #147) [https://avatars.githubusercontent.com/u/8531932?s=20&v=4]acschick left a comment (JeffersonLab/gluex_MCwrapper#147)<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JeffersonLab_gluex-5FMCwrapper_issues_147-23issuecomment-2D3155538831&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=9eFpzU3UbYgYQRdXDV_W-ybyqzFvhkpnAaV04_vaS29qiZaaB9PRgx-ko8yz7sPb&s=VqrTcNN6A-7nIuxJqXSJL2qBy7yhYryXoVqi5vyit_A&e=> My copied version xml = /work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/MCWrapper/recon-2018_01-ver02_32.xml singularity exec --bind /u,/group,/w,/work,/scigroup,/volatile,/scratch,/u/home,/usr/local,/farm_out /cvmfs/singularity.opensciencegrid.org/jeffersonlab/gluex_prod:v1 tcsh source /group/halld/Software/build_scripts/gluex_env_jlab.csh /work/halld/home/acschick/channels/epemmissprot/newRBHG/output/PS4/dblrad_FFN/GlueX_1801_ee_Berlin_v110/1801_0DEG/MCWrapper/recon-2018_01-ver02_32.xml which hdgeant4 /group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/hdgeant4/hdgeant4-2.26.0/bin/Linux-g++/hdgeant4 which mcsmear /group/halld/Software/builds/Linux_CentOS7-x86_64-gcc4.8.5-cntr/halld_sim/halld_sim-4.31.0^at0121/Linux_CentOS7-x86_64-gcc4.8.5-cntr/bin/mcsmear — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_JeffersonLab_gluex-5FMCwrapper_issues_147-23issuecomment-2D3155538831&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=9eFpzU3UbYgYQRdXDV_W-ybyqzFvhkpnAaV04_vaS29qiZaaB9PRgx-ko8yz7sPb&s=VqrTcNN6A-7nIuxJqXSJL2qBy7yhYryXoVqi5vyit_A&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFRO2BERXYZSFPPWWR7I2XT3MC73LAVCNFSM6AAAAACAPW7P5GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNJVGUZTQOBTGE&d=DwMFaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=ccFffx721N71hPpKcJGvJIqY9RM4gBTuzp9ir7rze5Q&m=9eFpzU3UbYgYQRdXDV_W-ybyqzFvhkpnAaV04_vaS29qiZaaB9PRgx-ko8yz7sPb&s=CL95ESZZWKD57RYbtxsGntM6aKiYnG2oJamF68at1sw&e=>. You are receiving this because you were mentioned.

[acschick]

You mean where it says "Local OS Linux_Alma9-x86_64-gcc11.5.0-cntr"? I don't see any references specifically to the location of hdgeant4 or mcsmear in the logs.
Is it just that swif2cont no longer defaults to centos7? In my singularity container cat /etc/os-release shows CentOS Linux 7, which is probably why things work interactively there.