Archive ND Simulation merged tar files to tape

[twalton-fnal]

Please archive the following datasets to tape:
https://metacat.fnal.gov:9443/dune_meta_prod/app/gui/datasets?selection=namespace%3Aneardet-lar-tms-sand&namematch=merged&submit=Submit

1. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_ndlarfid_sanddrift_merged_tar
2. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_ndlarfid_sanddrift_nueelastic_merged_tar
3. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_ndlarfid_sandstt_merged_tar
4. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_rockantindlarfid_sanddrift_merged_tar
5. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_rockantindlarfid_sandstt_merged_tar
6. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_ndlarfid_sanddrift_ghep_merged_tar
7. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_ndlarfid_sanddrift_nueelastic_ghep_merged_tar
8. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_ndlarfid_sandstt_ghep_merged_tar
9. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_rockantindlarfid_sanddrift_ghep_merged_tar
10. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_rockantindlarfid_sandstt_ghep_merged_tar
11. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_ndlarfid_sanddrift_merged_tar
12. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_ndlarfid_sanddrift_nueelastic_merged_tar
13. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_ndlarfid_sandstt_merged_tar
14. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_rockantindlarfid_sanddrift_merged_tar
15. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_rockantindlarfid_sandstt_merged_tar
16. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_ndlarfid_sanddrift_ghep_merged_tar
17. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_ndlarfid_sanddrift_nueelastic_ghep_merged_tar
18. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_ndlarfid_sandstt_ghep_merged_tar
19. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_rockantindlarfid_sanddrift_ghep_merged_tar
20. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_rockantindlarfid_sandstt_ghep_merged_tar

The dataset definition is:
" files where dune.dataset_name = '${NAMESPACE}:${name}' "

The following commands have been applied.

• rucio did add

• rucio did content add --to-did

[StevenCTimm]

I am spot-checking the data sets in question. the first one looks ok.
The files appear to be held on disk at the moment via rse-specific disk rules.
Tammy, are the ND people going to want these datasets to stay resident on disk long-term as an official dataset?

[twalton-fnal]

These "merged" datasets do not need to remain on disk for very long. Currently, the children's files are being uploaded to the staging disk.

[StevenCTimm]

I have spot checked them all and they look OK.. just waiting for Aaron to see if he wants to rename the datasets to put an
official tag on them. Once he gives the word I can make the rules to go to tape.

Do I understand correctly that these tarballs have got many child files and you are now in the process of loading up the individual child files on to the staging disks?

This opens a procedure question about what happens when the things we have stored on tape (tarballs) are not quite the same as what's on disk, individual constituent files.. we don't necessarily have to resolve this just now but we should think about it in future.

[StevenCTimm]

Tammy--I guess we need to hear from you.. do you want these to be considered "official" datasets or not.

Steve

[twalton-fnal]

Yes, the datasets can be named as official. Tammy Walton Scientist Great minds think alike differently. -unknown Computational Science & AI Directorate Data Science, Simulation, and Learning Division Fermi National Accelerator Laboratory 630 840 3178<tel:630%20840%203178> office ***@***.******@***.***> ***@***.******@***.***>

…

________________________________ From: Steven Timm ***@***.***> Sent: Wednesday, March 4, 2026 11:51 AM To: DUNE/data-mgmt-ops ***@***.***> Cc: Tammy Walton ***@***.***>; Author ***@***.***> Subject: Re: [DUNE/data-mgmt-ops] Archive ND Simulation merged tar files to tape (Issue #1104) [EXTERNAL] – This message is from an external sender [https://avatars.githubusercontent.com/u/26069304?s=20&v=4]StevenCTimm left a comment (DUNE/data-mgmt-ops#1104)<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_DUNE_data-2Dmgmt-2Dops_issues_1104-23issuecomment-2D3999162999&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=MaGbcCtSTqTLjrFyw_vDxFMm6ErxrjT7HZgZzholxLg&m=54_cpXdDNs8o1m44sPbTxV8clJbpUdDtdE41EZYKwWk-njV3vKwqN7Jp1aceA7Av&s=PBtHe7LwBCi1LVXDSgpptVOhHnLW9ZCzg-LegfVGUV8&e=> Tammy--I guess we need to hear from you.. do you want these to be considered "official" datasets or not. Steve — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_DUNE_data-2Dmgmt-2Dops_issues_1104-23issuecomment-2D3999162999&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=MaGbcCtSTqTLjrFyw_vDxFMm6ErxrjT7HZgZzholxLg&m=54_cpXdDNs8o1m44sPbTxV8clJbpUdDtdE41EZYKwWk-njV3vKwqN7Jp1aceA7Av&s=PBtHe7LwBCi1LVXDSgpptVOhHnLW9ZCzg-LegfVGUV8&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AQ6NGLPOQIDOKFBL524CJ3L4PBUKPAVCNFSM6AAAAACWEXU2ISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSOJZGE3DEOJZHE&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=MaGbcCtSTqTLjrFyw_vDxFMm6ErxrjT7HZgZzholxLg&m=54_cpXdDNs8o1m44sPbTxV8clJbpUdDtdE41EZYKwWk-njV3vKwqN7Jp1aceA7Av&s=2IgUp_PiAyoh8i6E_Pp5AbfkfYL93hhKDBW1PqXPxRg&e=>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ahiguera-mx]

Rules on tape have been put in place, here are the rule IDs

neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_ndlarfid_sanddrift_merged_tar 8a3cdb87a6354939a892d792c02d188f 
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_ndlarfid_sanddrift_nueelastic_merged_tar 096509d119fc4fe09a33b01b01b2ebf7
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_ndlarfid_sandstt_merged_tar c1de131c006347a59d47168737f69cb9
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_rockantindlarfid_sanddrift_merged_tar e2f7902b1a6642898294a333f3c4d787
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_edep_rockantindlarfid_sandstt_merged_tar be32b4d258974cb2ae09eff8f2f426a5
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_ndlarfid_sanddrift_ghep_merged_tar 11ca020cc6204d3fa67d6b6b1caa7a2e
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_ndlarfid_sanddrift_nueelastic_ghep_merged_tar b1d99f0d5ddf4e7a808ce1c21ec9df3b
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_ndlarfid_sandstt_ghep_merged_tar 4ee9deb5f56b45298b7dd66aa2fd81a3
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_rockantindlarfid_sanddrift_ghep_merged_tar
0f6adaf6c7254a69bc5eb0657749ba3d
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_genie_rockantindlarfid_sandstt_ghep_merged_tar ff928fc227e944aa8a922113e04ffe34
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_ndlarfid_sanddrift_merged_tar 52584bfad981449b9a14f54c65ad336e
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_ndlarfid_sanddrift_nueelastic_merged_tar 13ddb48e7ef149a9b053a5d0e7f297fe
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_ndlarfid_sandstt_merged_tar
c57c91951ed54ee6b5d19b35bf189baf
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_rockantindlarfid_sanddrift_merged_tar 9e330b46c07b49e587f2d1d9e0d54f6f
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_edep_rockantindlarfid_sandstt_merged_tar
9595543c488c47aa85eaae8d47060e2a
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_ndlarfid_sanddrift_ghep_merged_tar
7bd2d5e5edf94c94855731430580b7e8
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_ndlarfid_sanddrift_nueelastic_ghep_merged_tar
4bf93c5ea5194831a888a12c36bc8507
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_ndlarfid_sandstt_ghep_merged_tar 
35ef4545bc3b436a8da2e5da08ecbfa6
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_rockantindlarfid_sanddrift_ghep_merged_tar db73134741e545ecb32905dc88eef7b2
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_genie_rockantindlarfid_sandstt_ghep_merged_tar
5315e8e77f744bb9bd897f2049c9f567

[twalton-fnal]

Please archive the following small file (merged) datasets to tape.

https://metacat.fnal.gov:9443/dune_meta_prod/app/gui/datasets?selection=namespace%3Aneardet-lar-tms-sand&namematch=tms*tar&submit=Submit

1. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsreco_full_sanddrift_merged_tar
2. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsrecoreadout_full_sanddrift_merged_tar
3. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsreco_full_sandstt_merged_tar
4. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsrecoreadout_full_sandstt_merged_tar
5. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsreco_full_sanddrift_merged_tar
6. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsrecoreadout_full_sanddrift_merged_tar
7. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsreco_full_sandstt_merged_tar
8. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsrecoreadout_full_sandstt_merged_tar

[twalton-fnal]

Please add these additional datasets. (when you have time)

1. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_spill_ndlarfid_sanddrift_nueelastic_singles_merged_tar
2. neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_spill_ndlarfid_sanddrift_nueelastic_singles_merged_tar

[twalton-fnal]

Two more datasets.

1. neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_convert2h5_ndlarfid_sanddrift_nueelastic_singles_merged_tar
   2.neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_convert2h5_ndlarfid_sanddrift_nueelastic_singles_merged_tar

[StevenCTimm]

OK we will have a look at the 3 groups of datasets this afternoon,.

[StevenCTimm]

OK I have verified that all these datasets are consistent in rucio and metacat and have files of sufficient size to send to tape.
Aaron any reason why not to make the rules? I can do it.

[ahiguera-mx]

I will go ahead and create the rules

[ahiguera-mx]

Rules on tape have been put in place, here are the rule IDs

neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsreco_full_sanddrift_merged_tar	cac78857482b4c6cb269e64ebc162578
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsrecoreadout_full_sanddrift_merged_tar	79474fae24b743c080f985404b25b342
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsreco_full_sandstt_merged_tar	b8ebe03ab9a6499aab6b5ed91c06383f
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_tmsrecoreadout_full_sandstt_merged_tar	71159daac4b94bf0891ceb456be29c1e
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsreco_full_sanddrift_merged_tar	23376714ed484490b0d343d8c9904c8d
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsrecoreadout_full_sanddrift_merged_tar	d1763d11309145aeab7c4b848491ea05
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsreco_full_sandstt_merged_tar	1be0d23b09f448ccbad149cf259abacb
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_tmsrecoreadout_full_sandstt_merged_tar	85cee3918cc442d4996c68bf1d249249
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_spill_ndlarfid_sanddrift_nueelastic_singles_merged_tar	779e8bb695bc481f97356158993d58bc
neardet-lar-tms-sand:MiniProdN5p1_NDComplex_FHC_convert2h5_ndlarfid_sanddrift_nueelastic_singles_merged_tar		8cae712e8d9c4fdf9bd711db13f108dc
neardet-lar-tms-sand:MiniProdN5p2_NDComplex_RHC_convert2h5_ndlarfid_sanddrift_nueelastic_singles_merged_tar		6ac936de474741eaa31d02da5d3f1bed