feat (core): added replica catalog

[ryuwd]

Summary

Adds ReplicaCatalog, a Pydantic model for mapping Logical File Names (LFNs) to their physical replicas across distributed storage elements.

Depends on #746

Description

ReplicaCatalog provides a structured, validated representation of file replica information intended to be stored in JSON format. It serves as a more user-friendly replacement for Pool XML Catalog.

Key features:

• LFN/PFN validation: Automatic stripping of LFN: and PFN: prefixes, with validation
• Checksum support: Adler-32 (8 hex chars) and GUID (UUID format) checksums with format validation
• Storage element tracking: Each replica is associated with a storage element identifier
• Optional metadata: File size in bytes and checksum information

Example usage:

from diracx.core.replica_catalog import ReplicaCatalog

catalog = ReplicaCatalog(root={
    "/lhcb/MC/2024/file.dst": {
        "replicas": [
            {"url": "https://storage1.cern.ch/file.dst", "se": "CERN-DST"},
            {"url": "https://storage2.in2p3.fr/file.dst", "se": "IN2P3-DST"},
        ],
        "size_bytes": 1048576,
        "checksum": {"adler32": "788c5caa"},
    }
})

[read-the-docs-community]

Documentation build overview

📚 diracx | 🛠️ Build #31116593 | 📁 Comparing 9506a40 against latest (ccb1f48)

🔍 Preview build

No files changed.

[chrisburr]

I think we need to decide on the layout of core as replica_catalogue feels to specific for the top level

[aldbr]

I think we need to decide on the layout of core as replica_catalogue feels to specific for the top level

I guess that, at some point (may be now?), it would make sense to have a core/models directory that would contain a module per type of models (instead of having all our models in the same models.py module). Example:

• core/models/:
  • auth.py
  • jobs.py
  • sandbox.py
  • metadata.py
  • search.py
  • and replica_catalog.py

Any opinion?

[fstagni]

I think we need to decide on the layout of core as replica_catalogue feels to specific for the top level

I guess that, at some point (may be now?), it would make sense to have a core/models directory that would contain a module per type of models (instead of having all our models in the same models.py module). Example:

• core/models/:

  • auth.py
  • jobs.py
  • sandbox.py
  • metadata.py
  • search.py
  • and replica_catalog.py

Any opinion?

It makes sense to me

[fstagni]

I would split the refactoring in a separate PR

[ryuwd]

Will rebase after #746

[chaen]

Historically, the DFC (before LFC) has always been referred to as the Replica Catalog. Do you think you could come up with an alternative name (reflecting for example that it's a local file) ? Otherwise it's not dramatic :-)