Adding catalogs section in the pyaml configuration file

[gupichon]

Proposal: Introducing a Value Catalog in pyAML

As discussed during the workshop, I would like to propose a new concept in pyAML: a catalog of values.

Motivation

Today, control-system references (Tango attributes, strings, metadata, etc.) are spread across device and model definitions. This makes configurations harder to read, maintain, and reuse.

The proposed value catalog addresses this by centralizing and formalizing how such values are defined and accessed.

Objectives

The main goals of this proposal are:

• Centralize all control-system references in a single, well-identified section of the configuration.
• Expose a catalog of values that can be accessed uniformly across pyAML.
• Make these values usable as metadata, enabling easy recording, logging, and export alongside measurements.

Proposed Configuration Structure

With this approach, device and model definitions remain focused on physics and behavior, while all control-system bindings are moved to dedicated catalogs.

A possible configuration would look like this:

type: pyaml.accelerator
facility: ESRF
machine: sr
energy: 6e9

simulators:
  - type: pyaml.lattice.simulator
    lattice: sr/lattices/ebs.mat
    name: design

data_folder: /data/store

arrays:
  - type: pyaml.arrays.serialized_magnet
    name: series
    elements:
      - mySeriesOfMagnets

devices:
  - type: pyaml.magnet.serialized_magnet
    name: mySeriesOfMagnets
    function: B1
    elements:
      - QF8B-C04
      - QF8D-C04
      - QD5D-C04
      - QF6D-C04
      - QF4D-C04
    model:
      type: pyaml.magnet.linear_serialized_model
      calibration_factors: 1.00504
      calibration_offsets: 0.0
      unit: m-1
      curves: sr/magnet_models/quadcurve.yaml

  - type: pyaml.magnet.quadrupole
    name: QF1E-C04
    model:
      type: pyaml.magnet.linear_model
      calibration_factor: 1.00054
      crosstalk: 1.0
      curve:
        type: pyaml.configuration.csvcurve
        file: sr/magnet_models/QF1_strength.csv
      unit: 1/m

  - type: pyaml.magnet.cfm_magnet
    name: SH1A-C01
    mapping:
      - [B0, SH1A-C01-H]
      - [A0, SH1A-C01-V]
      - [A1, SH1A-C01-SQ]
    model:
      type: pyaml.magnet.identity_cfm_model
      multipoles: [B0, A0, A1]
      units: [rad, rad, m-1]
    use-physics: true

Control-System Catalog

All control-system bindings are moved into a dedicated catalog:

control-system-catalog:
  type: pyaml.common.catalog
  refs:
    - reference: mySeriesOfMagnets
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/current
        unit: A

    - reference: QF1E-C04
      device:
        type: tango.pyaml.attribute
        attribute: srmag/vps-qf1/c04-e/current
        unit: A

    - reference: SH1A-C01-H
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/strength
        unit: rad

    - reference: SH1A-C01-V
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch2/strength
        unit: rad

    - reference: SH1A-C01-SQ
      device:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch3/strength
        unit: m-1

User-Defined Data Catalog

The same mechanism can also be used for global values that are not tied to a specific pyaml model:

user-defined-data:
  type: pyaml.common.catalog
  refs:
    - reference: machine-current
      device:
        type: tango.pyaml.attribute
        attribute: ans/ca/machinestatus/current
        unit: mA

    - reference: machine-mode
      device:
        type: tango.pyaml.string_attribute
        attribute: ans/ca/machinestatus/functionMode
        unit: ""

Key Properties of the Catalog

• All catalog entries are identified by a key (reference).

• Any value can be retrieved anywhere in pyAML using:

  • its exact key, or
  • a regular expression (for bulk access and grouping).

• Catalog entries are first-class objects and can be:

  • recorded automatically as metadata,
  • reused by multiple models or tools,
  • queried independently of the device hierarchy.

Open Design Questions

This proposal raises important points that need further discussion:

• How do models declare which catalog keys they depend on?
  In other words, how do we clearly define which values a model expects and how it resolves them from the catalog?
• Whether a validation step is required to ensure that all mandatory catalog entries are present, and at which stage this validation should occur.

[JeanLucPons]

I do not see in this scheme how to handle magnet model that does not have a 1 to 1 relationship.
i.e. : 4 Strength / 5 PS
How you will apply to external magnet module such as : https://github.com/python-accelerator-middle-layer/ebs_sextupole ?

[gupichon]

It is one of the points to be addressed. The model should clearly specify how it will look for keys in the catalog, and it is up to the user to fill it accordingly.
In your case, the model could look for keys matching a pattern like SF2-16246*, and the catalog would return the five power supplies. It is the user’s responsibility to provide the correct information in the catalog. Clearly, these cases are the tricky ones. That is also why I am questioning the need for a validation step.

[JeanLucPons]

We expect unique name in PyAML, so may be ?

control-system-catalog:
  type: pyaml.common.catalog
  refs:
    - reference: mySeriesOfMagnets
      devices:
        - type: tango.pyaml.attribute
          attribute: srmag/ps-corr-sh1/c01-a-ch1/current
          unit: A

    - reference: QF1E-C04
      devices:
         - type: tango.pyaml.attribute
           attribute: srmag/vps-qf1/c04-e/current
           unit: A

    - reference: SH1A-C01 # Reference refers to PyAML elements
      devices:
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/strength
        unit: rad
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch2/strength
        unit: rad
      - type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch3/strength
        unit: m-1

[gupichon]

It might be a solution, yes

[gupichon]

As the order may matter, your proposal is very good.

[gupichon]

One advantage of the idea I propose is that if you don’t want to use the control system, you just have to delete the section (or the reference to the file it points to).

[JeanLucPons]

It seems OK to me.
It is just an indirection so I don't see potential issue in the design.

[JeanLucPons]

May be:

control-system-catalog:
  type: pyaml.configuration.catalog # instead of common
  refs:

[gupichon]

Yes, this kind of detail needs to be clarified. All models will have to be modified, and almost all tests as well, so I expect to encounter some tricky parts during the development.

[JeanLucPons]

OK keep in mind that model need to be shared (no copy allowed) between virtual magnets belonging to the same CFM.
aggregator rely on this.

[gupichon]

Yes, it should ‘just’ impact the retrieval of elements, not the whole algorithm.

[gupichon]

I would like to have the opinion of others on this subject. @TeresiaOlsson, @gubaidulinvadim for instance ?
For development ease, this should be handled in close collaboration with issue #184

[gupichon]

Additionally, we should be able to retrieve string values from the control system for metadata and potential future checks (machine mode, etc.)

[gupichon]

Yes, this is one of the issues. The catalog is not meant to replace the declaration of elements such as BPMs. You will keep the BPM declarations in the model (using a filling pattern), and only associate the corresponding devices in the catalog (also using a filling pattern).

[JeanLucPons]

I meant by filling pattern the real filling pattern you have your accelerator (en français mode de remplissage).
At ESRF we get it from an oscilloscope. May be at soleil your are using an other device.

[gupichon]

Oh ! sorry ! My mind was stuck in the configuration file !
This should be treated by an external module to access elements not in the control system (databases, LDAP, hardware, ...)

[gupichon]

At SOLEIL, I think it is provided by a device (I have a doubt).

[JeanLucPons]

The filling pattern should be available from your control system.
This kind of devices we all share that has no link to AT should be discussed and standardized.
They will be used for metedata.

[JeanLucPons]

For this implementation please start with only one single object (BPM for instance) otherwise the code review will be almost impossible.
So we can focus on it and check that everything is ok.
Then we can perfrom the whole refurbishment.

[gupichon]

I will create the issues, but I really would like to have the opinion of others first.

[TeresiaOlsson]

I need to think a bit about this. For us pyAML is primarily an interface to the control system and in second hand an interface to a simulator so for us we might rather want to be able to skip the parts about the simulator when it's not needed. But I guess that is doable in a similar way as you suggest to separate out the information connected to the control system?

[gupichon]

I hadn’t considered that use case (since we are currently working on the development of SOLEIL II, I am mostly focused on the design part right now, sorry).
We should probably provide a solution to simplify the configuration file for your use case. You don’t use the design at all?

[TeresiaOlsson]

Neither BESSY II or MLS have a design model anymore. To many changes have been introduced over the years so the lattice is completely different compared to the design and the lattice model have been found experimentally.

We use the simulator in MML but calling the mode design in pyAML makes no sense for us. A more generic name would be better. We however also use MML for parts of the machine where we currently don't have the simulator configured like transfer lines and booster.

I think one very nice thing about MML is that you can choose which modes you want to use and you don't have to configure the other parts. So you can use only online, only model or both depending on your needs.

In MML the data is not separated so nicely as you suggest but for each device there is clearly fields that are for online and fields that are for simulator and you can just ignore adding the ones you don't need. So I imagine it should be possible to have a config where you include the parts common for all backends (like names) and then the config for each backend can be separated out?

[gupichon]

Ok, I get it now. So you just need access to your devices, with unit conversion, and no link to the model.
I may be wrong, but removing the simulator part and all references to it in the magnet models should be sufficient. If not, issues will need to be created so that the models can be corrected to work without the simulator.
I should add a small unit test where the dummy control system is used without the simulator.

[JeanLucPons]

Neither BESSY II or MLS have a design model anymore. To many changes have been introduced over the years so the lattice is completely different compared to the design and the lattice model have been found experimentally.

We use the simulator in MML but calling the mode design in pyAML makes no sense for us. A more generic name would be better. We however also use MML for parts of the machine where we currently don't have the simulator configured like transfer lines and booster.

I recall that it is 2 lines to add to allow both keywords design and model that refer to the same simulator instance.

I think one very nice thing about MML is that you can choose which modes you want to use and you don't have to configure the other parts. So you can use only online, only model or both depending on your needs.

This should be similar in PyAML but with the aim to write portable code.

[gupichon]

I have already notice some cases where referencing the model is mandatory. Like the RFTransmitter.

[TeresiaOlsson]

What does this mean for when objects for devices should be created? If there is a catalog I imagine that users don't only want to read the info in there but also change it after the config has been loaded. We do that sometimes in MML when we need to temporarily change a value and don't want to make a permanent change to the config. How would a change in the catalog be propagated everywhere?

I had before the suggestion that devices shouldn't be created directly when loading the config but only when the user needs them to allow dynamically changing the config. But that would still make the device static after it has been created. Now I'm thinking that if there is a catalog, maybe an option is that a device is always reading from there and doesn't contain any data itself? So if you change the config of a device you are actually changing the data in the catalog? I would say the AcceleratorObject in MML works like that to always keep a single source of truth. To me it looks like the catalog you suggest is pretty much the MML AO except that the AO also contains the values for the simulator and the arrays. It is a catalog of all device related data.

[TeresiaOlsson]

Basically, at what point does the device retrieve data from the catalog? If that happens before the user might want to change the config in their code it will not be possible to change the config dynamically or they risk having devices that are out of sync with the catalog.

[gupichon]

So the use-case scenario would be the following:

1. Load a configuration with a reference to a device.
2. Change that reference at runtime.
3. All PyAML tools using that device will automatically refer to the new reference.

Am I correct?
This is actually easier to implement with a catalog, provided that no references to the catalog items are kept anywhere (coding rule).

[JeanLucPons]

Yes, the PyAML device retrieves config during the loading phase. Then it depends of what you change dynamically. The tricky point is to change the PyAML object itself (to change its type for instance). In that case you will have to update all refs in others tools. The idea there was rather to allow (for instance) several TuneMonitor objects and to select which one to use in tuning (or measurement) tools by its name.

[gupichon]

If you define a coding rule that tools are not allowed to keep direct references to devices and can only reference the catalog (retrieving the object at each use), dynamic changes will be easier, as there will be no changes to propagate.

[gubaidulinvadim]

If you want to change the value temporarily, it should be done in pyAML via the code itself, not in the config file. At least, I would prefer it to be done this way and not to touch the config file a lot.

[gubaidulinvadim]

@gupichon I think your proposal is solid. It would be good to have a draft PR with just an example for a simple (BPM) or a more complex object (combined function magnet) just like @JeanLucPons suggested.

[JeanLucPons]

I agree.
I would like to know how it will look in the catalog when the BPM have several kind of devices.

- type: pyaml.bpm.bpm
  name: BPM_C01-01
  model:
    type: pyaml.bpm.bpm_tiltoffset_model
    x_pos:
      type: tango.pyaml.attribute_read_only
      attribute: srdiag/bpm/c01-01/SA_HPosition
      unit: mm
    y_pos:
      type: tango.pyaml.attribute_read_only
      attribute: srdiag/bpm/c01-01/SA_VPosition
      unit: mm

Soemthing like: ?

control-system-catalog:
  type: pyaml.common.catalog
  refs:
    - reference: BPM_C01-01
      x_pos_devices:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/current
        unit: mm
      y_pos_devices:
        type: tango.pyaml.attribute
        attribute: srmag/ps-corr-sh1/c01-a-ch1/current
        unit: mm

When offset and tilt are there ?
What will be the pyndatic model of each reference ?

[JeanLucPons]

I think to something like this to get the catalog config in the BPM model module.
Then the catalog can get it and it will be easy to disable it.

PYAMLCLASS = "BPMSimpleModel"

class CatalogModel(BaseModel):

    """
    Configuration model for BPM devices

    Parameters
    ----------
   reference: str
       Name of the BPM element
    x_pos : DeviceAccess
        Horizontal position device
    y_pos : DeviceAccess
        Vertical position device
    """
    model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")

    reference: str
    x_pos: DeviceAccess
    y_pos: DeviceAccess

class ConfigModel(BaseModel):
    """
    Configuration model for BPM simple model

    x_pos_index : int, optional
        Index in the array when specified, otherwise scalar
        value is expected
    y_pos_index : int, optional
        Index in the array when specified, otherwise scalar
        value is expected
    """
    model_config = ConfigDict(arbitrary_types_allowed=True, extra="forbid")

    x_pos_index: int | None = None
    y_pos_index: int | None = None