NexusVisitor and Annotator architecture

[lukaspie]

Motivation

The previous HandleNexus class mixed three unrelated responsibilities in a single monolithic object: HDF5 file I/O, depth-first traversal, and application-specific output logic (annotation, NOMAD archive population). This made every processing mode tightly coupled to every other, and prevented the traversal from being reused independently.

In addition, the code for pynx read (the old read_nexus) was very hard to read and heavily dependent on the nxdl_utils from the `definitions .

This PR introduces:

• a clean separation of concerns based on the visitor pattern.
• a refactoring of the annotation tool, which becomes self-contained and produces more sensible output.

What changes

New architecture

Layer	Key type	Location
Schema	NexusNode	pynxtools.nexus.nexus_tree
Traversal	NexusFileHandler	pynxtools.nexus.handler
Processing	NexusVisitor (ABC)	pynxtools.nexus.handler

NexusVisitor is an abstract base class with four lifecycle hooks:

class NexusVisitor(ABC):
    @abstractmethod
    def on_group(self, hdf_path: str, hdf_node: h5py.Group) -> None: ...
    @abstractmethod
    def on_field(self, hdf_path: str, hdf_node: h5py.Dataset) -> None: ...
    @abstractmethod
    def on_attribute(self, hdf_path: str, attr_name: str, attr_value, parent) -> None: ...
    @abstractmethod
    def on_complete(self, root: h5py.File) -> None: ...

NexusFileHandler owns file I/O and traversal. It knows nothing about what to do with each node; that is entirely the visitor's responsibility:

NexusFileHandler("file.nxs").process(my_visitor)

Annotator (pynxtools.nexus.annotation)

Replaces the annotation logic previously embedded in HandleNexus. Uses NexusNode for schema resolution instead of the legacy nxdl_utils /get_inherited_hdf_nodes path. Supports all three operating modes of read_nexus (-d documentation, -c concept query, default full annotation).

NexusNode enhancements (pynxtools.nexus.nexus_tree)

New methods and properties on NexusNode that the Annotator and downstream visitors rely on:

• get_inheritance_docs(): most-specific-first list of (nxdl_stem, doc_text) tuples across the full inheritance chain
• get_inheritance_enums(): most-specific-first list of (nxdl_stem, [values]) tuples
• get_inheritance_concept_paths(): list of (concept_label, path) tuples with full within-NXDL paths (e.g. NXinstrument/energy)
• best_child_for(name, node_type, nx_class): name-fit child selection
• find_node_at_path(path, ..., _cache): full-path traversal with optional dict cache
• has_nxcollection_parent(): checks for NXcollection ancestor (exempts required-field checks)
• concept_path property:: "NXarpes/ENTRY/INSTRUMENT" display string

HandleNexus deprecation (pynxtools.nexus.nexus)

HandleNexus is kept for backward compatibility and emits a DeprecationWarning. All internal call sites have been migrated to
NexusFileHandler + Annotator. External code can continue to use HandleNexus without changes.

pynx read: completely redesigned output

This is the most user-visible change in the PR. The read_nexus / pynx read command produces fundamentally different output compared to before.

Before: schema information came from nxdl_utils.get_inherited_hdf_nodes, which resolved inheritance lazily, inconsistently, and without type awareness. Output was a flat log of field values interleaved with NXDL lookups; schema matches were often
missing or wrong for namefitted groups and fields.

After: schema resolution goes through NexusNode, which builds the full inheritance chain from NXDL XML at tree-construction time and exposes it via typed methods. For every group, field, and attribute the annotator now emits:

• Concept — the exact NXDL path from the active application definition (e.g. NXarpes/ENTRY/INSTRUMENT) with its optionality (REQUIRED / RECOMMENDED / OPTIONAL)
• Inheritance — the ordered chain of NXDL base classes that define this concept, with full within-NXDL paths for each level (e.g. NXarpes/NXentry/NXinstrument, NXentry/NXinstrument, NXinstrument) and the most-specific documentation inline
• Value / shape / dtype — for fields, as before

The node resolution is now type-aware: groups, fields, and attributes are resolved through separate lookup paths, eliminating the confusion between nodes that share a name at the same level but have different XML tag types (field vs. attribute). Namefit matching (e.g. data_1 → DATA) is applied consistently across groups and fields at every depth.

Module restructuring

What	Before	After
NeXus schema tree	pynxtools.dataconverter.nexus_tree	pynxtools.nexus.nexus_tree
String decoding helper	inline in nexus.py	pynxtools.nexus.utils.decode_if_string
NXdata axis/signal helpers	inline in nexus.py	pynxtools.nexus.nxdata

pynxtools.dataconverter.nexus_tree is kept as a re-export shim for backward compatibility.

Circular import fix

Moving nexus_tree out of dataconverter/ created a circular import: nexus_tree → dataconverter.helpers → dataconverter.__init__ → validation → nexus_tree. Fixed by defining validate_data_dict as a lazy forwarder in helpers.py, making the eager import in dataconverter/__init__.py unnecessary. The monkey-patch pattern is preserved as _apply_monkey_patch() for code that needs
the real function object.

NOMAD parser (pynxtools.nomad.parsers.parser)

NomadParser is now a thin wrapper: it constructs an Annotator(parser=callback) and passes it to NexusFileHandler. The legacy _nexus_populate callback contract is fully preserved.

Documentation

• New: docs/learn/pynxtools/architecture.md: three-layer architecture overview
• New: docs/how-tos/pynxtools/implement-a-visitor.md: how-to guide with two worked examples
• Updated: docs/reference/cli-api.md: improved read_nexus documentation
• Updated: docs/index.md and mkdocs.yaml: navigation for new pages

What this PR does NOT change

• NOMAD archive population logic: _to_section, _populate_data and related functions in parser.py are unchanged.

How to review

The logical reading order matches the dependency graph:

1. nexus/nexus_tree.py: new NexusNode methods (pure data model, no side effects)
2. nexus/handler.py: NexusVisitor ABC and NexusFileHandler
3. nexus/annotation.py: Annotator implementation
4. nexus/nexus.py: HandleNexus deprecation and CLI cleanup
5. nomad/parsers/parser.py: thin NomadParser wrapper
6. dataconverter/__init__.py + dataconverter/helpers.py: circular import fix
7. docs/: architecture and how-to docs

Open issues addressed

The following open issues under the read_nexus label and the umbrella tracking issue #526 are affected by this PR:

Issue	Description	Status
#59	Child resolution conflates fields and attributes sharing a name	Fixed — best_child_for is type-aware (node_type parameter)
#139	NxdlAttributeError crash resolving @units at deep nesting depths	Crash eliminated — get_node_at_nxdl_path is no longer used; @units may still show as NOT IN SCHEMA at some depths
#422	Namefitted attributes (e.g. AXISNAME_indices) not recognized	Fixed — _find_attr_node now uses best_child_for(node_type="attribute") which applies get_nx_namefit to variadic attribute names
#436	Crash (KeyError) on AXISNAME_indices pointing to missing field	Not fixed — crash site in axis_helper is unchanged
#128	Concept query (-c) broken for inherited appdef paths	Not fixed — _handle_concept_query still delegates to get_all_is_a_rel_from_hdf_node

Fixes #59
Fixes #139
Fixes #422