Core Functions

Core Functions

Shared utility functions used across the project, located in Code/functions/core_functions/.

parse_APPN_dataset_path

File: Code/functions/core_functions/parse_APPN_dataset_path.py

Parses metadata encoded in an APPN dataset folder path. Given any path within the folder structure, it extracts the node, project, site, sensor, date, run, and tier information.

Function Signature

parse_APPN_dataset_path(
    path: pathlib.Path,
    path_level: str = "auto",
) -> Dict[str, Any]

Parameters

Parameter	Type	Default	Description
path	pathlib.Path	—	Path to parse (file or directory at any level)
path_level	str	"auto"	Level represented by the path. One of: auto, root, node, project, site, sensor, date, run, tier, sub_tier

Return Value

A dictionary with the following keys:

Key	Type	Description
root	str or None	Path to the storage root
node	str or None	Node name (e.g. USYD_Narrabri)
project	str or None	Project folder name
site_folder	str or None	Full site folder name
site	str or None	Site name (without year prefix)
year	int or None	Year extracted from the site folder
sensor	str or None	Sensor platform name
date	pd.Timestamp or None	Date of data collection
run_folder	str or None	Run folder name
run	int or None	Run number
tier	str or None	Tier folder name
sub_tier	str or None	Sub-tier folder name
stem	str or None	Remaining path below sub_tier
valid	bool	Whether the path is valid
errors	list[str]	Validation error messages
path_level	str	Detected or specified level
input_path	str	Original input path as string

Auto-Detection

When path_level="auto" (default), the function:

1. Searches the path ancestry for a date folder matching YYYYMMDD
2. If found, walks upward to populate all parent fields (sensor, site, project, node, root) and downward for run/tier
3. If no date folder found, uses heuristics:
   • Tier folder: matches T\d+_.+ pattern
   • Sensor: matches a known sensor name
   • Project: matches \d{4}_.+ pattern (checked before site — the site pattern is more permissive and would otherwise match first)
   • Site: matches \d{4}.+ pattern
   • Node vs root: uses glob patterns to detect date folders at expected depths

Filesystem checks (the node/root glob and the date-folder depth validation) only run when the path exists on disk; nonexistent paths are validated by name shape alone, so parsing is machine-independent.

Known Sensor Names

The function recognises these sensor platform names:

GOBI, HIRES, M3M, CALVIS, PHENOMATE, MOLE, TEMS, PTEMS,
MPROBES, LITERAL, H30T, RHIZO, MAXAR, JILIN, FIELDOBS, IRT, M3T, SVC,
RGB, FIELDCAMS, ITRES

An unrecognised all-caps token at the sensor position is tolerated as a new sensor rather than flagged as an error.

Usage Example

from Code.functions.core_functions import parse_APPN_dataset_path
import pathlib

# Parse a run-level path
result = parse_APPN_dataset_path(
    pathlib.Path("/data/USYD_Narrabri/2025_Chickpea/2025IAWatson/GOBI/20250119/run_00/T0_raw")
)
print(result["node"])     # "USYD_Narrabri"
print(result["project"])  # "2025_Chickpea"
print(result["sensor"])   # "GOBI"
print(result["date"])     # Timestamp('2025-01-19')
print(result["run"])      # 0
print(result["tier"])     # "T0_raw"

# Explicit level specification
result = parse_APPN_dataset_path(
    pathlib.Path("/data/USYD_Narrabri"),
    path_level="node"
)

outputs_up_to_date

File: Code/functions/core_functions/outputs_up_to_date.py

mtime-based caching helper: returns True when every output file exists and is newer than every input file. Used by QA00 (and QA01) to skip already processed runs unless --force is given.

from Code.functions.core_functions import outputs_up_to_date
if outputs_up_to_date(inputs=[panel_path, raster_path], outputs=[table_path]):
    ...  # skip

spectral_qc module

File: Code/functions/spectral_qc/__init__.py

Shared spectral-QC helpers used by QA00_SpectralValidation.py and QA02_SpectralRunComparison.py:

Function	Purpose
default_bad_wavelengths()	Known-bad wavelength ranges (nm) per sensor/EM region
bad_wavelength_mask(wl, ranges)	Boolean mask for wavelengths inside bad ranges
reflectance_pct(values)	Normalise reflectance to percent (dtype-dependent)
run_sort_key(label) / resolve_run_palette(labels)	Stable run ordering + colour palettes (CARTO Bold ≤10 / Tableau_20 ≤20 / glasbey_dark >20)
known_panel_sets() / identify_panel_set(refs)	Physical panel-set signatures (Gryfn4P = {11,30,56,82}, Gryfn2P = {20,45}) and classification
snap_wavelengths(df)	Snap wavelengths from different sensor units onto a shared per-sensor/EM-region reference grid

Tests

Tests are in Code/functions/core_functions/tests/test_parse_APPN_dataset_path.py. The suite is machine-independent (fake paths, nothing touched on disk).

Run with:

pytest Code/functions/core_functions/tests/test_parse_APPN_dataset_path.py -v

The test suite covers:

• Invalid path_level raises ValueError
• Explicit level parsing for root, node, project, site, sensor, date, run, and tier levels
• Auto-detection at various path depths
• Multiple storage roots