Skip to content

Code Modules

Jump to bottom
Arden Burrell edited this page Aug 13, 2026 · 5 revisions

Code Modules

The Code/ directory contains processing scripts, QA tools, and one-time utilities. Scripts are organised into prefixed subdirectories.

Directory Structure

Code/
├── DS02_DatasetQA/          # Dataset quality assurance scripts
│   ├── QA00_SpectralValidation.py       # per-run panel spectra extraction + QC report
│   ├── QA01_PointDistanceComparison.py  # per-run GCP vs groundtruth distances
│   ├── QA02_SpectralRunComparison.py    # cross-run panel spectra comparison
│   ├── QA03_GCPRunComparison.py         # cross-run GCP accuracy comparison
│   └── README.md
├── DS03_PlotExtractionCode/  # Plot-level data extraction
│   ├── PE00_LIDAR_extraction.py         # LiDAR point cloud → per-plot points
│   ├── PE01_HyperspecPlotExtraction.py  # hyperspectral ortho → per-plot pixels/metrics
│   └── README.md
├── DS05_SpectralIndices/     # Spectral index map computation
│   ├── SI00_SpectralIndices.py
│   └── README.md
├── functions/                # Shared utility functions
│   ├── core_functions/       # path parsing, caching, band wavelengths, metadata
│   ├── gcp_qc/               # GCP accuracy stats (QA01/QA03)
│   ├── plot_layout/          # plot-file discovery/validation (PE00/PE01)
│   ├── spectral_indices/     # band → spyndex mapping (SI00)
│   └── spectral_qc/          # shared spectral-QC helpers (QA00/QA02)
└── OT00_OneTimeScripts/      # One-time migration/maintenance scripts
    ├── OT00_RenameTiertoT.py
    ├── OT01_MoveGrawToRaw.py
    └── OT02_CollectCalvisGobiLogs.py

Naming Convention

Scripts use a prefix system:

Prefix Meaning Example
DS Dataset processing step DS02_DatasetQA, DS05_SpectralIndices
QA Quality assurance QA00_SpectralValidation.py, QA03_GCPRunComparison.py
PE Plot extraction PE00_LIDAR_extraction.py, PE01_HyperspecPlotExtraction.py
SI Spectral indices SI00_SpectralIndices.py
OT One-time script OT00_RenameTiertoT.py, OT01_MoveGrawToRaw.py

Within DS02 the QA scripts pair up: the per-run scripts (QA00 panels, QA01 GCPs) open the source data and write stable-named artefacts into each run's T1_proc/QC_data/; the cross-run scripts (QA02 panels, QA03 GCPs) consume those artefacts only — never re-opening rasters or geojson — and write comparison tables/figures into the routed QCReports/ folder (project → Documentation/QCReports/, node → Documents/QCReports/).

QA00 — Spectral Validation (per-run extraction + QC report)

File: Code/DS02_DatasetQA/QA00_SpectralValidation.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v2.2 (13.08.2026)

Extracts spectral data from ELM/VAL reflectance panels in hyperspectral imaging datasets and produces a per-run QC report and figure. Panel files must follow the official AerialDataQC naming convention. The summary below covers the essentials — see the linked README for tested package versions, full folder-structure assumptions, output schemas, troubleshooting, and future enhancements.

What It Does

  1. Searches for QC panel vector files matching QC_{ELM|VAL}[_{id}]_Panels[_{extra}].geojson (or .shp) under T1_proc/QC_data/
  2. Locates corresponding VNIR and SWIR orthomosaic rasters
  3. Extracts per-pixel reflectance with per-band wavelengths (nm) into QC_data/QC_Spectral_Tables/QC_{ELM|VAL}[_{id}]_spectra_{VNIR|SWIR}[_gproN].{parquet|csv}
  4. Identifies the physical panel set from its Panel_ref signature (panel_set column: Gryfn4P/Gryfn2P/unknown)
  5. Writes a per-run QC report (QC_data/QC_spectra_report.json) and per-target spectra figure (QC_data/QC_plots/)
  6. Skips up-to-date outputs (mtime caching); prints a REPORTED/SKIPPED summary table

Command-Line Arguments

Argument Description
--path Path to search for QC panel files (default: git root)
-f, --force Force overwrite of existing output files
--type Output format: parquet (default) or csv
-s, --skipplot Skip per-run figure generation
--skip-processing Never process rasters; only load existing outputs for reporting
--exclude-dir Directory names to exclude from the panel search
--no-radiance-check Disable the reflectance vs radiance range check
--keep-xy Retain per-pixel x/y coordinate columns
--allow-multi-gpro Process runs with multiple .gpro folders (debugging only)
-v, --verbose Detailed output

Usage

# Standard run (extraction + report + figure per run)
python Code/DS02_DatasetQA/QA00_SpectralValidation.py --path /path/to/data

# Extraction only, skip figures
python Code/DS02_DatasetQA/QA00_SpectralValidation.py --path /path/to/data -s

QA02 — Spectral Run Comparison (multi-run figures + sharing)

File: Code/DS02_DatasetQA/QA02_SpectralRunComparison.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v1.3 (13.08.2026)

Gathers the extracted spectra tables produced by QA00 across every run under a path and produces cross-run comparison figures (per-panel reflectance and residual spectra, one line per run). Never opens raster files. Residuals are each run's deviation from the cross-run mean spectrum at each snapped wavelength.

Command-Line Arguments

Argument Description
--path Node or project folder to crawl (default: git root)
--output-dir Explicit figure directory (required for other path levels)
--no-save Display figures instead of saving
--type Table format: parquet (default) or csv
--load-dir Also load spectra tables from this folder (other nodes)
--save-dir Save copies of every gathered table for sharing
--start-date / --end-date Inclusive date window limiting which runs are compared
--errorbar Spread band per run line: pi (default), sd, none
--exclude-dir Directory names to exclude from the table search
-v, --verbose Detailed output

Usage

# Compare every run under a node (figures -> <Node>/Documents/QCReports/)
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node

# Share with other nodes / combine external data
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node --save-dir /shared/spectra
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node --load-dir /shared/spectra

# Only compare runs after 1 June 2026
python Code/DS02_DatasetQA/QA02_SpectralRunComparison.py --path /path/to/Node --start-date 2026-06-01

Dependencies

numpy, pandas, xarray, rioxarray, rasterio, geopandas, shapely, matplotlib, seaborn, tqdm, GitPython, palettable (+ colorcet for >20 runs, pyarrow for parquet)

See the DS02_DatasetQA README for tested package versions and conda environment setup.

QA01 — GCP Point Distance Comparison (per-run accuracy)

File: Code/DS02_DatasetQA/QA01_PointDistanceComparison.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v1.1 (13.08.2026)

Compares digitised QC GCP point locations against surveyed groundtruth points for each run, reporting planar (and where available 3D) distances per matched ID plus an accuracy report that decomposes error into systematic bias and random scatter (rmse² = bias² + std², with bias bearing and bias_fraction).

What It Does

  1. Crawls for pairs of point files under <run>/T1_proc/QC_data/: groundtruth QC_GCP_groundtruth_points.geojson + one or more QC layers (QC_GCP_points.geojson single-layer, or QC_GCP_{Product}_points.geojson per product; optional trailing _extra info is carried through)
  2. Matches features by ID column, reprojecting to UTM when the input CRS is not metre-based
  3. Writes a per-pair distance table QC_GCP[_{Product}]_distances[_{extra}].{csv|parquet} and a companion _report.json next to the QC file
  4. Writes a per-pair displacement figure to QC_data/QC_plots/
  5. Prints summary statistics (count, mean, median, min, max, RMSE) and unmatched IDs

Command-Line Arguments

Argument Description
--path Root directory to search for groundtruth/QC pairs (default: git root)
--id-column Candidate ID column name(s) for matching (default: ID GCP_name)
--type Output table format: csv (default) or parquet
--exclude-dir Directory names to exclude from the search
--plot Also display the displacement plot interactively
-v, --verbose Detailed output

QA03 — GCP Run Comparison (multi-run accuracy figures + sharing)

File: Code/DS02_DatasetQA/QA03_GCPRunComparison.py
Full documentation: Code/DS02_DatasetQA/README.md
Author: Arden Burrell
Version: v1.0 (13.08.2026)

QA03 is to QA01 what QA02 is to QA00: it gathers the per-run GCP distance tables and accuracy reports written by QA01 across every run under --path and compares them. It consumes QA01 artefacts only (run QA01 first); missing, stale, or foreign-schema reports are recomputed from the distance tables with the shared maths in Code/functions/gcp_qc/.

Outputs (into the routed QCReports/ folder)

  • QC_GCP_run_comparison.{parquet,csv} — per run × product summary (counts, 2D/3D RMSE, mean/median/max, bias magnitude + bearing + fraction + class, QA01 pass/fail)
  • QC_GCP_{sensor}_metrics.png — RMSE/median/bias per run
  • QC_GCP_{sensor}_bias_vectors.png — per-run 2D bias vectors on a compass polar axis
  • QC_GCP_{sensor}_per_gcp.png — per-GCP-id displacement across runs
  • QC_GCP_run_comparison.md — overview report embedding the figures

Command-Line Arguments

Argument Description
--path Node or project folder to crawl for QA01 tables (default: git root)
--output-dir Explicit output directory (required for other path levels)
--no-save Display figures instead of saving
--save-dir Build a portable sharing container (tables + reports + figures + manifest)
--load-dir Merge a received container (or folder of QA01 tables) into the comparison
--start-date / --end-date Inclusive date window limiting which runs are compared
--exclude-dir Directory names to exclude from the search
-f, --force Regenerate outputs even when mtime-cached
-v, --verbose Detailed output

Usage

# Project-level comparison (saves to <Project>/Documentation/QCReports/)
python Code/DS02_DatasetQA/QA03_GCPRunComparison.py --path <Node>/<Project>

# Share with other nodes / merge external data
python Code/DS02_DatasetQA/QA03_GCPRunComparison.py --path <node> --save-dir /shared/gcp
python Code/DS02_DatasetQA/QA03_GCPRunComparison.py --path <node> --load-dir /shared/gcp

PE00 — LIDAR Plot Extraction

File: Code/DS03_PlotExtractionCode/PE00_LIDAR_extraction.py
Full documentation: Code/DS03_PlotExtractionCode/README.md
Authors: Arden Burrell & Richard Harwood
Version: v2.0 (13.08.2026)

Extracts plot-level data from LIDAR point clouds (GOBI and CALVIS sensors) by clipping to the site's Plot_Layout vector files and attaching DSM/DTM elevations. Outputs are Tier 1 products written to <run>/T1_proc/PlotExtracts/ (T2_traits/ is reserved for ML-model-derived products). The summary below covers the essentials — see the linked README for full folder-structure assumptions, output schemas, and the behaviour shared with PE01.

What It Does

  1. Crawls --path for *LiDAR_CombinedPointCloud.las/.laz (+ DSM/DTM rasters) under the official <run>/T1_proc/*.gpro/products/ location
  2. Discovers and validates the site's plot file Documentation/Plot_Layout/{YYYYSiteName}_plots.geojson (shared Code/functions/plot_layout/ helpers; --plot-variant selects alternates, _deprecated files are ignored)
  3. Reads the point cloud in chunks, pre-filters against the plot bounds, then assigns points to plots with a spatial join
  4. Samples DTM/DSM at every point and computes canopy height (Delta_z = z - DTM)
  5. Writes per-run parquet/CSV tables plus a YAML provenance sidecar to T1_proc/PlotExtracts/
  6. Skips up-to-date outputs (mtime caching); ends with a REPORTED/SKIPPED summary table

Command-Line Arguments

Argument Description
--path Folder to crawl for LiDAR products (default: git root)
--plot-variant Select a plot-file variant instead of the mandatory main plot file
--join-trial-info Join Documentation/Trial_Info/{YYYYSiteName}_trial_info.csv via plot_id
-f, --force Force overwrite of existing output files
--type Output table format: parquet (default) or csv
--exclude-dir Directory names to exclude from the crawl
--allow-multi-gpro Process runs with multiple .gpro folders (debugging only)
-v, --verbose Detailed output

Usage

python Code/DS03_PlotExtractionCode/PE00_LIDAR_extraction.py --path <Node>/<Project>

Dependencies

numpy, pandas, xarray, rioxarray, laspy (+ lazrs for .laz), geopandas, shapely, pyyaml, tqdm, GitPython

PE01 — Hyperspectral Plot Extraction

File: Code/DS03_PlotExtractionCode/PE01_HyperspecPlotExtraction.py
Full documentation: Code/DS03_PlotExtractionCode/README.md
Author: Arden Burrell
Version: v1.0 (13.08.2026)

Extracts per-plot pixel values and plot metrics from the *_{VNIR|SWIR}_Orthomosaic.bin hyperspectral orthomosaics (GOBI: VNIR; CALVIS: VNIR+SWIR). The .bin files are 16 GB+, so nothing is read whole: each plot polygon is read through its own bounding-box window and raw pixel rows stream to parquet via a pyarrow writer. Outputs go to <run>/T1_proc/PlotExtracts/.

Outputs (per run × EM region)

  • Raw pixels — PE_{REGION}_pixels[…].parquet, long format, one row per pixel × band (plot_id, band, wavelength, value; --keep-xy adds coordinates)
  • Plot metrics — PE_{REGION}_plot_metrics[…].parquet, per plot × band mean/median/std/count/valid_fraction plus run metadata (and trial-info columns when joined). Always derived from the saved raw table, never a second ortho read
  • Report — PE_extraction_report[…].md with extraction statistics and embedded QC figures (PE_figures/)
  • YAML provenance sidecars for every table

Command-Line Arguments

Argument Description
--path Folder to crawl for orthomosaics (default: git root)
--plot-variant Select a plot-file variant instead of the mandatory main plot file
--join-trial-info Join the site's trial-info CSV onto the metrics tables via plot_id
-f, --force Force re-extraction from the ortho even when outputs are up to date
--raw-only Only produce the raw per-pixel tables (skip metrics + report)
--metrics-only Refresh metrics/report from existing raw tables; the ortho is never opened
--read-strategy plot (default, one GDAL window per plot) or block (window per block of plots)
--block-size Plots per read block for the block strategy (default 24)
--keep-xy Retain per-pixel x/y coordinate columns in the raw tables
-s, --skipplot Skip report figure generation
--exclude-dir Directory names to exclude from the crawl
--allow-multi-gpro Process runs with multiple .gpro folders (debugging only)
-v, --verbose Detailed output

Usage

python Code/DS03_PlotExtractionCode/PE01_HyperspecPlotExtraction.py --path <Node>/<Project>

# Metrics/report refresh without touching the .bin
python Code/DS03_PlotExtractionCode/PE01_HyperspecPlotExtraction.py --path <Node>/<Project> --metrics-only

Dependencies

numpy, pandas, xarray, rioxarray, rasterio, geopandas, shapely, pyarrow, pyyaml, matplotlib, seaborn, tqdm, GitPython

SI00 — Spectral Index Calculation

File: Code/DS05_SpectralIndices/SI00_SpectralIndices.py
Full documentation: Code/DS05_SpectralIndices/README.md
Author: Arden Burrell
Version: v1.0 (13.08.2026)

Computes spyndex spectral index maps from the hyperspectral orthomosaics produced by the GRYFN processing chain. Raster in → raster out: DS05 never opens plot geojson or parquet — plot-level extraction from the index maps lives in DS03 (planned PE02).

What It Does

  1. Crawls runs for *_{VNIR|SWIR}_Orthomosaic.bin products
  2. Maps sensor bands → spyndex symbols via the shared Code/functions/spectral_indices/ helpers (band wavelengths from the GDAL/ENVI band tags)
  3. Computes every computable index from the ~280-index spyndex catalogue (or a curated --indices list)
  4. Writes per-run index maps + reports to <run>/T1_proc/SpectralIndices/

Products: GOBI → VNIR. CALVIS → VNIR (full VNIR index set at native resolution) + VNIRSWIR (VNIR resampled onto the SWIR grid; holds only the indices that need a SWIR band).

Outputs (per run × EM region × method)

  • SI_{region}_{method}[_gproN].nc — one variable per index, compressed NetCDF (split into _partNNofMM.nc when memory-bound); or one single-band GTiff per index with --format geotiff
  • SI_{region}_{method}[_gproN]_report.json — manifest (sources, band mapping, indices computed/skipped/delegated, per-index stats); consumed by the caching check and the planned PE02
  • SI_{region}_{method}[_gproN]_overview.md — human overview with stats tables and embedded figures
  • SI_figures/ — headline-index histogram grid + map thumbnail

Command-Line Arguments

Argument Description
--path Folder to crawl for orthomosaics (default: git root)
--method Band aggregation: Peak (default, band nearest the symbol centre), Mean (average across the symbol window), or both
--indices Curated spyndex index list (default: all computable from the sensor's bands)
--format netcdf (default) or geotiff
--resample-method VNIR→SWIR-grid resampling for the VNIRSWIR product: nearest (default) or linear
-f, --force Force recomputation even when outputs are newer than the source
-s, --skipplot Skip figure generation (maps and reports still produced)
--exclude-dir Directory names to exclude from the crawl
--allow-multi-gpro Process runs with multiple .gpro folders (debugging only)
-v, --verbose Detailed output

Usage

python Code/DS05_SpectralIndices/SI00_SpectralIndices.py --path <Node>/<Project>

Dependencies

Beyond the DS02 environment: spyndex, dask, h5netcdf, psutil, rasterio (conda install -c conda-forge spyndex dask h5netcdf psutil rasterio)

OT00 — Rename Tier to T

File: Code/OT00_OneTimeScripts/OT00_RenameTiertoT.py

A one-time migration script that renames folders starting with Tier to start with T (e.g. Tier0_raw → T0_raw).

Usage

# Use git root as search path
python OT00_RenameTiertoT.py

# Specify a custom path
python OT00_RenameTiertoT.py --path /path/to/data

The script searches recursively, lists all matching folders, asks for confirmation, then performs the rename.

OT01 — Move .graw Folders from T1_proc to T0_raw

File: Code/OT00_OneTimeScripts/OT01_MoveGrawToRaw.py

A one-time migration script that relocates *.graw folders from T1_proc into the adjacent T0_raw folder for GOBI and CALVIS sensors only.

What It Does

  1. Walks the search root looking for directories named T1_proc
  2. Keeps only those whose ancestry includes a folder whose name contains GOBI or CALVIS (case-insensitive)
  3. Collects the direct child folders of T1_proc ending in .graw
  4. Lists every proposed move and asks for confirmation
  5. Moves each .graw folder into the sibling T0_raw/ (created if missing), refusing to overwrite existing targets

Layout Assumption

.../<sensor>/<date>/run_XX/T1_proc/<name>.graw
        ->
.../<sensor>/<date>/run_XX/T0_raw/<name>.graw

Command-Line Arguments

Argument Description
--path Optional root directory to search. When supplied, the git repository check is skipped.
--dry-run List candidate moves without performing them.

Usage

# Use git root as search path
python OT01_MoveGrawToRaw.py

# Preview moves only
python OT01_MoveGrawToRaw.py --dry-run

# Specify a custom path
python OT01_MoveGrawToRaw.py --path /path/to/data

Dependencies

tqdm (and git on PATH when --path is not provided).

OT02 — Collect CALVIS/GOBI Log Files

File: Code/OT00_OneTimeScripts/OT02_CollectCalvisGobiLogs.py

A one-time script that walks a search directory, locates .graw/.gpro folders beneath CALVIS or GOBI sensor folders, and copies a curated set of log/configuration files into a single output directory (grouped as <SENSOR>_RUN_<NN>/) so they can be sent for review. Writes a manifest.csv recording each collected folder and whether its source path follows the APPN folder structure.

Usage

python OT02_CollectCalvisGobiLogs.py --path /path/to/data --output /path/to/collected_logs

Repository · Issues · MIT License · See Contributing-to-the-Wiki to edit these pages.

APPN DataStorage Wiki

Start here

APPN Folder Structure

Guides

Reference

Project

Clone this wiki locally