Thermal Image Co-Registration & Mapping

By: Roberta Dal Lago Garcia, Nihar Nanjappa, Alex Alvarez, Philip Anand and Gustavo Vejarano

🔄 Overview

This project introduces a robust computer vision pipeline for thermal hotspot co-registration using grayscale imagery captured by aerial drones. The system operates on normalized thermal images — including those derived from Celsius-based TIFFs — to detect, match, and align key thermal regions (hotspots) across multiple viewpoints.

Instead of relying on color heatmaps or pseudocolor, the pipeline strictly uses grayscale thermal data, which enhances compatibility with scientific sensors and provides consistent pixel intensity interpretation across varying datasets.

The solution integrates geolocation-aware field mapping, feature matching via SIFT descriptors, and homography-based contour transformation to allow precise visual tracking of thermal anomalies like fire zones or machinery heat leaks — even under partial occlusion or different flight paths.

It is ideal for real-time drone monitoring, thermal survey benchmarking, and longitudinal hotspot tracking across multiple flights.

🎓 Objectives

This project aims to develop a robust and scalable pipeline for hotspot co-registration across thermal grayscale imagery captured by drones. The primary goals include:

• Accurate Feature Alignment: Identify and align corresponding regions (hotspots) across sequential or partially overlapping thermal drone images using SIFT-based feature matching and homography filtering.
• Modular & Extensible Architecture: Provide a clean and modular object-oriented design to facilitate future development, integration with telemetry data, and experimental algorithms.
• Benchmarking at Scale: Automatically evaluate image pair trials, log success metrics, and generate interpretable visual diagnostics for hundreds of hotspot scenarios.
• Research-grade Dataset Support: Incorporate normalized grayscale outputs from high-resolution thermal sources like Celsius TIFFs to support academic use cases.

---

🗂️ Wildfire Dataset Description (FLAME and FLAME3)

This project utilizes thermal grayscale images from multiple drone-based datasets. Each dataset is structured to support accurate hotspot registration and benchmarking at scale.

🔥 FLAME Dataset

• Extracted from the 3-WhiteHot.mov drone footage (duration: 89 seconds, frame rate: 29.97 fps).
• Frames from second 13–16 and second 31–33 were selected, resulting in two clusters of trials:
  • Frames frame_00389 to frame_00479 (second 13–16)
  • Frames frame_00929 to frame_00989 (second 31–33)
• Each frame was saved as a PNG and manually paired based on visual correspondence.

🔥 FLAME3 Dataset

• Original images were in Celsius .tiff format. These were normalized and converted to grayscale PNG images.
• Image filenames like 00056.tiff, 00362.tiff were renamed to simplified versions (56.png, 362.png) for consistency.
• Out of 622 frames, a carefully selected subset with consistent perspectives was used to form valid comparison pairs.

These datasets are organized within data/img/ for direct access during benchmarking.

🔢 Core Modules

co_registration.py

This contains six classes responsible for processing and aligning drone images:

► Telemetry

A dataclass for storing drone camera metadata (HFOV, VFOV, GPS, yaw, etc.) and computing real-world dimensions of captured frames.

► ImageProcessor

Processes one image:

• Applies Gaussian blur and mean-based thresholding.
• Extracts and filters contours based on intensity/area.
• Computes SIFT keypoints and descriptors.

► DroneSnapshot

Binds a thermal image with telemetry data.

• Maps image pixel points to GPS coordinates.
• Holds filtered contours and post-homography transformed hotspots.

► FeatureComparison

Compares two processed images:

• Applies Lowe's ratio test with adaptive thresholds.
• Filters matches using RANSAC-based homography.
• Extracts final list of corresponding SIFT keypoints.

► ContourTransform

Given a set of pixel coordinates and their averaged GPS projection:

• Computes a homography matrix to warp contours.
• Transforms hotspot contours between views.

► FieldMap

Represents a pixel-based map:

• Converts GPS to canvas coordinates.
• Projects contours onto the field.
• Tracks snapshots and compares image pairs.

---

💡 Directory Structure

thermal_img_registration/
├── data/
│   ├── img/                          # PNG images of all trials (e.g., flame_a.png, jellyfish_b.png, frame_00389.png etc.)
│   ├── telemetrydata.csv            # Telemetry information for all images (Metadata per image)
│   └── pereira.png                  # Reference overview image
│
├── results/
│   ├── results.xlsx                 # Summary of transformation stats and match success/failure
│   ├── flame/                       # FLAME trial results (trial1.a_frame_00390_vs_frame_00391_sift_matches.png, etc.)
│   ├── flame3/                      # FLAME3 trial results (trial0.a_56_vs_415_sift_matches.png, etc.)
│   ├── multiple_hotspots/           # Multiple hotspot trial results (trial3.a_bog_b_vs_bog_d_transformed_contour.png, etc)
│   ├── partially_covered_hotspots/  # Partial-covered trial results (trial0.a_xo_a_vs_xo_d_p_sift_matches.png, etc)
│   ├── single_hotspots_fully_covered_complex/ # Single complex trial results (trial8.a_elf_c_vs_elf_d_sift_matches.png,..)
│   ├── single_hotspots_fully_covered_simple/ # Single simple trial results (trial11.a_dot_b_vs_dot_d_sift_matches.png, etc)
│   └── single_hotspots_sunny/       # Single sunny trial results (trial11.a_staff_a_vs_staff_d_sift_matches.png, etc)
│
├── src/
│   └── co_registration.py           # All core class definitions and logic for hotspot registration
│
├── tests/
│   ├── benchmarking.py              # Script to benchmark all trial pairs and log results
│   ├── visualizer_funcs.py         # All image drawing/visualization functions
│   └── output_images/              # Auto-created folder containing visual results for each benchmarking run

---

🎨 Visualization Output

From benchmarking.py, each image pair generates the following visualizations:

• *_original.png: side-by-side raw thermal images.
• *_thresholded.png: binary masks after Gaussian + threshold.
• *_contoured.png: overlaid contours on original images.
• *_sift_matches.png: SIFT matches post-RANSAC.
• *_field_map.png: Hotspots projected on the global field map.
• *_transformed_map.png: Transformed overlay of the two images on canvas.
• *_transformed_contour.png: Homography-adjusted contours.

---

📊 Benchmarking Notes

• Trials can be adjusted in tests/benchmarking.py via the pairs list
• Visual outputs include step-by-step images and maps
• Errors or failed visualizations are gracefully handled and logged

---

📅 How to Run Trials

1. Install dependencies:

pip install -r requirements.txt

2. Launch benchmarking script:

python benchmarking.py

3. All images will be saved under /output_images/.

---

📍 Data Structure

Each image must have a corresponding telemetry entry in the telemetry.csv file. Example image naming conventions:

• bear_a.png, bear_b.png: two views of the same hotspot.
• bear_c_p.png: partial image of the same region.
• For the FLAME3 dataset, images originally named like 00056.tiff, 00362.tiff were converted to grayscale PNGs and renamed as 56.png, 362.png respectively for better readability and consistency.
• For the FLAME dataset, frames were extracted from the video file (3-WhiteHot.mov) and stored with zero-padded filenames like frame_00389.png, frame_00965.png, indicating the frame number from which they were derived.

---

📸 Example Visual Output

Once benchmarking is complete, the following sequence is generated per image pair:

• original: Raw input images side by side

• thresholded: Preprocessed binary mask visualization

• contoured: Extracted hotspots with contours

• sift_matches: Validated feature matches

• transformed_map: Projected contours over canvas map

• field_map: Final integrated visualization

---

📄 Credits

Project debugged, enhanced by Philip Anand, building on prior work by Roberta Dal Lago Garcia, Nihar Nanjappa and Alex Alvarez. Special thanks to Roberta Dal Lago Garcia, Nihar Nanjappa who structured and benchmarked the original prototype.

---