add blog and detailed flight log extraction process

Author: swardiantara

docs/blog/index.md

@@ -0,0 +1,27 @@
+# Project Blog
+
+Welcome to the "NLP for Drone Flight Log Analysis" project blog! Here, we share updates, detailed how-to guides, insights, and behind-the-scenes looks at our research.
+
+---
+
+## Latest Posts:
+
+### 1. Extracting Flight Logs from VTO Labs Forensic Images
+
+**Date:** July 23, 2025
+**Author:** S. Silalahi
+
+This post provides a comprehensive, step-by-step guide on how we extracted human-readable flight log files from the encrypted forensic images of drone controller devices provided by VTO Labs. Learn about the tools and procedures involved in converting raw data into analyzable messages.
+
+[:octicons-arrow-right-24: Read More](vto-labs-extraction.md)
+
+---
+<!-- 
+### [Future Post Title Example]
+
+**Date:** YYYY-MM-DD
+**Author:** [Author Name]
+
+[A brief summary of your next blog post idea. You can copy-paste and modify the structure above for new posts.]
+
+[:octicons-arrow-right-24: Read More](future-post-filename.md) -->

docs/blog/vto-labs-extraction.md

@@ -0,0 +1,106 @@
+# Extracting Flight Logs from VTO Labs Forensic Images
+
+**Date:** July 23, 2025
+**Author:** S. Silalahi
+
+This blog post provides a detailed, step-by-step walkthrough of the process we employed to extract and decrypt human-readable flight log messages from the forensic images of drone controller devices, as sourced from the VTO Labs Drone Forensics Program. This process was a crucial part of building our comprehensive "NLP for Drone Flight Log Analysis" dataset.
+
+---
+
+## 1. Understanding the Source Data
+
+The VTO Labs dataset (accessible via [https://drive.google.com/drive/folders/1-UrxFGpCo54bVujwFmmqNbsZEV28dSNz](https://drive.google.com/drive/folders/1-UrxFGpCo54bVujwFmmqNbsZEV28dSNz)) consists of forensic images from various drone models and components. Our initial analysis revealed that relevant, human-readable flight log messages were predominantly found within data acquired from **controller devices**. Other components often contained encrypted, proprietary, or purely telemetry data not suitable for direct NLP analysis.
+
+## 2. Locating and Extracting Flight Log Files from Controller Artifacts
+
+The VTO Labs collection of drone images includes data acquired from various controller devices, specifically **Android phones**, **Android tablets**, and **iOS phones**. The extraction methodology varies slightly depending on the operating system and artifact file type. Our goal was to identify and extract files containing human-readable log messages.
+
+### 2.1. Android-Based Controllers (Phones & Tablets)
+
+For Android-based controller artifacts, VTO Labs provides two main types of files: `.zip` archives and `.001` forensic images.
+
+#### 2.1.1. For `.zip` Archives:
+
+These archives can often be directly unzipped using standard archival tools. Once unzipped, the flight logs are typically found in the following directories:
+
+* **Encrypted `.txt` Flight Log Files (`DJIFlightRecord_YYYY-MM-DD_[HH-MM-SS].txt`):**
+    * `/dji/dji.go.v4/FlightRecord/`
+    * `/dji/dji.pilot/FlightRecord/`
+* **Unencrypted Human-Readable Error Logs:**
+    * `/dji/dji.go.v4/LOG/ERROR_POP_LOG/`
+    * `/dji/dji.pilot/LOG/ERROR_POP_LOG/`
+    These folders contain simpler, often plain-text, human-readable error messages that did not require decryption.
+
+#### 2.1.2. For `.001` Forensic Images:
+
+For artifacts provided as `.001` forensic images (which require specialized forensic tools to mount and access), we used [**Autopsy**](https://www.autopsy.com/), an open-source digital forensics platform, to navigate and extract relevant files.
+
+* **Tool Used:** Autopsy (open-source digital forensics platform)
+* **Extraction Path:** Within Autopsy, we gained access to the file system and extracted the encrypted `.txt` flight log files from paths similar to:
+    * `/dji/dji.go.v4/FlightRecord/`
+    * `/dji/dji.pilot/FlightRecord/`
+
+### 2.2. iOS-Based Controllers (iPhones)
+
+For iOS-based controller artifacts, all data was provided in `.zip` archives.
+
+* **Extraction Method:**
+    * Most `.zip` files from iOS controllers could be **directly unzipped** to reveal their contents.
+    * For any `.zip` files that resisted direct unzipping or appeared corrupted, we reverted to using **Autopsy** to access their internal structure and extract the files containing human-readable log messages.
+
+### 2.3. Post-Extraction Processing for All Controller Logs
+
+After collecting both the encrypted `.txt` flight log files and the unencrypted `ERROR_POP_LOG` files (where applicable) from both Android and iOS sources:
+
+* The encrypted `.txt` log files (e.g., `DJIFlightRecord_YYYY-MM-DD_[HH-MM-SS].txt`) were then decrypted using the DJI Phantom Help Log Viewer, as detailed in the next section.
+* The unencrypted `ERROR_POP_LOG` files were immediately ready for inclusion in our raw message collection without further decryption.
+
+<!-- ## 2. Locating Encrypted Flight Log Files
+
+Within the controller device artifacts, we identified files typically named in the format `DJIFlightRecord_YYYY-MM-DD_[HH-MM-SS].txt`. These files are encrypted and contain the raw flight data. After going through all the available artifacts, we compile the found flight log files, named them DroNER and made it publicly accessible on [Mendeley Data](https://data.mendeley.com/datasets/fwcjyc754h/1).
+
+**File Path Structure (Conceptual):**
+    ```
+    Drone_Model/
+    └── DatasetID/
+        └── YYYY_Month/
+            └── controller_device/
+                └── DJIFlightRecord_2017-08-29_[14-30-27].txt
+    ```
+Under the `raw` folder of the `droner`, the extracted log files are arranged in the above structure. For instance, a log file from DJI Matrice 600 model is stored in `DJI_Matrice_600\df034\2018_June\mobile_android_logical` directory, with a sample flight log file name of `DJIFlightRecord_2018-06-20_[10-11-44]`. -->
+
+## 3. Decrypting the Logs with DJI Phantom Help
+
+DJI flight logs are encrypted, necessitating a decryption step. We utilized the online **DJI Phantom Help Log Viewer** for this purpose.
+
+* **Tool Used:** [https://www.phantomhelp.com/logviewer/upload/](https://www.phantomhelp.com/logviewer/upload/)
+
+* **Step-by-Step Decryption:**
+    1.  **Access the Tool:** Navigate to the DJI Phantom Help Log Viewer website.
+    2.  **Upload File:** Click the "Upload" button and select an encrypted `DJIFlightRecord_*.txt` file from your local `controller_device` directory.
+    3.  **Process:** The tool automatically processes and decrypts the file.
+    4.  **Download CSV:** Once decrypted, download the resulting human-readable data as a `.csv` file.
+    5.  **Repeat:** This manual process was performed for each individual encrypted flight log file.
+
+    !!! warning "Manual Process Note"
+        It's important to note that this was a file-by-file manual decryption process due to the nature of the online tool. For very large collections, this step can be quite time-consuming.
+
+## 4. Extracting Relevant Messages for NLP
+
+The downloaded CSV files contain numerous columns, encompassing various types of flight data. For our NLP analysis, we focused on columns specifically containing human-readable messages.
+
+* **Key Columns Extracted:**
+    * `APP.message`: Contains general operational messages.
+    * `APP.tip`: Provides advisory or tip messages.
+    * `APP.warning`: Contains warning or error messages.
+
+* **Constructing the Forensic Timeline:**
+    For each extracted log, we then paired these messages with their precise timestamps. This allowed us to reconstruct a chronological "forensic timeline" of events and communications from the drone system during the flight. This timeline became the raw input for our data cleansing procedures.
+
+    * **Timeline Elements:** `Timestamp (Date and Time)` + `Message Content`
+
+---
+
+## Conclusion
+
+This meticulous extraction and decryption process was fundamental in transforming raw, often inaccessible, drone flight data into a rich source of textual information for our NLP research. It highlights the practical challenges involved in acquiring and preparing data from real-world forensic artifacts.

docs/methodology/data-collection.md

@@ -33,19 +33,24 @@ In addition to AirData, a significant portion of our raw data was derived from f
 * **Message Source Identification:**
     Upon initial examination of the VTO Labs data, it was observed that human-readable flight log messages were consistently found **only within log files acquired from controller devices**. Logs from other drone components (e.g., the aircraft itself) were often in proprietary binary formats or contained only telemetry data without explicit textual messages relevant to our NLP tasks.
 
+    !!! info "Detailed Extraction Guide"
+        For a comprehensive, step-by-step guide on how we extracted and decrypted these flight log files from the VTO Labs forensic images, please refer to our dedicated blog post:
+        [:octicons-arrow-right-24: Extracting Flight Logs from VTO Labs Forensic Images](../blog/vto-labs-extraction.md)
+
+
 * **Extraction and Decryption Process:**
     The extraction process for VTO Labs data involved the following steps:
 
     1.  **Controller Log Identification:** For each drone model and dataset ID, all flight log files originating from the `controller_device` artifact were identified.
-    2.  **Local Storage Structure:** These raw, encrypted flight logs were organized and stored locally using the following hierarchical folder structure to maintain provenance:
+    <!-- 2.  **Local Storage Structure:** These raw, encrypted flight logs were organized and stored locally using the following hierarchical folder structure to maintain provenance:
         ```
         Drone_Model/
         ├── DatasetID/
         │   └── YYYY_Month/
         │       └── controller_device/
         │           └── DJIFlightRecord_YYYY-MM-DD_[HH-MM-SS].txt
         ```
-        *Example Filename:* `DJIFlightRecord_2017-08-29_[14-30-27].txt`
+        *Example Filename:* `DJIFlightRecord_2017-08-29_[14-30-27].txt` -->
     3.  **Decryption:** The raw `.txt` flight log files from DJI drones are typically encrypted. To obtain human-readable messages, each file was individually decrypted using the **DJI Phantom Help Log Viewer** online tool.
         * **Tool Used:** [https://www.phantomhelp.com/logviewer/upload/](https://www.phantomhelp.com/logviewer/upload/)
         * **Procedure:** Each encrypted `.txt` log file was uploaded to the Phantom Help Log Viewer, decrypted, and then the resulting human-readable data was downloaded as a `.csv` file. This was a manual, file-by-file process.

docs/methodology/index.md

@@ -0,0 +1,48 @@
+# Our Methodology: A Transparent Approach
+
+The success and reliability of any data-driven research, especially in critical domains like aviation safety, hinge upon a meticulously documented and rigorously applied methodology. This section provides a comprehensive overview of the processes involved in creating the "NLP for Drone Flight Log Analysis" dataset and the associated analytical frameworks.
+
+Our commitment to open science and reproducibility means that every step, from initial data collection and cleansing to detailed annotation procedures, is thoroughly explained and justified by relevant industry standards and academic best practices.
+
+---
+
+## Key Phases of Our Methodology:
+
+Our approach is structured around the following interconnected phases:
+
+<div class="grid cards" markdown>
+
+-   :material-cloud-download:{ .lg .middle } __Data Collection__
+
+    Details on how raw drone flight log messages were acquired from diverse sources, including AirData UAV and forensic artifacts from VTO Labs.
+
+    [:octicons-arrow-right-24: Learn More](data-collection.md)
+
+-   :material-broom:{ .lg .middle } __Data Cleansing__
+
+    An in-depth explanation of the procedures used to transform noisy, inconsistent raw messages into a clean, standardized, and machine-readable format.
+
+    [:octicons-arrow-right-24: Learn More](cleansing.md)
+
+-   :material-clipboard-edit-outline:{ .lg .middle } __Annotation Procedures__
+
+    Detailed guidelines for how flight log messages were annotated for various NLP tasks, ensuring consistency and high inter-annotator agreement.
+
+    [:octicons-arrow-right-24: Learn More](general-annotation.md)
+
+-   :material-scale-balance:{ .lg .middle } __Justification & Standards__
+
+    How our annotation guidelines and cleansing decisions are grounded in existing aviation regulations and standard documents (e.g., FAA, NTSB, INTERPOL).
+
+    [:octicons-arrow-right-24: Learn More](annotation-justification.md)
+
+</div>
+
+---
+
+## Our Commitment to Reproducibility
+
+By detailing our methodology here, we aim to provide researchers, practitioners, and enthusiasts with the necessary context and understanding to replicate our work, build upon our findings, and contribute to the advancement of NLP in drone aviation. We encourage thorough review and welcome feedback on our methods.
+
+!!! tip "Dive Deeper"
+    Explore the sub-sections in the sidebar for granular details on each phase of our methodology.

mkdocs.yml

@@ -1,5 +1,5 @@
 site_name: NLP for Drone Flight Log Analysis
-site_url: https://swardiantara.github.io/DroneNLP/ # IMPORTANT: Update this with your GitHub username and repository name
+site_url: https://dronenlp.github.io/documentation/
 site_description: Unlocking Insights from Drone Flight Logs with NLP.
 site_author: Swardiantara Silalahi
 
@@ -35,6 +35,7 @@ nav:
   - Home: index.md
   - About the Project: about.md
   - Methodology:
+    - methodology/index.md
     - Data Collection & Initial Scope: methodology/data-collection.md
     - Data Cleansing Procedure: methodology/cleansing.md
     - Annotation Justification & Standards: methodology/annotation-justification.md
@@ -46,7 +47,7 @@ nav:
     - Challenge 002: Autel EVO II - Flight Y - dataset/challenge-002.md
   - Dataset Download: download.md
   - Publications: publications.md
-  - Resources: resources.md
+  - Blog: blog/index.md
   - Contact: contact.md
 
 markdown_extensions: