Project Title & Badges

Low-Cost Pre-Training Heuristic for Dataset Cartography: Approximating Training Dynamics via Class-Aware Geometry

TL;DR / Executive Summary

In modern Machine Learning, diagnosing why models struggle with certain data points usually requires expensive "Dataset Cartography"—training deep neural networks for dozens of epochs just to observe their behavior.

This project explores a fundamental hypothesis: Can we approximate a model's training confusion using a low-cost, pre-training geometric heuristic?

The Journey & Findings:

1. The Falsification: We initially hypothesized that local data Density (crowdedness) caused model confusion. Rigorous testing across 5 datasets proved this false; density is largely class-agnostic and only weakly correlates with learning difficulty.
2. The Pivot: We pivoted to Class-Aware Margin Distance (the distance from a sample to its nearest opposite-class neighbor in high-dimensional space).
3. The Proof: This geometric metric demonstrated a strong Pearson correlation with Neural Network training variability, serving as a meaningful proxy for learning difficulty without requiring full model training.
4. The Application: By pruning the top 5% lowest-margin samples, we successfully maintained or improved Random Forest accuracy across multiple domains (Financial, Medical, Census).
5. The Demographic Autopsy: Analysis of the pruned data revealed that this geometric proxy systematically highlighted socio-economic minorities in the Adult Income dataset, functioning as a diagnostic signal for implicit bias.

The Problem

Goal: Develop a low-cost, pre-training approximation of dataset cartography signals. Rather than replacing full cartography pipelines, this approach serves as a computationally cheaper diagnostic tool to evaluate dataset geometry and sample difficulty before heavy training begins.

Phase 1: The Density Hypothesis (A Negative Result)

To evaluate Margin Distance as a proxy for model confusion, I mapped the Euclidean Margin of the Breast Cancer dataset against the AI's training Variability (calculated over 15 epochs).

• Quantitative Proof: There is a strong negative Pearson correlation. As the distance to the nearest opposite-class point shrinks, the AI's training confusion measurably increases.
• Visual Representation: I reduced the 29-dimensional dataset to 2D using t-SNE. Highlighting the top 15% most "confusing" points (yellow) shows them clustering along the decision boundaries. (Note: While t-SNE is useful for visual intuition, it distorts global distances; the Pearson correlation serves as the primary quantitative validation of this relationship).

Phase 2: The Pivot to Class-Aware Geometry

Realizing that class-agnostic density was the wrong metric, I shifted my focus to Class-Aware Geometry. Instead of asking "how close is this point to any neighbor?", I asked, "how close is this point to its nearest enemy?"

I defined this as the Margin Distance: the physical distance from a sample to the nearest sample of an opposite class in high-dimensional space.

• High Margin Distance: The point is safely deep inside its own class territory.
• Low Margin Distance: The point is sitting directly on the decision boundary between two conflicting classes.

To calculate this, I utilized an unsupervised NearestNeighbors algorithm ($k=1$) with both Euclidean and Cosine metrics. I calculated the Margin Distance for every single row across 5 diverse datasets: Bank Marketing, Breast Cancer, Lung Cancer, Wine Quality (Multi-class), and Adult Income.

By doing this, I mathematically mapped the physical frontlines of the data without training a single predictive model.

Key Finding 1: Predicting AI Confusion

To prove that Margin Distance dictates model confusion, I mapped the Euclidean Margin of the Breast Cancer dataset against the AI's training Variability (calculated over 15 epochs).

• Statistical Proof: There is a definitive negative correlation. As the distance to the nearest opposite-class point shrinks (approaches 0), the AI's training confusion spikes.
• Visual Proof: I reduced the 29-dimensional dataset to 2D using t-SNE. When highlighting the top 15% most "confusing" points (yellow), they perfectly trace the frontline borders between the two classes.

Key Finding 2: The Data-Centric Drop Test

If these borderline points are truly the root cause of AI confusion, what happens if we simply delete them?

Traditional ML ideology assumes "more data is better." To challenge this, I ran a comprehensive test across all 5 datasets (including financial, medical, and census domains).

1. Baseline Model: Trained a Random Forest on 100% of the training data.
2. Cleaned Model: Used my geometric math to identify the top 5% most "borderline" points, deleted them, and trained the exact same model on the remaining 95%.

The Result: Pruning the geometrically confusing data consistently improved or maintained model accuracy on the unseen test set across the board.

Key Finding 3: The Pruning Breaking Point

Dropping 5% of the data worked, but what happens if we drop 10%, 20%, or 90%?

To find the mathematical breaking point, I ran an Ablation Study. The resulting curve revealed a fascinating ML dynamic:

• The Goldilocks Zone (0% - 5%): Deleting a small percentage of borderline data removes overlapping noise, making the model smarter.
• The Collapse (> 15%): Eventually, the accuracy falls off a cliff. Why? Because after deleting the noise, the algorithm begins deleting Support Vectors—the crucial data points that define the shape of the decision boundary.

This empirically proves that dataset pruning cannot be random; it must be surgically targeted at the extreme margins.

Unexpected Discovery: Automated Bias Detection

While analyzing the geometric properties of the pruned samples, I ran a demographic autopsy on the lowest-margin points from the Adult Income (1994 Census) dataset.

The geometry engine calculated mathematical distances in high-dimensional space without explicit access to demographic weights. However, isolating the top 5% lowest-margin samples revealed a significant systemic skew:

• Income Skew: Nearly 100% of the lowest-margin individuals were in the high-income (>50K) bracket. Because the majority of the dataset consists of lower-income earners, the algorithm structurally struggles to map the minority high-income class.
• Gender Skew: The low-margin points heavily and disproportionately skewed toward males compared to the general population baseline.

Conclusion: Margin Distance does not just identify random noise; it systematically flags minority clusters that lack sufficient feature representation. This geometric heuristic can double as a diagnostic tool to audit datasets for implicit socio-economic biases before model deployment.

Limitations & System Constraints

While effective as a pre-training heuristic, this approach has strict bounds:

1. Scalability: The current implementation relies on K-Nearest Neighbors (KNN). While optimal for datasets in the 10k-50k range, calculating exact $O(n^2)$ distances scales poorly for massive datasets. Future implementations would require Approximate Nearest Neighbor (ANN) indices (e.g., FAISS).
2. Feature Dependency: Margin Distance relies entirely on the quality of the static feature space. It is highly sensitive to scaling and cannot easily handle unstructured data (raw images/text) without first passing through a deep embedding layer.
3. Static vs. Temporal: This heuristic maps the static geometry of a dataset. It approximates, but cannot fully replace, the complex temporal learning dynamics (e.g., forgetting events) captured by full Dataset Cartography.