fastforest

Fast approximate-forest regression and multiclass classification in Rust, with Python bindings. It can quickly and accurately fit datasets with arbitrarily large row counts (millions of rows or more), and scales down to tiny datasets too.

Across twelve numeric and mixed-data benchmarks spanning 1,030 to 20,216,100 row datasets covering regression, binary classification, and multiclass classification, fastforest is always either the fastest both to fit and predict, or the most accurate. For more results, see the benchmarks section.

Regression

Dataset	Model	RMSE ↓	R² ↑	Fit (s) ↓	Predict (s) ↓
SGEMM GPU 241,600 rows · 14 features · numeric	fastforest	0.06	1.00	0.11	0.017
	fastforest tuned	0.04	1.00	0.49	0.010
	sklearn RF	0.03	1.00	1.86	0.136
	sklearn HistGBM	0.20	0.97	1.23	0.022
Diamonds 53,940 rows · 9 features · mixed	fastforest	543	0.98	0.09	0.008
	fastforest tuned	543	0.98	0.21	0.008
	sklearn RF	550	0.98	0.97	0.032
	sklearn HistGBM	541	0.98	1.27	0.018

_{Bold is best for that dataset and metric. Results were measured on an Apple M4 Pro; fit includes preprocessing.}

The SGEMM target is the log-transformed mean runtime.

Classification

Dataset	Model	F1 acc ↑	Log loss ↓	Fit (s) ↓	Proba (s) ↓
Covertype 581,012 rows · binary features	fastforest	0.93	0.16	0.59	0.037
	fastforest tuned	0.93	0.16	1.59	0.039
	sklearn RF	0.92	0.17	4.33	0.210
	sklearn HistGBM	0.74	0.57	2.38	0.076
Bank Marketing 45,211 rows · 16 features · mixed	fastforest	0.73	0.21	0.05	0.006
	fastforest tuned	0.69	0.22	0.15	0.005
	sklearn RF	0.72	0.23	0.29	0.025
	sklearn HistGBM	0.76	0.20	1.37	0.026

F1 acc is macro-averaged F1, giving every class equal weight. Covertype uses its supplied binary features unchanged.

_{Untuned rows use model defaults. fastforest tuned uses the held-out meta-advisor described below.}

Install

pip install fastforest

This installs the Python library and the native fastforest-fit, fastforest-predict, fastforest-convert, and fastforest-compile executables.

Training and Validation

import numpy as np
from fastforest import FastForest,FastForestClassifier

rng = np.random.default_rng(42)
X = rng.random((1_000, 6))
y = 4*X[:, 0] - 2*X[:, 1] + X[:, 5]

model = FastForest(seed=42, oob=True).fit(X, y)
predictions = model.predict(X[:5])
oob_predictions = model.oob_prediction_
oob_counts = model.oob_counts_
oob_rows = model.oob_indices_

labels = np.where(X[:, 0]+X[:, 1] > 1, "high", "low")
classifier = FastForestClassifier(seed=42, oob=True).fit(X, labels)
probabilities = classifier.predict_proba(X[:5])
classes = classifier.predict(X[:5])
oob_probabilities = classifier.oob_decision_function_

X may contain numeric values, numeric strings, ordinary strings, and configured missing values. Regression y is converted to contiguous float32 and must be finite. Classification labels may be numeric or strings; classes_ records their probability-column order. Missing labels and single-class targets are rejected.

Creating and Using Models

Models can be saved as compact, portable .ffm files containing the forest, fitted preprocessing schema, task, and class labels. A loaded model supports ordinary in-memory prediction as well as bounded file prediction:

from fastforest import load

model.save("model.ffm")
restored = load("model.ffm")
predictions = restored.predict(X)
restored.predict_file("test.csv", "predictions.csv")
restored.save_executable("model-predict")

predict_file processes CSV or Arrow IPC/Feather in bounded batches rather than loading the whole input. save_executable builds a standalone predictor for the current platform, embedding both the model and Rust prediction runtime; building it requires a Rust toolchain, but running it requires neither Python nor a separate model file.

Installing fastforest also provides four commands. Their parsing, preprocessing, fitting, persistence, and prediction run in Rust:

fastforest-fit train.csv --target price --task regression --output model.ffm
fastforest-predict model.ffm test.csv --output predictions.csv
fastforest-convert numeric.csv --output numeric.arrow
fastforest-compile model.ffm --output model-predict
./model-predict test.csv --output predictions.csv

fastforest-fit accepts mixed CSV or numeric Arrow input and supports regression and classification; classification prediction accepts --proba. fastforest-convert streams numeric CSV into standard Arrow IPC for faster repeated ingestion. Run any command with --help for its complete estimator, schema, and batching options.

The native fastforest-predict binary, using a default model trained on an 80% Concrete Strength split, predicts from Arrow end-to-end in 4.5 ms for one row and 4.9 ms for all 206 validation rows. Reproduce it with python tools/cli_bench.py.

Benchmarking

This section contains additional results; all benchmarks, including those at the top of the README, follow the approaches described here. Unless noted otherwise, results use one reproducible 80/20 split, stratified for classification. Fit timing includes model construction, schema inspection, preprocessing, and fitting, but excludes process startup and inter-process transfer. Prediction timing includes input transformation. Every model/dataset combination has a 180-second limit.

Untuned rows use model defaults. For each fastforest tuned row, the corresponding meta-forest scores a batched OOB sweep of eight configurations with eight trees each, reduced to four configurations when the default forest has 20 trees. The meta-forests were trained on 49 other regression datasets or 125 other classification datasets. The predicted-best configuration is fitted normally, and the complete sweep, advisor inference, and final fit are included in fit time. Every dataset shown in this README—including alternate representations of the same dataset—is excluded from meta-forest training.

Regression

Dataset	Model	RMSE ↓	R² ↑	Fit (s) ↓	Predict (s) ↓
California Housing 20,640 rows · 8 features	fastforest	0.50	0.81	0.05	0.003
	fastforest tuned	0.50	0.81	0.10	0.002
	sklearn RF	0.51	0.80	0.44	0.013
	sklearn HistGBM	0.47	0.83	0.96	0.006
Concrete Strength 1,030 rows · 8 features	fastforest	5.48	0.88	0.01	0.000
	fastforest tuned	6.01	0.86	0.02	0.000
	sklearn RF	5.46	0.88	0.06	0.013
	sklearn HistGBM	4.65	0.92	0.78	0.005
Allstate Claims 188,318 rows · 130 features	fastforest	1,939	0.54	0.79	0.062
	fastforest tuned	1,939	0.54	2.28	0.059
	sklearn RF	timed out at 180s with 50 trees
	sklearn HistGBM	1,861	0.58	4.29	0.362
Diabetes 130-US Hospitals 101,766 rows · 46 features	fastforest	2.22	0.44	0.37	0.022
	fastforest tuned	2.22	0.43	0.86	0.024
	sklearn RF	2.20	0.45	5.17	0.158
	sklearn HistGBM	2.13	0.48	2.35	0.147
Walmart Store Sales 421,570 rows · 15 features	fastforest	5,394	0.94	0.29	0.021
	fastforest tuned	4,084	0.97	1.12	0.013
	sklearn RF	5,028	0.95	13.53	0.110
	sklearn HistGBM	6,604	0.91	2.30	0.065
Rossmann Store Sales 844,338 rows · 16 features	fastforest	0.16	0.85	0.54	0.028
	fastforest tuned	0.14	0.89	2.06	0.018
	sklearn RF	0.26	0.61	20.18	0.078
	sklearn HistGBM	0.30	0.46	2.87	0.051
ASHRAE Great Energy Predictor III 20,216,100 rows · 15 features	fastforest	1.03	0.77	1.15	1.182
	fastforest tuned	0.90	0.82	4.21	0.780
	sklearn RF	timed out
	sklearn HistGBM	1.43	0.55	28.75	0.856

For mixed data, the sklearn benchmarks use a custom pipeline based on scikit-learn's official preprocessing guidance and examples: wholly numeric columns are parsed and median-imputed, categorical columns use one-hot encoding through 20 levels and target encoding above that, and HistGBM uses native categoricals through its 255-level limit. This numeric parsing is needed for sensible handling of raw CSV-like tables; otherwise the pipeline uses the documented sklearn behavior. fastforest requires no custom preprocessing and takes the original datasets directly. sklearn RF timed out on a smaller Allstate run, so its default configuration was not run. For validation, Walmart uses a 12-week chronological holdout to match the competition's future-period forecasting setup, Rossmann uses its final six weeks, and ASHRAE uses December 2016. fastforest detects and expands their date fields automatically; the sklearn pipeline receives them as ordinary high-cardinality categoricals.

Classification

Dataset	Model	F1 acc ↑	Log loss ↓	Fit (s) ↓	Proba (s) ↓
Adult Census Income 48,842 rows · 14 mixed features	fastforest	0.80	0.31	0.07	0.007
	fastforest tuned	0.79	0.30	0.17	0.005
	sklearn RF	0.80	0.37	0.96	0.026
	sklearn HistGBM	0.82	0.27	1.42	0.027

Reproducing the benchmarks

Install the development dependencies and release build, then reproduce one dataset with:

pip install -e '.[dev]'
cargo build --release --bins
python tools/stage_binaries.py
maturin develop --release
python tools/accuracy.py --dataset california

Available regression datasets are sgemm, california, concrete, diamonds, allstate, diabetes, bluebook, bluebook_raw, walmart, walmart_raw, and walmart_nodate. Classification choices are covertype, covertype_grouped, adult, and bank. Run one forest alone with --ff_only or --rf_only, or reproduce all displayed results with:

for dataset in sgemm california concrete diamonds allstate diabetes; do
  python tools/accuracy.py --dataset "$dataset"
done

for dataset in covertype covertype_grouped adult bank; do
  python tools/accuracy.py --dataset "$dataset"
done

python tools/accuracy.py --dataset walmart --ff_only

Data preparation

FastForest fits a deterministic schema for every input column:

1. Non-missing values are parsed as float32 when every value can be parsed and are otherwise treated as strings. Numeric columns sort numerically and other columns sort lexically. Numeric columns whose values are all integral retain that metadata so analysis displays them with no decimal places.
2. A constant column is discarded and a binary column becomes one boolean feature. Cardinalities from 3 through max_dummy_cardinality become one boolean feature per value; the default maximum is 4. Larger cardinalities become one zero-based rank in the sort order.
3. By default, any ranked value occurring in more than 8% of the training rows also receives a boolean feature while remaining in the ranked column. Selecting its ranked parent during split search adds one randomly chosen frequent-value indicator as an extra candidate, without displacing another feature. Set frequent_value_fraction=0 to disable this or provide another fraction.
4. The default missing value is the empty value. Override it per column with missing_values, using column names or indexes. When training contains a missing value, FastForest adds <column>_missing and fills the value feature with the observed median. By default, a column containing no training missing values rejects missing values during prediction; set allow_new_missing=True to fill them with the fitted training median instead. Entirely missing columns are discarded.

X = np.array([
    ["18", "red",   ""],
    ["42", "blue",  "3.5"],
    ["31", "green", "2.0"],
], dtype=object)

model = FastForest(missing_values={2: ""}).fit(X, [1, 4, 3])

Columns that are an existing one-hot representation of one categorical predictor can be declared explicitly. Each group is validated as exactly one active 0/1 value per row, collapsed natively, and treated as one feature during fitting, importance, and explanations:

model = FastForestClassifier(one_hot_groups={
    "wilderness_area": ["Wilderness_Area1", "Wilderness_Area2", "Wilderness_Area3", "Wilderness_Area4"],
    "soil_type": [f"Soil_Type{i}" for i in range(1, 41)],
}).fit(X, y)

On Covertype, grouping its supplied wilderness and soil indicator columns gives:

Dataset	Model	F1 acc ↑	Log loss ↓	Fit (s) ↓	Proba (s) ↓
Covertype 581,012 rows · grouped features	fastforest	0.92	0.16	0.45	0.055
	fastforest tuned	0.92	0.16	1.37	0.057

Date and time columns are detected by default from at most 200 random training-pool rows using a conservative list of common formats. Every sampled non-missing value must match; ambiguous day/month forms remain candidates until a value above 12 resolves them, with month-first used if they remain ambiguous. Detected formats are saved with the model and never inferred again during prediction. Date columns are expanded natively using the same parts as fastai's add_datepart: year, month, ISO week, day, day-of-week, day-of-year, month/quarter/year boundary flags, hour, minute, second, and Unix elapsed seconds. Constant parts are discarded automatically, while missing or unparsable date values produce ordinary missing date parts.

Set date_columns={} to disable detection, or provide explicit strftime formats to override it:

model = FastForest(date_columns={"saledate":"%m/%d/%Y %H:%M"}).fit(X, y)

Ranking is a compact training representation, not a prediction-time requirement for numeric columns. After fitting, rank cutoffs are converted back to native numeric boundaries, so seen and unseen numeric values are compared directly without a rank lookup. Nonnumeric values are mapped through their fitted lexical ordering. An unseen low-cardinality value naturally receives all-zero dummies; an unseen high-cardinality value receives its insertion rank. The fitted median is retained for every nonempty column so optional inference-time missing-value imputation never inspects prediction data.

Python accepts pandas data frames, NumPy arrays, and Arrow tables, selects the bounded training pool first, converts only retained rows, and performs the bounded 200-row date-format check. The native CSV path likewise builds Arrow arrays only for retained rows, while Arrow IPC keeps its existing typed buffers. Full-column schema fitting and inference transformation then run in Rust behind the Arrow boundary, including numeric and lexical interpretation, missing values, categories, date expansion, and parallel column processing. The compact ranked training matrix and native-value prediction matrix remain internal implementation details.

Generated ranks, dummies, and missing indicators remain internal. Feature importance, explanations, and partial-dependence results aggregate them back to the original column and display its original values. Fitted interpretations are available in model.column_info_.

For reproducible sklearn comparisons on the same raw dataframe, sklearn_preprocessor implements the policy used by the benchmark: wholly numeric columns are parsed and median-imputed, categorical columns are one-hot encoded through 20 levels and target encoded above 20, and explicitly supplied missing markers are converted to nulls.

from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import make_pipeline
from fastforest import sklearn_preprocessor

preprocess = sklearn_preprocessor(X_train, missing_values={"age":"?"})
model = make_pipeline(preprocess, RandomForestRegressor(n_jobs=-1))
model.fit(X_train, y_train)

Install the optional dependencies with pip install 'fastforest[sklearn]'.

Algorithm

Each regression tree draws min(floor(bootstrap_fraction * n_rows), bootstrap_max) training rows. Classification treats bootstrap_max as a per-output cap and therefore uses bootstrap_max * max(1, n_classes-1) total rows per tree. replacement=None adaptively samples with replacement below 10,000 regression rows or 40,000 classification rows, and otherwise without it; pass True or False to override this. When bootstrap_fraction=None, it resolves to 0.8 with OOB enabled and 1 otherwise. Fractions above 1 are supported with replacement; without replacement the maximum is 1. Pass bootstrap_max=None to disable the cap. At each node, the default histogram splitter:

1. A node with fewer than min_node_size rows, or whose first max_node_samples sampled targets are equal, becomes a leaf.
2. A random contiguous window containing at most max_node_samples of the node's shuffled rows is selected.
3. The tree randomly selects the configured fraction of feature units, with a minimum of one. Encoded features are independent units except that every numeric value feature and its missingness indicator form one atomic unit.
4. For each selected feature, the sampled rows are sorted by their encoded rank and every distinct observed boundary is evaluated. Regression selects the split that most improves size-weighted negative sample standard deviation; classification uses multiclass Gini impurity. Both enforce the sampled child-size minimum.
5. Every regression leaf predicts the mean target of all tree-sampled rows that reached it. A classification leaf stores their class-probability vector. Thus leaf fitting processes each tree's capped sample once in total; it does not route the whole dataset through every tree.

By default, forest size targets two million sampled rows across its trees: n_trees = clamp(ceil(2_000_000 / sampled_rows_per_tree), 20, 50). Set n_trees to override it. The standard regression cap resolves to 50 trees; Covertype's seven-class cap resolves to 20. Other defaults are minimum node size 8, all rows capped at 40,000 per output, 60% feature sampling, and at most 320 evaluated rows per node. Enabling OOB changes the default sampling fraction to 0.8 so every row can receive held-out predictions. Preprocessing and trees build in parallel over columns and trees respectively. Classification prediction divides rows into roughly four blocks per Rayon worker and calculates how many fitted trees fit in a conservative 512 KiB working-set budget, including nodes and leaf probabilities. It processes those cache-sized tree batches within each row block; small trees retain row locality, while large trees automatically become tree-major. Supplying seed makes the fitted forest deterministic regardless of parallel scheduling.

max_features accepts "sqrt" or a fraction in (0, 1]; its default is 0.6. A numeric value and its missing indicator are always sampled as one feature unit.

FastForestClassifier.predict_proba averages the leaf probabilities over trees, while predict returns the corresponding original label. With OOB enabled, oob_decision_function_, oob_counts_, and OOB accuracy oob_score_ are available; oob_indices_ maps the bounded results to original training rows. Ordinary fitting remains bounded by the shared pool, per-output row cap, and max_node_samples rows per node.

Split selection and tuning

The histogram splitter is the production default. The original random-cutoff search remains available as a simpler teaching implementation:

model = FastForest(random_splitter=True, seed=42).fit(X, y)
fixed = FastForest(max_features="sqrt", seed=42).fit(X, y)

The histogram search randomly selects max_features, builds sparse target-statistic histograms from the node evaluation window, and checks every observed boundary for those features. The random splitter instead proposes random (feature, value) cutoffs, deduplicates them, and evaluates them on the same kind of node window. Its candidate count is controlled by cutoff_divisor; max_features is ignored when random_splitter=True.

The focused sweep tool takes comma-separated levels for every tree hyperparameter. The first value is the shared baseline and each later value creates one one-axis configuration. It compares an eight-tree batched OOB screen with ordinary resolved-tree fits on the dataset's canonical validation split, recording OOB, validation, and both training losses in one per-dataset CSV:

python tools/sweep.py --dataset california

Out-of-bag predictions

OOB calculation is opt-in with oob=True. After fitting:

• oob_prediction_ contains each training row's mean prediction from trees that did not sample that row.
• oob_counts_ contains the number of contributing trees.
• A row with no contributing tree has count zero and prediction NaN.
• Sampling without replacement at bootstrap_fraction=1.0 leaves no OOB rows, so all counts are zero and predictions are NaN.

Both attributes are None when OOB is disabled.

Model analysis

FastForest includes analysis tools with ordinary NumPy results. Data frames are accepted and supply feature names automatically; arrays use x0, x1, and so on. Sampling happens before Arrow conversion: permutation importance and feature relations use at most 5,000 rows, PDP/ICE uses 500, feature dependence uses 5,000, and drop-column importance uses at most 40,000 training and 5,000 validation rows by default. These limits are configurable through each function's sampling arguments. Plot methods import matplotlib only when called.

Importance

Use validation-set permutation importance by default. It measures the drop in model score after shuffling a feature without retraining:

importance = model.feature_importance(X_valid, y_valid)
importance.sorted()
importance.plot()

Correlated features can substitute for one another and therefore look individually unimportant. Permute them together to measure their joint importance:

importance = model.feature_importance(X_valid, y_valid,
    features={"location": ["latitude", "longitude"]})

model.drop_column_importance(X_train, y_train, X_valid, y_valid) performs the slower complementary analysis: it refits the forest without each feature. It accepts the same features groups. model.split_importance() returns the nearly free, normalized training-time split-gain measure, but permutation or grouped permutation is preferable because split importance is biased by the available cutoffs and correlated predictors.

Individual predictions and uncertainty

explanation = model.explain(X_valid[:3])
explanation.row(0)   # (feature, observed value, contribution), strongest first
explanation.plot(0)

tree_predictions = model.predict_trees(X_valid)
prediction_std = model.predict_std(X_valid)

For every row, prediction = bias + contributions.sum(). Contributions telescope through each tree's decision path and are then averaged across trees. They explain this forest's computation, not causality; correlated features can redistribute contributions between themselves.

Partial dependence and ICE

year = model.partial_dependence(X_train, "year_made")
year.plot()                   # average PDP plus individual conditional-expectation lines
year.plot(centered=True)
year.plot(clusters=5)         # representative centered ICE curves

interaction = model.partial_dependence(X_train, ["year_made", "sale_year"])
interaction.plot()

enclosure = model.partial_dependence(X_train,
    {"enclosure": ["enclosure_ac", "enclosure_erops", "enclosure_orops"]})

Partial dependence repeatedly replaces the selected feature values and averages the resulting predictions. ICE retains the individual prediction lines. These plots describe the fitted model rather than a causal intervention, and highly correlated features can produce unrealistic synthetic rows.

Collinearity and redundancy

from fastforest import feature_dependence,feature_relations

relations = feature_relations(X_train)
relations.groups(threshold=0.2)
relations.plot()
relations.plot_dendrogram()

dependence = feature_dependence(X_train)
dependence.predictability     # validation R² for predicting each feature from the others
dependence.plot()             # which other features provide that predictive information

feature_relations uses tie-aware Spearman correlation and average linkage implemented directly with NumPy. feature_dependence detects nonlinear redundancy by treating each feature in turn as a target, fitting a small forest from the remaining features, and measuring grouped prediction and permutation dependence.

Development

The project is locally installed with maturin until it joins the aai-ws workspace:

cargo build --release --bins
python tools/stage_binaries.py
maturin develop
cargo test
pytest -q

For performance work, build the extension in release mode and run the benchmark:

maturin develop --release
python tools/bench.py

Compare accuracy and timings against sklearn's random forest and histogram GBM on one fixed California Housing split:

python tools/accuracy.py

The displayed results live in tools/results/. After updating those CSVs, regenerate every table—including summary projections, displayed-value ties, formatting, links, and rowspans—with python tools/mk_readme.py. Edit prose in README.tmpl; README.md is generated.

Use --dataset concrete for the smaller Concrete Compressive Strength regression dataset, or --dataset sgemm for the 241,600-row SGEMM GPU Kernel Performance dataset. Each model/dataset combination runs in an isolated process with a three-minute timeout; process startup and input transfer are excluded from reported timings.

Use --ff_only with --min_node_size, --bootstrap_fraction, --bootstrap_max, --replacement, --max_node_samples, and --cutoff_divisor for focused FastForest experiments. These spellings come directly from the call_parse function parameters.