Evalbar

Evalbar is a round-outcome probability model and analysis toolkit for Counter-Strike 2. It produces calibrated estimates of each side's probability of winning from the current game state, generates per-round probability timelines, and connects every prediction back to the exact round state in an interactive web application. It currently supports offline analysis of de_inferno demos.

Project overview

• A deterministic .dem processing pipeline with semantic QA.
• A tabular gradient-boosted baseline for fast experiments.
• A roughly 70K-parameter GraphTemporalTransformer over a 20-second, 4 Hz state window.
• Multi-task outputs for round outcome, plant, retake, save, defuse, contact, and next-kill events.
• Calibration and phase-aware evaluation using log-loss, Brier score, accuracy, and ECE.
• Offline latent extraction, PCA/UMAP projection, original-space clustering, nearest neighbors, generated findings, and feature attributions.
• A local-first FastAPI and React investigation workbench.
• Reproducible training automation for short-lived Vast.ai instances with an explicit destroy workflow.

Pipeline

flowchart TD
    DEM["CS2 .dem files"] -->|parse| ROUND["Canonical round states"]
    ROUND --> BASE["Tabular baseline"]
    ROUND -->|build-neural-dataset| TENSOR["20s / 4Hz token tensors"]
    TENSOR -->|train-neural| MODEL["GraphTemporalTransformer"]
    MODEL --> PRED["Calibrated validation predictions"]
    MODEL --> ATLAS["Latent atlas + explanations"]
    ROUND --> WORKBENCH["Investigation workbench"]
    PRED --> WORKBENCH
    ATLAS --> WORKBENCH

Loading

Neural network input features

Schema v8 represents each prediction as a causal 20-second window sampled at 4 Hz. The final sample is the prediction point, so the default window contains 80 ticks of current and historical state. Continuous values are normalized, categorical IDs use learned embeddings, and masks keep padding or missing tokens out of attention.

family	default tensor shape per example	channels
global round state	[80, 75]	75
players	[80, 10, 39]	39 per player
bomb	[80, 1, 7]	7
map zones	[80, zone_count, 20]	20 per zone
active utility	[80, 24, 14]	14 per utility token
recent salient events	[80, 16, 11]	11 per event token

The utility and event caps are configurable when building a dataset. Player count is fixed at ten, the bomb has one slot, and zone count comes from the checked-in Inferno map configuration.

Global round state: 75 channels

• Clock and score: seconds_since_live_norm, seconds_remaining_norm, score_ct_norm, score_t_norm, and score_diff_ct_norm.
• Alive players and total health: ct_alive_norm, t_alive_norm, ct_hp_norm, and t_hp_norm.
• Current economy and equipment: ct_equipment_norm, t_equipment_norm, ct_money_norm, t_money_norm, ct_armor_norm, t_armor_norm, ct_helmet_norm, t_helmet_norm, ct_defuser_norm, ct_primary_weapon_norm, t_primary_weapon_norm, ct_rifle_norm, t_rifle_norm, ct_sniper_norm, t_sniper_norm, ct_smg_norm, and t_smg_norm.
• Remaining team utility: ct_smokes_norm, t_smokes_norm, ct_flashes_norm, t_flashes_norm, ct_molotovs_norm, and t_molotovs_norm.
• Round context: ct_buy_type_id, t_buy_type_id, round_phase_id, is_pistol_round_est, and prior_winner_id.
• Bomb context: bomb_status_id, bomb_commitment_id, bomb_time_remaining_norm, and can_defuse_in_time. The timer is zero before a plant and decays from one to zero over the 40-second post-plant timer. Defuse feasibility uses the observed alive CTs, kit availability, and the 5-second or 10-second defuse requirement.
• Map pressure and tactical structure: a_pressure_norm, b_pressure_norm, ct_isolated_norm, t_isolated_norm, ct_tradeable_norm, t_tradeable_norm, flank_active, and flank_side_id.
• Recent sound and combat activity: shot_count_ct_norm, shot_count_t_norm, footstep_count_ct_norm, and footstep_count_t_norm.
• Active utility pressure: utility_a_pressure_norm, utility_b_pressure_norm, utility_banana_pressure_norm, utility_mid_pressure_norm, and utility_rotate_pressure_norm.
• Opening equipment differentials: opening_equipment_diff_ct_norm, opening_money_diff_ct_norm, opening_ct_buy_type_id, opening_t_buy_type_id, opening_armor_diff_ct_norm, opening_helmet_diff_ct_norm, opening_kit_diff_ct_norm, opening_primary_diff_ct_norm, opening_rifle_diff_ct_norm, opening_awp_diff_ct_norm, opening_smg_diff_ct_norm, opening_pistol_diff_ct_norm, opening_he_diff_ct_norm, opening_smoke_diff_ct_norm, opening_flash_diff_ct_norm, opening_molotov_diff_ct_norm, and opening_utility_points_diff_ct_norm.

The opening features are CT-minus-T differentials except for the two categorical buy types. The older per-side opening totals, route features, loss-streak features, and per-location execute counts were measured as unhelpful and are not present in GLOBAL_FEATURES.

Player tokens: 39 channels per player

• Identity-free state: side_id, alive, health_norm, armor_norm, helmet, defuser, has_bomb, money_norm, and equipment_norm.
• Position and view direction: x_norm, y_norm, z_norm, position_known, pitch_sin, pitch_cos, yaw_sin, and yaw_cos.
• Weapon and carried utility: weapon_category_id, smokes_norm, flashes_norm, molotovs_norm, and he_norm.
• Spatial and tactical context: distance_to_bomb_norm, nearest_teammate_norm, nearest_enemy_norm, trade_support_norm, is_isolated, is_lurking_candidate, is_flank_candidate, zone_id, zone_group_id, and site_relation_id.
• Current status: flash_duration_norm, flash_alpha_norm, is_scoped, is_walking, and is_defusing.
• Optional external skill input: skill_hltv_rating and skill_hltv_known. These channels are zero or marked unknown unless a skill table is joined during dataset construction. They are disabled in the keeper data because the measured result was null.

The player graph derives six pairwise edge channels from these inputs: normalized x displacement, normalized y displacement, Euclidean distance, same-team status, both-positions-known, and both-players-alive.

Bomb token: 7 channels

status_id, x_norm, y_norm, z_norm, position_known, site_relation_id, and commitment_id.

This token separates the bomb's physical location and map relation from the aggregate bomb timing and feasibility features in global state.

Zone tokens: 20 channels per authored map zone

• Zone identity: zone_id, zone_group_id, and site_relation_id.
• Occupancy and resources: ct_count_norm, t_count_norm, ct_hp_norm, t_hp_norm, ct_equipment_norm, and t_equipment_norm.
• Control and objective state: control_id and bomb_here.
• Utility state: active_utility_norm, ct_smoke_active_norm, t_smoke_active_norm, ct_inferno_active_norm, t_inferno_active_norm, ct_flash_recent_norm, and t_flash_recent_norm.
• Local activity: shot_count_norm and footstep_count_norm.

Zone identity and adjacency come from data/config/maps/de_inferno.yml. The optional zone graph uses that authored topology as an attention bias; it does not add future state.

Active utility tokens: 14 channels per token

type_id, owner_side_id, x_norm, y_norm, z_norm, position_known, zone_id, utility_area_id, landing_zone_id, effect_id, target_site_id, blocked_route_id, age_norm, and remaining_norm.

The type vocabulary covers smoke, inferno, and decoy lifecycle tokens. Recent flash effects are represented in zone and player channels. Semantic fields capture authored utility areas, affected site, and the first blocked route where those mappings are available.

Recent event tokens: 11 channels per token

type_id, side_id, x_norm, y_norm, z_norm, position_known, zone_id, relative_time_norm, weapon_category_id, damage_health_norm, and headshot.

Event types cover player deaths and damage; bomb drop, pickup, plant, defuse, explosion, defuse start, and defuse abort; grenade throws and detonations; smoke start and expiry; and inferno start and expiry. Events are causally selected from the history available at each sampled tick.

Categorical values, masks, and excluded data

Categorical embeddings cover side, prior winner, buy type, half or overtime phase, bomb status, bomb commitment, site relation, zone control, utility type and semantics, event type, weapon category, zone, and zone group. Buy types distinguish eco, upgraded pistols, force buy, rifle buy, full buy, and pistol rounds. Weapon categories distinguish rifles, snipers, SMGs, shotguns, LMGs, pistols, grenades, and knives.

The model receives masks for valid history ticks, players, bomb, zones, utility slots, and event slots. It also learns temporal-position and token-family embeddings. Those embeddings are model parameters, not demo-derived features.

Player names, Steam IDs, team names, and team identities are deliberately excluded from every tensor and checked by the dataset QA. Demo ID, match-group ID, round number, prediction tick, split, and example_id are indexing or evaluation metadata, not model inputs.

The multi-task labels such as round outcome, future plant, next kill, contact zone, save, retake, defuse, and round-end reason are training targets. They are never included in the causal input window.

Quick start

Requirements

• Python 3.12
• uv
• Node.js 20.19+ or 22.12+ for the web frontend
• A CUDA GPU for practical neural training; parsing, tests, the baseline, and the workbench run locally without one

Install the Python environment and frontend dependencies:

uv sync --extra dev --extra neural --extra viz
cd web/app && npm install && cd ../..

For parsing and baseline work only, omit the neural and viz extras.

Run the investigation workbench

The workbench discovers artifacts below data/processed, data/neural, and artifacts/neural when it starts.

uv run evalbar web --open-browser

If web/app/dist does not exist, the command installs frontend dependencies when necessary and builds the application. Override artifact roots with --processed-root, --dataset-root, and --artifact-root.

For frontend development:

# terminal 1
uv run evalbar web --port 8765

# terminal 2
cd web/app
npm run dev

The Vite server proxies /api to http://127.0.0.1:8765.

Build the modeling pipeline

Place legally obtained Inferno demos under data/raw, then run:

# 1. Parse demos into canonical snapshots and semantic round state.
uv run evalbar parse data/raw --sample-rate-hz 4.0
uv run evalbar inspect data/processed/<demo_id>

# 2. Train the tabular reference model.
uv run evalbar train-baseline data/processed
uv run evalbar generate-timelines artifacts/baseline/<run_id>

# 3. Build the neural tensor dataset with a frozen validation population.
uv run evalbar build-neural-dataset data/processed \
  --prediction-stride 4 \
  --workers 8 \
  --write-splits \
  --val-groups data/config/holdout_groups.txt

# 4. Inspect and train the neural model.
uv run evalbar inspect-neural data/neural/<dataset_id>
uv run evalbar train-neural data/neural/<dataset_id> --preset tiny_pg

# 5. Render validation timelines.
uv run evalbar generate-neural-timelines artifacts/neural/<run_id>

Use uv run evalbar --help and uv run evalbar <command> --help for the complete CLI.

Web application

The FastAPI and React application connects model predictions to the underlying rounds. It combines a synchronized 2D replay, probability timeline, latent atlas, state inspector, generated findings, and same-example model comparisons. Selecting a point or finding opens the corresponding demo, round, and tick across every view.

Datasets, model runs, and atlases are discovered from configured artifact directories. The default view opens the demo and round with the largest cumulative probability movement, while investigation state is stored in the URL for reproducible links. Local mode can generate atlas artifacts through persisted jobs; public mode is read-only and hides filesystem paths.

Model architecture

Two predictors operate on the same processed rounds:

1. HistGradientBoostingClassifier, a fast aggregate-feature baseline.
2. GraphTemporalTransformer, the primary multi-task neural model.

The neural forward pass is:

1. Encode continuous and categorical fields per token family.
2. Apply geometric edge-bias attention over the ten players at each tick.
3. Optionally apply static map-topology attention over zone tokens.
4. Fuse all spatial tokens through learned Perceiver-style latent queries.
5. Process the 80 fused states with a temporal transformer.
6. Read the final valid tick through 13 binary and 6 categorical heads.

tiny_pg is the keeper preset. Player-skill features can be joined during dataset construction, but are disabled by default because they did not improve ct_win.

Results and experimental conclusions

Training runs and curves are available in the Evalbar W&B project.

The honest keeper result is the multi-seed distribution, not the best seed:

configuration	validation ct_win log-loss
tiny_pg, 5-seed single-model mean	0.534 +/- 0.012
best keeper seed	0.5148
4-5 seed probability ensemble	about 0.517

The seed standard deviation of 0.012 is larger than most measured configuration gaps. Treat a single-config delta below that value as no demonstrated effect and report at least five seeds.

A widened d_model=64, four-latent exploratory run reached 0.4948 on the 142-group dataset, but it is a single seed on an older hash-based split. It is not directly comparable to the frozen holdout results above and is not presented as the expected model score.

For keeper seed 1337, performance improves sharply as the round becomes determined:

phase	log-loss	accuracy
<15s	0.656	0.60
30-60s	0.571	0.68
>=90s	0.317	0.88

Adding independent match groups remains the strongest lever. A single-seed rebuild from roughly 95 to 142 groups improved the same keeper configuration from about 0.5155 to 0.5042, concentrated in early and mid-round states. The estimated irreducible log-loss is about 0.47 to 0.48.

Confirmed or practically null results:

• Player-skill features do not improve ct_win.
• The v8 bomb-timer features do not improve ct_win.
• Auxiliary-loss weights from 0.1 through 0.9 remain within seed noise.
• Zone-graph bias is neutral and disabled in scaled presets.
• Extra capacity overfits near 95 groups, but widening can help once the dataset reaches 142 groups. Those larger-model results remain single-seed and preliminary.

Permutation importance shows that ct_win relies mainly on global round state, player state, equipment/economy, and the bomb token. Raw aim, skill, bomb-timer, event, utility, and sound groups contribute approximately zero in the current model. Redundant position proxies mean sub-feature importance should be read as a lower bound; family-level permutations are more trustworthy.

Latent atlas and explanations

infra/latent_atlas.py captures the final temporal representation for every validation state, caches inference separately from projection, and emits a self-contained Plotly atlas or the artifacts consumed by the workbench.

Explore the hosted latent atlas without generating the artifacts locally.

uv sync --extra neural --extra viz
uv run python infra/latent_atlas.py \
  artifacts/neural/<run_id> \
  data/neural/<dataset_id> \
  --dump /tmp/states.parquet \
  --generate-explanations \
  --explanations-out /tmp/explanations.parquet

Use --refresh-cache to repeat model inference and --refresh-projection to recompute only the projection. When evaluating a checkpoint on a newer dataset, exclude every match group from its training dataset:

uv run python infra/latent_atlas.py <run_dir> <new_dataset_dir> \
  --exclude-groups-from <checkpoint_training_dataset_dir>

UMAP geometry is used for visualization, not cluster identity. Cluster summaries are computed in the original latent space. Independently trained model embeddings are not aligned and are shown as separate atlas views.

Reproducible evaluation

Always build comparable experiments with a frozen validation-group file:

uv run python scripts/pick_holdout_groups.py data/neural/<dataset_id>/examples.parquet

Pass the resulting file to build-neural-dataset --val-groups. Without a fixed manifest, adding demos reshuffles the hash-based train/validation assignment and invalidates direct comparisons.

On CUDA, training selects bf16 autocast automatically while weights and optimizer state remain fp32. Dataset QA is skipped by default after construction; pass --qa to re-run it. torch.compile is experimental because variable batch shapes trigger repeated compilation and can stall validation.

GPU workflow

The Mac remains the controller and a short-lived Vast.ai instance is only a GPU executor:

infra/bootstrap.sh       # one-time local setup
infra/vast.sh up         # rent, provision, clone, install, and fetch the dataset
infra/vast.sh status
infra/vast.sh sync       # synchronize local src/ changes
infra/vast.sh ssh -- <command>
infra/vast.sh down       # destroy the instance and stop billing

See infra/SETUP.md for the required read-only credentials, R2 dataset setup, cost controls, and threat model. Always use down when the session is finished; the workflow is designed for zero idle cost.

Repository layout

data/raw/             local input demos (gitignored)
data/processed/       parsed snapshots and semantic rounds (gitignored)
data/neural/          tensor datasets and split metadata (gitignored)
data/config/maps/     checked-in map semantics and topology
artifacts/            trained models and generated outputs (gitignored)
infra/                GPU lifecycle, ensembles, importance, and atlas tools
scripts/              demo, ratings, and holdout utilities
src/evalbar/          Python package and CLI
src/evalbar/web/      FastAPI registry and investigation API
web/app/              React, TypeScript, deck.gl, and ECharts frontend
tests/                pipeline, model, artifact, and API tests
.evalbar/             local SQLite state and job logs (gitignored)

Testing

# Python pipeline, model, registry, API, comparison, and job tests
uv run pytest

# React state and panel tests
cd web/app
npm test

# TypeScript check and production frontend build
npm run build

The test suite uses synthetic fixtures. It does not require the private demo corpus, trained checkpoints, W&B access, or a rented GPU.