Probabilistic Chess Outcome Prediction: MAP vs Posterior Predictive Estimation (PPE)

This project investigates whether posterior predictive estimation (PPE) provides measurable benefits over MAP estimation for multiclass logistic regression on a real-world prediction task: forecasting chess game outcomes (White win / Black win / Draw) using only early game information.

Problem

Given features available at game start (player ratings, time controls, and opening family), predict the game outcome as a probability distribution over three classes.

Data

• Source: Lichess Chess Game Dataset (Kaggle)
• Filtered to games ending in checkmate or draw (excluded resignations/timeouts)
• Final dataset size: 6,820 games (after filtering + de-duplication)
• Notable property: class imbalance, with draws as the rarest class

Methods

• Model: Multiclass logistic regression with Categorical (softmax) likelihood
• Baseline: MAP estimation with a Gaussian prior on weights
  • MAP objective optimized via gradient descent (implemented from scratch)
• Upgrade: Posterior Predictive Estimation (PPE)
  • Approximate posterior over weights using Random Walk Metropolis MCMC (PyMC)
  • Compute predictions by averaging softmax probabilities over posterior samples
• Metric: Average held-out log-likelihood (overall + per-class)

Results

Average test-set log-likelihood (higher is better):

Method	Overall	White	Black	Draw
Random chance	-1.099	-1.099	-1.099	-1.099
Base-rate	-0.980	-0.769	-0.886	-2.087
MAP	-0.894	-0.699	-0.762	-2.061
PPE	-0.893	-0.699	-0.762	-2.056

Key Insight

PPE did not yield significant gains over MAP in this setting. Diagnostics and qualitative comparisons suggest the posterior over weights was tightly concentrated around the MAP estimate, so posterior averaging produced predictions nearly identical to a point estimate. This highlights that PPE is most beneficial when posterior uncertainty is meaningfully broader or asymmetric.

Repository Contents

• notebooks/01_data_prep.ipynb: data exploration + preprocessing
• notebooks/02_run_map.ipynb: MAP training + evaluation
• notebooks/03_run_mcmc_ppe.ipynb: MCMC sampling + posterior predictive evaluation
• src/LogisticRegressionMAPEstimator.py: MAP logistic regression implementation
• src/FeatureTransformPolynomial.py: feature transform utility

How to Run

1. Install dependencies: pip install -r requirements.txt
2. Run notebooks in order:
   • 01_data_prep.ipynb
   • 02_run_map.ipynb
   • 03_run_mcmc_ppe.ipynb

Collaboration

This project was completed as a two-person collaboration. My primary contributions included implementing the MAP estimator, training, and evaluation, setting up the PPE computation and evaluation, and leading the comparative analysis and interpretation.