feat: Migrate from pandas to polars

README.md

@@ -13,7 +13,7 @@ This package is a simple order book matching engine implementation in Python. It
 - price-time priority
 - limit and market orders
 - order cancellation and expiration
-- conversion into pandas DataFrame of orders, executed trades, order book summary
+- conversion into polars LazyFrame of orders, executed trades, order book summary
 
 ## Install
 
@@ -30,7 +30,7 @@ pip install order-matching
 ```python
 >>> from datetime import datetime, timedelta
 >>> from pprint import pp
->>> import pandas as pd
+>>> import polars as pl
 
 >>> from order_matching.matching_engine import MatchingEngine
 >>> from order_matching.order import LimitOrder

openspec/changes/archive/2026-07-02-migrate-pandas-to-polars/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-02

openspec/changes/archive/2026-07-02-migrate-pandas-to-polars/design.md

@@ -0,0 +1,160 @@
+## Context
+
+The OrderBookMatchingEngine currently uses pandas for DataFrame operations in three key areas:
+1. `Orders.to_frame()` - converting order collections to DataFrames
+2. `ExecutedTrades.to_frame()` - converting trade collections to DataFrames
+3. `OrderBook.summary()` - generating order book summary DataFrames
+
+All DataFrames are validated using pandera schemas (`OrderDataSchema`, `TradeDataSchema`, `OrderBookSummarySchema`) that define column types, constraints, and validation rules. These schemas are critical for downstream code and analysis pipelines.
+
+**Current Implementation:**
+- Uses `pandas.DataFrame.from_records()` to convert dataclass instances
+- Uses pandas-specific operations like `.assign()` with lambda functions
+- Type hints use `pandera.typing.pandas.DataFrame[Schema]`
+- Schema validation uses pandera's pandas backend
+
+**Constraints:**
+- Must maintain 100% backward compatibility with existing DataFrame schemas
+- Cannot break existing tests or downstream code that consumes these DataFrames
+
+## Goals / Non-Goals
+
+**Goals:**
+- Replace pandas with polars for all DataFrame conversion operations
+- Maintain identical DataFrame schemas (column names, types, constraints)
+- Preserve all pandera validation behavior
+- Improve performance through polars' optimized operations
+- Update type hints to reflect polars usage while keeping schema validation
+- Use LazyFrame and lazy evaluation whenever possible
+
+**Non-Goals:**
+- Changing the pandera schema definitions or validation rules
+- Modifying the API surface (method signatures stay the same)
+- Migrating internal data structures from dataclasses
+- Supporting both pandas and polars simultaneously (full migration only)
+- Changing behavior of existing methods beyond the DataFrame type
+
+## Decisions
+
+### Decision 1: Use polars DataFrame construction directly
+**Rationale:** Polars provides native DataFrame construction from dictionaries and records. Instead of using pandas `from_records()`, we'll use `pl.LazyFrame()` constructor.
+
+**Implementation:**
+```python
+# Before (pandas):
+pd.DataFrame.from_records([asdict(order) for order in orders])
+
+# After (polars):
+pl.LazyFrame([asdict(order) for order in orders])
+```
+
+**Alternatives considered:**
+- Converting pandas to polars via arrow - rejected as unnecessary intermediate step
+
+### Decision 2: Handle type conversions with polars expressions
+**Rationale:** Pandas uses `.assign()` with lambda functions for type conversions. Polars uses `.with_columns()` and expressions.
+
+**Implementation:**
+```python
+# Before (pandas):
+.assign(
+    side=lambda df: df['side'].astype(str),
+    timestamp=lambda df: pd.to_datetime(df['timestamp'])
+)
+
+# After (polars):
+.with_columns([
+    pl.col('side').cast(pl.Utf8),
+    pl.col('timestamp').cast(pl.Datetime)
+])
+```
+
+**Alternatives considered:**
+- Keep using assign-style API via polars extensions - rejected for being less idiomatic
+
+### Decision 3: Update pandera imports to support polars
+**Rationale:** Pandera 0.17.0+ supports both pandas and polars backends through separate import paths.
+
+**Implementation:**
+```python
+# Before:
+from pandera.pandas import DataFrameModel
+from pandera.typing.pandas import DataFrame
+
+# After:
+from pandera.polars import DataFrameModel
+from pandera.typing.polars import DataFrame
+from pandera.typing.polars import LazyFrame
+```
+
+**Alternatives considered:**
+- Maintaining dual compatibility - rejected per non-goals (full migration only)
+- Creating polars-specific schema copies - rejected to avoid duplication
+
+### Decision 4: Handle empty DataFrames with explicit schema
+**Rationale:** Empty polars DataFrames need explicit schema definition to maintain type information.
+
+**Implementation:**
+```python
+# Empty DataFrame case:
+if len(collection) == 0:
+    return pl.DataFrame(schema={
+        'column1': pl.Float64,
+        'column2': pl.Utf8,
+        # ... all columns
+    })
+```
+
+**Alternatives considered:**
+- Returning empty DataFrame without schema - rejected because pandera validation would fail
+- Using schema inference from empty records - rejected as unreliable
+
+## Risks / Trade-offs
+
+**Risk:** Pandera polars support may have edge cases or bugs
+- **Mitigation:** Comprehensive test suite covers all schema validation scenarios. Run full test suite before merging.
+
+**Risk:** Polars API differences may cause subtle behavioral changes
+- **Mitigation:** Carefully review all DataFrame operations and validate output schemas match exactly. Add explicit type casts where needed.
+
+**Risk:** Breaking change for downstream code expecting pandas DataFrames
+- **Mitigation:** This is intentional per requirements. Document clearly in CHANGELOG and update README examples.
+
+**Trade-off:** Lose pandas ecosystem compatibility
+- **Impact:** Code using these DataFrames must work with polars instead of pandas
+- **Benefit:** Significantly better performance, especially for large order books
+
+**Trade-off:** Increased dependency on polars library maturity
+- **Impact:** Polars is newer than pandas, potential for API changes
+- **Benefit:** Polars is stable (1.0+ released), actively developed, and widely adopted
+
+## Migration Plan
+
+**Phase 1: Update schemas**
+1. Change imports in `src/order_matching/schemas.py` from pandas to polars
+2. Update type hints to use `pandera.typing.polars.DataFrame`
+3. Verify schemas still define the same constraints
+
+**Phase 2: Update conversion methods**
+1. Update `Orders.to_frame()` in `orders.py`
+2. Update `ExecutedTrades.to_frame()` in `executed_trades.py`
+3. Update `OrderBook.summary()` in `order_book.py`
+4. Ensure each method handles empty cases with explicit schemas
+
+**Phase 3: Update tests**
+1. Run existing test suite - most should still pass
+2. Fix any tests that check for pandas-specific behavior
+3. Add new tests if needed for polars-specific edge cases
+
+**Phase 4: Validate**
+1. Run full test suite: `uv run pytest`
+2. Run linter and formatter: `uv run prek run -v --show-diff-on-failure --all-files`
+
+**Rollback strategy:**
+- If tests fail after updates, revert changes file by file
+- Git branch allows clean rollback to pandas implementation
+- No database or persistent state changes, so rollback is safe
+
+## Open Questions
+
+None - requirements and approach are clear.

openspec/changes/archive/2026-07-02-migrate-pandas-to-polars/proposal.md

@@ -0,0 +1,36 @@
+## Why
+
+The project currently uses pandas for DataFrame operations in orders, trades, and order book summaries.
+Polars offers significantly better performance and a more modern API, but the migration must maintain 100% schema compatibility with existing pandera validations to avoid breaking downstream code or analysis pipelines.
+
+## What Changes
+
+- Replace pandas DataFrame operations with polars equivalents across all data conversion methods
+- Maintain pandera schema validation compatibility - all schemas must be converted to work polars DataFrames
+- Update `to_frame()` methods in `Orders`, `ExecutedTrades`, and `OrderBook` classes to return polars DataFrames
+- Ensure column names, dtypes, and constraints remain identical to current pandera-validated schemas
+- Update type hints from `pandera.typing.pandas.DataFrame` to support polars while preserving schema validation
+
+## Capabilities
+
+### New Capabilities
+- `polars-dataframe-conversion`: Converting internal data structures (Orders, ExecutedTrades, OrderBook) to polars DataFrames with schema validation
+- `pandera-polars-validation`: Schema validation using pandera with polars backend to ensure schema compatibility
+
+### Modified Capabilities
+
+## Impact
+
+**Code:**
+- `src/order_matching/orders.py`: Update `to_frame()` method to use polars
+- `src/order_matching/executed_trades.py`: Update `to_frame()` method to use polars
+- `src/order_matching/order_book.py`: Update `summary()` method to use polars
+- `src/order_matching/schemas.py`: Update schema definitions to support polars DataFrames
+
+**Tests:**
+- All existing tests that validate DataFrame schemas must continue to pass
+- Tests in `tests/test_order.py`, `tests/test_trade.py`, `tests/test_executed_trades.py` that check DataFrame output
+- Documentation examples in README.md that demonstrate DataFrame usage
+
+**API:**
+- Return type changes from pandas DataFrame to polars DataFrame, but schema validation ensures column names and dtypes remain compatible

openspec/changes/archive/2026-07-02-migrate-pandas-to-polars/specs/pandera-polars-validation/spec.md

@@ -0,0 +1,51 @@
+## ADDED Requirements
+
+### Requirement: Validate polars DataFrames with pandera schemas
+The system SHALL use pandera to validate polars DataFrames against existing schema definitions (`OrderDataSchema`, `TradeDataSchema`, `OrderBookSummarySchema`).
+
+#### Scenario: Valid polars DataFrame passes schema validation
+- **WHEN** a polars DataFrame with correct schema is validated against `OrderDataSchema`, `TradeDataSchema`, or `OrderBookSummarySchema`
+- **THEN** validation succeeds without errors
+
+#### Scenario: Invalid polars DataFrame fails schema validation
+- **WHEN** a polars DataFrame with incorrect schema (wrong columns, wrong types, or constraint violations) is validated
+- **THEN** validation raises a pandera validation error with details
+
+### Requirement: Maintain schema constraint validation
+The system SHALL validate all pandera Field constraints (gt, ge, isin, unique, nullable) on polars DataFrames.
+
+#### Scenario: Price constraint validation
+- **WHEN** a polars DataFrame contains price values <= 0
+- **THEN** validation against schemas with `price: Field(gt=0)` fails
+
+#### Scenario: Enum value constraint validation
+- **WHEN** a polars DataFrame contains invalid enum values in side/execution/status columns
+- **THEN** validation against schemas with `Field(isin=[...])` fails
+
+#### Scenario: Unique constraint validation
+- **WHEN** a polars DataFrame for OrderBookSummary contains duplicate price values
+- **THEN** validation against `OrderBookSummarySchema` with `price: Field(unique=True)` fails
+
+#### Scenario: Nullable constraint validation
+- **WHEN** a polars DataFrame contains null values in non-nullable columns
+- **THEN** validation fails
+- **WHEN** a polars DataFrame contains null values in nullable columns (e.g., expiration)
+- **THEN** validation succeeds
+
+### Requirement: Update schema type hints for polars
+The system SHALL update type hints in schema definitions to support both pandas and polars DataFrames via pandera's typing system.
+
+#### Scenario: Schema works with pandera.typing.polars.DataFrame and pandera.typing.polars.LazyFrame
+- **WHEN** schemas are imported and used with polars DataFrames
+- **THEN** type hints correctly indicate polars DataFrame compatibility
+
+### Requirement: Preserve strict mode behavior
+The system SHALL maintain strict mode validation configured in schema Config classes when validating polars DataFrames.
+
+#### Scenario: Strict schema rejects extra columns
+- **WHEN** a polars DataFrame with extra columns is validated against a schema with `strict = True`
+- **THEN** validation fails
+
+#### Scenario: Strict schema accepts exact columns
+- **WHEN** a polars DataFrame with exact required columns is validated against a schema with `strict = True`
+- **THEN** validation succeeds

openspec/changes/archive/2026-07-02-migrate-pandas-to-polars/specs/polars-dataframe-conversion/spec.md

@@ -0,0 +1,58 @@
+## ADDED Requirements
+
+### Requirement: Convert Orders to polars DataFrame
+The system SHALL convert `Orders` objects to polars DataFrames using the `to_frame()` method with all columns matching the `OrderDataSchema`.
+
+#### Scenario: Convert empty Orders to DataFrame
+- **WHEN** `to_frame()` is called on an empty `Orders` object
+- **THEN** system returns an empty polars DataFrame
+
+#### Scenario: Convert populated Orders to DataFrame
+- **WHEN** `to_frame()` is called on `Orders` containing one or more orders
+- **THEN** system returns a polars DataFrame with columns: timestamp, expiration, order_id, trader_id, side, execution, status, price, size, price_number_of_digits
+- **THEN** all string enum fields (side, execution, status) are converted to string representations
+- **THEN** timestamp and expiration columns are datetime type
+
+### Requirement: Convert ExecutedTrades to polars DataFrame
+The system SHALL convert `ExecutedTrades` objects to polars DataFrames using the `to_frame()` method with all columns matching the `TradeDataSchema`.
+
+#### Scenario: Convert empty ExecutedTrades to DataFrame
+- **WHEN** `to_frame()` is called on an empty `ExecutedTrades` object
+- **THEN** system returns an empty polars DataFrame
+
+#### Scenario: Convert populated ExecutedTrades to DataFrame
+- **WHEN** `to_frame()` is called on `ExecutedTrades` containing one or more trades
+- **THEN** system returns a polars DataFrame with columns: timestamp, incoming_order_id, book_order_id, trade_id, side, execution, price, size
+- **THEN** string enum fields (side, execution) are converted to string representations
+- **THEN** timestamp column is datetime type
+
+### Requirement: Convert OrderBook to polars DataFrame summary
+The system SHALL convert `OrderBook` objects to polars DataFrames using the `summary()` method with all columns matching the `OrderBookSummarySchema`.
+
+#### Scenario: Convert empty OrderBook to summary DataFrame
+- **WHEN** `summary()` is called on an empty `OrderBook`
+- **THEN** system returns an empty polars DataFrame
+
+#### Scenario: Convert OrderBook with bids and offers to summary
+- **WHEN** `summary()` is called on `OrderBook` containing bids and/or offers
+- **THEN** system returns a polars DataFrame with columns: side, price, size, count
+- **THEN** each unique price level is represented as a single row
+- **THEN** count column is integer type
+- **THEN** side column contains "BUY" or "SELL" string values
+
+### Requirement: Maintain DataFrame schema compatibility
+The system SHALL ensure polars DataFrames produced by conversion methods have identical column names, dtypes, and value formats as the pandas DataFrames they replace.
+
+#### Scenario: Column names match exactly
+- **WHEN** any conversion method produces a polars DataFrame
+- **THEN** column names MUST match the corresponding pandera schema field names exactly
+
+#### Scenario: Data types are compatible
+- **WHEN** any conversion method produces a polars DataFrame
+- **THEN** numeric columns (price, size, count) use appropriate numeric types
+- **THEN** datetime columns use polars datetime type
+- **THEN** string columns (order_id, trader_id, trade_id, side, execution, status) use string type
+
+#### Scenario: Empty DataFrames have correct schema
+- **WHEN** conversion methods are called on empty collections
+- **THEN** resulting empty DataFrames MUST still have the correct column names and types defined

openspec/changes/archive/2026-07-02-migrate-pandas-to-polars/tasks.md

@@ -0,0 +1,57 @@
+## 1. Update Schema Definitions
+
+- [x] 1.1 Update imports in src/order_matching/schemas.py from `pandera.pandas` to `pandera.polars`
+- [x] 1.2 Update DataFrame type import from `pandera.typing.pandas` to `pandera.typing.polars`
+- [x] 1.3 Ensure all Field constraints (gt, ge, isin, unique, nullable) are preserved
+- [x] 1.4 Verify Config classes with strict=True remain unchanged
+- [x] 1.5 Run mypy to check type annotations are correct
+
+## 2. Update Orders.to_frame() Method
+
+- [x] 2.1 Replace pandas import with polars in src/order_matching/orders.py
+- [x] 2.2 Update empty DataFrame case to return polars DataFrame with explicit OrderDataSchema schema
+- [x] 2.3 Replace `pd.DataFrame.from_records()` with `polars.DataFrame()` constructor
+- [x] 2.4 Replace `.assign()` with `.with_columns()` for type conversions (side, execution, status to string)
+- [x] 2.5 Update timestamp and expiration columns to use polars datetime casting
+- [x] 2.6 Update return type hint to `DataFrame[OrderDataSchema]` using polars typing
+- [x] 2.7 Verify method signature remains unchanged (backward compatible API)
+
+## 3. Update ExecutedTrades.to_frame() Method
+
+- [x] 3.1 Replace pandas import with polars in src/order_matching/executed_trades.py
+- [x] 3.2 Update empty DataFrame case to return polars DataFrame with explicit TradeDataSchema schema
+- [x] 3.3 Replace `pd.DataFrame.from_records()` with `polars.DataFrame()` constructor
+- [x] 3.4 Replace `.assign()` with `.with_columns()` for type conversions (side, execution to string)
+- [x] 3.5 Update timestamp column to use polars datetime casting
+- [x] 3.6 Update return type hint to `DataFrame[TradeDataSchema]` using polars typing
+- [x] 3.7 Verify method signature remains unchanged
+
+## 4. Update OrderBook.summary() Method
+
+- [x] 4.1 Replace pandas import with polars in src/order_matching/order_book.py
+- [x] 4.2 Replace pandas DataFrame construction in summary() with polars DataFrame
+- [x] 4.3 Update count column casting to use polars integer type
+- [x] 4.4 Replace `pd.concat()` with polars concatenation (use `pl.concat()`)
+- [x] 4.5 Update return type hint to `DataFrame[OrderBookSummarySchema]` using polars typing
+- [x] 4.6 Verify empty OrderBook case returns empty polars DataFrame
+- [x] 4.7 Update `get_imbalance()` method to work with polars DataFrames
+
+## 5. Run Tests and Validate
+
+- [x] 5.1 Run full test suite: `uv run pytest`
+- [x] 5.2 Fix any tests in tests/test_order.py that check pandas-specific behavior
+- [x] 5.3 Fix any tests in tests/test_trade.py that check pandas-specific behavior
+- [x] 5.4 Fix any tests in tests/test_executed_trades.py that check pandas-specific behavior
+- [x] 5.5 Verify all pandera schema validations still pass
+- [x] 5.6 Check test coverage remains at current level
+
+## 6. Type Checking and Linting
+
+- [x] 6.1 Run type checker: `uv run prek run -v --show-diff-on-failure --all-files`
+- [x] 6.2 Fix any type errors related to polars DataFrame usage
+
+## 7. Documentation and Examples
+
+- [x] 7.1 Update README.md code examples to reflect polars DataFrames
+- [x] 7.2 Update any docstrings that mention pandas to reference polars
+- [x] 7.3 Verify documentation builds successfully: `uv run mkdocs build`