optimalgiv

A Python wrapper for the Julia package OptimalGIV.jl

This wrapper uses PythonCall.jl to call the Julia package directly from Python. Julia is automatically installed and all dependencies are resolved without manual setup.

This python package is under active development The core algorithms are implemented in Julia, and thoroughly tested under simulations, but documentations are working in progress, and bugs may exists for minor features. Feature requests and bug reports are welcomed.

This README focuses on the use for Python API. For more technical documentation, please see the Julia package and the companion paper.

The multithreading capacity from PythonCall.jl is experimental. We recommend to set import os; os.environ["PYTHON_JULIACALL_HANDLE_SIGNALS"] = "yes" before import this package to avoid segfault or other crashes. See this page for details.

Installation

pip install optimalgiv

First import

The first time you run:

import optimalgiv as og

it will:

1. Install Julia (if not present; ≈ 1-2 min),
2. Set up a Julia environment with OptimalGIV.jl and precompile (≈ 2–4 min).

Later imports will be much faster (≈ 6–10 s), which is typical for Julia project activation—the environment is compiled once and then reused.

---

Model Specification

The Granular Instrumental Variables (GIV) model estimated by this package follows the specification:

$$\begin{aligned} \left.\begin{array}{c} \begin{array}{cl} q_{i,t} & =-p_{t}\times\mathbf{C}_{i,t}'\boldsymbol{\zeta}+\mathbf{X}_{i,t}'\boldsymbol{\beta}+u_{i,t},\\\ 0 & =\sum_{i}S_{i,t}q_{i,t} \end{array}\end{array}\right\} \implies & p_{t}=\frac{1}{\mathbf{C}_{S,t}'\boldsymbol{\zeta}}\left[\mathbf{X}_{S,t}'\boldsymbol{\beta}+u_{S,t}\right], \end{aligned}$$

where:

• $q_{i,t}$ and $p_t$ are endogenous,
• $\mathbf{C}_{i,t}$ is a vector of controls for slopes,
• $\mathbf{X}_{i,t}$ is a vector of controls,
• $\boldsymbol{\zeta}$, $\boldsymbol{\beta}$ are coefficient vectors,
• $u_{i,t}$ is the idiosyncratic shock, and
• $S_{i,t}$ is the weighting variable.

The equilibrium price $p_t$ is derived by imposing the market clearing condition and the model is estimated using the moment condition:

$$ \mathbb{E}[u_{i,t} u_{j,t}] = 0 $$

for all $i \neq j$. This implies orthogonality across sectors' residuals.

---

Panel Data and Coverage

The GIV model supports unbalanced panel data. However, some estimation algorithms (e.g. "scalar_search" and "debiased_ols") require complete coverage, meaning:

$$ \sum_i S_{i,t} q_{i,t} = 0 $$

must hold exactly within the sample. This ensures internal consistency of the equilibrium condition.

If the adding-up constraint is not satisfied, the model will adjust accordingly, but the interpretation of estimated coefficients should be made with caution, as residual market imbalances may bias elasticities and standard errors. (See the complete_coverage argument below for details.)

---

Internal PC

Internal PC extractions are supported. With internal PCs, the moment conditions become:

$$ \mathbb E[u_{i,t}u_{j,t}] = \Lambda \Lambda' $$

where $\Lambda$ is the factor loadings estimated internally using HeteroPCA.jl from $u_{i,t}(z) \equiv q_{i,t} + p_{t}\times\mathbf{C}_{i,t}'\boldsymbol{z}$ at each guess of $z$.

However, with small samples, the exactly root solving the moment condition may not exist, and users may want to use an minimizer to minimize the error instead. Also, be noted that a model with fully flexible elasticity specification and fully flexible factor loadings is not theoretically identifiable.

---

Usage

Basic Example

import pandas as pd
import numpy as np
from optimalgiv import giv

df = pd.read_csv("./simdata1.csv") # you can find simdata under the git repo examples/
# or simulate using simulate_data below

df['id'] = df['id'].astype('category') # ensure id interactions map to distinct groups

# Define the model formula
formula = "q + id & endog(p) ~ 0 + fe(id) + fe(id) & (η1 + η2)"

# Provide an initial guess (a good guess is critical)
guess = np.ones(5)

# Estimate the model
model = giv(
    df = df,
    formula = "q + id & endog(p) ~ 0 + fe(id) + fe(id) & (η1 + η2)",
    id = "id",
    t = "t",
    weight = "absS",
    algorithm = "iv",
    guess = guess,
    save = 'all', # saves both fixed‐effects (model.fe) and residuals (model.residual_df)
)

# View the result
model.summary()

##                     GIVModel (Aggregate coef: 2.13)                     
## ─────────────────────────────────────────────────────────────────────────
##            Estimate  Std. Error    t-stat  Pr(>|t|)  Lower 95%  Upper 95%
## ─────────────────────────────────────────────────────────────────────────
## id: 1 & p  1.00723     1.30407   0.772377    0.4405  -1.55923    3.57369
## id: 2 & p  1.77335     0.475171  3.73204     0.0002   0.8382     2.70851
## id: 3 & p  1.36863     0.382177  3.58114     0.0004   0.616491   2.12077
## id: 4 & p  3.3846      0.382352  8.85207     <1e-16   2.63212    4.13709
## id: 5 & p  0.619882    0.161687  3.83385     0.0002   0.301676   0.938087

---

Formula Specification

The model formula follows the convention:

q + interactions & endog(p) ~ exog_controls + pc(k)

Where:

• q: Response variable (e.g., quantity).

• endog(p): Endogenous variable (e.g., price). Must appear on the left-hand side.

  Note: A positive estimated coefficient implies a negative response of q to p (i.e., a downward-sloping demand curve).

• interactions: Exogenous variables used to parameterize heterogeneous elasticities, such as entity identifiers or group characteristics.

• exog_controls: Exogenous control variables. Supports fixed effects (e.g., fe(id)) using the same syntax as FixedEffectModels.jl.

• pc(k): Principal component extraction with k factors (optional). When specified, k common factors are extracted from residuals using HeteroPCA.jl

Examples of formulas:

# Homogeneous elasticity with entity-specific loadings (estimated) and fixed effects (absorbed)
formula = "q + endog(p) ~ id & η + fe(id)"

# Heterogeneous elasticity by entity
formula = "q + id & endog(p) ~ id & η + fe(id)"

# Multiple interactions
formula = "q + id & endog(p) + category & endog(p) ~ fe(id) & η1 + η2"

formula = "q + id & endog(p) ~ 0 + id & η"

# With PC extraction (2 factors)
formula = "q + endog(p) ~ 0 + pc(2)"

# exogneous controls with PC extraction
formula = "q + endog(p) ~ fe(id) & η1 + pc(3)"

---

Key Function: giv()

giv(df, formula: str, id: str, t: str, weight: str, **kwargs) -> GIVModel

Required Arguments

• df: pandas.DataFrame containing panel data. Must be balanced for some algorithms (e.g., scalar_search).
• formula: A string representing the model (Julia-style formula syntax). See examples above.
• id: Name of the column identifying entities (e.g., "firm_id").
• t: Name of the time variable column.
• weight: Name of the weight/size column (e.g., market shares S_i,t).

Keyword Arguments (Optional)

• algorithm: One of "iv" (default), "iv_twopass", "debiased_ols", or "scalar_search".

• guess: Initial guess for ζ coefficients. (See below for usage details)

• exclude_pairs: Dictionary excluding pairs from moment conditions.

  • Example: {1: [2, 3], 4: [5]} excludes entity pair with code (1,2), (1,3), and (4,5) from the moment conditions entering the estimation.

• quiet: Set True to suppress warnings and info messages.

• save: "none" (default), "residuals", "fe", or "all" — controls what is stored on the returned model:

  • "none": neither residuals nor fixed-effects are saved
  • "residuals": saves residuals in model.residual_df
  • "fe": saves fixed-effects in model.fe
  • "all": saves both model.residual_df and model.fe

• save_df: If True, the full estimation dataframe (with residuals, coefficients, fixed effects) is stored in model.df.

• complete_coverage: Whether the dataset covers the full market in each time period, meaning $\sum_i S_{i,t} q_{i,t} = 0$ holds exactly within the sample.

  • Default is None, which triggers auto-detection: the model checks this condition period-by-period and sets the flag to True or False accordingly.
  • If the condition does not hold (False), you can still force estimation by setting quiet=True, but results may be biased. Use with caution.
  • Required for "scalar_search" and "debiased_ols" algorithms.

• return_vcov: Whether to compute and return the variance–covariance matrices. (default: True)

• tol: Convergence tolerance for the solver (: 1e-6)

• iterations: Maximum number of solver iterations (: 100)

• pca_option: Dictionary of options for PC extraction when using pc(k) in formula:

  • 'algorithm': HeteroPCA algorithm - DeflatedHeteroPCA, 'StandardHeteroPCA', or 'DiagonalDeletion'
  • 'impute_method': 'zero' or 'pairwise' for handling missing values (default: 'zero')
  • 'demean': Whether to demean data before PCA (default: False)
  • 'maxiter': Maximum iterations for PCA algorithm (default: 100)

Advanced keyword arguments (Optional; Use with caution)

• solver_options (Dict[str, Any]) Extra options passed to the nonlinear system solver from NLsolve.jl. The Python dict is converted to a Julia NamedTuple with keyword-style arguments. Common options include:

  • "method": "newton" , "anderson", "trust_region", etc.
  • "ftol": absolute residual tolerance
  • "xtol": absolute solution tolerance
  • "iterations": max iterations
  • "show_trace": verbose output

  Example:

  solver_opts = {
      "method": "newton",
      "ftol": 1e-8,
      "xtol": 1e-8,
      "iterations": 1000,
      "show_trace": True,
  }
  
  model = giv(df, formula, id="id", t="t", solver_options=solver_opts)

  For the full list of options, see the NLsolve.jl documentation.

---

Algorithms

The package implements four algorithms for GIV estimation:

1. "iv"

   • Default, recommended
   • Uses moment condition $$(\mathbb{E}[u_i,u_{S,-i}]=0)$$
   • $$O(N)$$ implementation
   • Supports exclude_pairs (exclude certain pairs $E[u_i u_j] = 0$ from the moment conditions)
   • Supports flexible elasticity specs, unbalanced panels

2. "iv_twopass": Numerically identical to iv but uses a more straightforward O(N²) implementation with two passes over entity pairs. This is useful for:

   • Debugging purposes
   • When the O(N) optimization in iv might cause numerical issues
   • When there are many pairs to be excluded, which will slow down the algorithm in iv
   • Understanding the computational flow of the moment conditions

3. "debiased_ols"

   • Uses $$\mathbb{E}[u_iC_{it}p_{it}] = \sigma_i^2 / \zeta_{St}$$
   • Requires complete market coverage
   • More efficient but restrictive

4. "scalar_search"

   • Finds a single aggregate elasticity
   • Requires balanced panel, constant weights, complete coverage
   • Useful for diagnostics or initial-guess formation

---

Initial Guesses

A good guess is key to stable estimation. If omitted, OLS‐based defaults will typically fail. Examples:

import numpy as np
from optimalgiv import giv
# 1) Scalar guess (for homogeneous elasticity)
guess = 1.0
model1 = giv(
    df,
    "q + endog(p) ~ n1 + fe(id)",
    id="id", t="t", weight="S",
    guess=guess
)

# 2) Dict by group name (heterogeneous by id)
guess = {"id": [1.2, 0.8]}
model2 = giv(
    df,
    "q + id & endog(p) ~ 1",
    id="id", t="t", weight="S",
    guess=guess
)

# 3) Dict for multiple interactions
guess = {
    "id": [1.0, 0.9],
    "n1": [0.5, 0.3]
}
model3 = giv(
    df,
    "q + id & endog(p) + n1 & endog(p) ~ fe(id)",
    id="id", t="t", weight="S",
    guess=guess
)

# 4) Dict keyed by exact coefnames
names = model3.coefnames
guess = {name: 0.1 for name in names}
model4 = giv(
    df,
    "q + id & endog(p) + n1 & endog(p) ~ fe(id)",
    id="id", t="t", weight="S",
    guess=guess
)

# 5) Scalar-search with heterogeneous formula
guess = {"Aggregate": 2.5}
model5 = giv(
    df,
    "q + id & endog(p) ~ 0 + fe(id) + fe(id)&(n1 + n2)",
    id="id", t="t", weight="S",
    algorithm="scalar_search",
    guess=guess
)

# 6) Use estimated ζ from model5 as initial guess
guess = model5.endog_coef
model6 = giv(
    df,
    "q + id & endog(p) ~ 0 + fe(id) + fe(id)&(n1 + n2)",
    id="id", t="t", weight="S",
    guess=guess
)

---

Principal Components (PC) in Formulas

The package supports extracting principal components from residuals to capture unobserved factors:

# Add pc(k) to the formula to extract k principal components
model = giv(
    df,
    "q + endog(p) ~ fe(id) + pc(2)",  # Extract 2 PCs from residuals
    id="id", t="t", weight="S",
    save_df=True  # Needed to access PC factors/loadings in df
)

# Access PC results
model.n_pcs          # Number of PCs extracted
model.pc_factors     # k×T matrix of time factors
model.pc_loadings    # N×k matrix of entity loadings
model.pc_model       # HeteroPCAModel object with details

Internal PCA

Internal PC extractions are supported. With internal PCs, the moment conditions become $\mathbb E[u_{i,t}u_{j,t}] = \Lambda \Lambda'$, where $\Lambda$ is the factor loadings estimated internally using HeteroPCA.jl from $u_{i,t}(z) \equiv q_{i,t} + p_{t}\times\mathbf{C}_{i,t}'\boldsymbol{z}$ at each guess of $z$. However, following caveats apply:

• With internal PC extraction, the weighting scheme is no longer optimal as it does not consider the covariance in the moment conditions due to common factor estimation. The standard error formula also no longer applies and hence was not returned. One can consider bootstrapping for statistical inference;

• In small samples, the exactly root solving the moment condition may not exist, and users may want to use an minimizer to minimize the error instead.

• A model with fully flexible elasticity specification and fully flexible internal factor loadings is not theoretically identifiable. Hence, one needs to assume certain level of homogeneity to estimate factors internally.

You can customize the PC extraction algorithm using the pca_option parameter:

# Example with custom PCA options
model = giv(
    df,
    "q + id & endog(p) ~ X + pc(3)",
    id="id", t="t", weight="S",
    pca_option={
        # Preferred: let the wrapper build the constructor for you
        'algorithm': 'DeflatedHeteroPCA',
        'algorithm_options': dict(
            t_block=20,
            condition_number_threshold=5.0,
        ),

        'impute_method': 'zero',   # auto-converted to :zero
        'demean': False,
        'maxiter': 200,
    }
)

Available algorithms:

• 'algorithm': 'DeflatedHeteroPCA', which supports additional 'algorithm_options': {'t_block': 10, 'condition_number_threshold': 4.0}: Deflated algorithm with adaptive block sizing
• 'algorithm': 'StandardHeteroPCA': Standard iterative algorithm
• 'algorithm': 'DiagonalDeletion': Single-step diagonal deletion method

When save_df=True, PC factors and loadings are added to the saved dataframe with columns like pc_factor_1, pc_factor_2, pc_loading_1, etc.

---

Working with Results

# Methods
model.summary()            # ▶ print full Julia-style summary
model.residuals()          # ▶ numpy array of the residuals for each observation
model.confint(level=0.95)  # ▶ (n×2) array of confidence intervals
model.coeftable(level=0.95)# ▶ pandas.DataFrame of estimates, SEs, t-stats, p-values

# Fields
model.endog_coef           # ▶ numpy array of ζ coefficients
model.exog_coef            # ▶ numpy array of β coefficients
model.agg_coef             # ▶ float: aggregate elasticity
model.endog_vcov           # ▶ VCOV of ζ coefficients
model.exog_vcov            # ▶ VCOV of β coefficients
model.nobs                 # ▶ int: number of observations
model.dof_residual         # ▶ int: residual degrees of freedom
model.formula              # ▶ str: Julia-style formula
model.formula_schema       # ▶ str: the internal schema of the Julia‐style formula after parsing
model.residual_variance    # ▶ numpy array of the estimated variance of the residuals for each entity (ûᵢ’s variance)
model.N                    # ▶ int: the number of cross‐section entities in the panel
model.T                    # ▶ int: the number of time periods per entity in the panel
model.dof                  # ▶ int: the total number of estimated parameters (length of ζ plus length of β)
model.responsename         # ▶ str: the name of the response variable(s)
model.converged            # ▶ bool: solver convergence status
model.endog_coefnames      # ▶ list[str]: ζ coefficient names
model.exog_coefnames       # ▶ list[str]: β coefficient names
model.idvar                # ▶ str: entity identifier column name
model.tvar                 # ▶ str: time identifier column name
model.weightvar            # ▶ str or None: weight column name
model.exclude_pairs        # ▶ dict: excluded moment-condition pairs
model.n_pcs                # ▶ int: number of principal components extracted
model.pc_factors           # ▶ numpy array (k×T) of PC time factors (if pc(k) used)
model.pc_loadings          # ▶ numpy array (N×k) of PC entity loadings (if pc(k) used)
model.pc_model             # ▶ HeteroPCAModel object with PC details (if pc(k) used)
model.coefdf               # ▶ pandas.DataFrame of entity-specific coefficients
model.fe                   # ▶ pandas.DataFrame of fixed-effects and fixed-effect interaction with exogenous controls (if saved) 
model.residual_df          # ▶ pandas.DataFrame of residuals (if saved)
model.df                   # ▶ pandas.DataFrame of full estimation output (if save_df=True)
model.coef                 # ▶ numpy array of [ζ; β]
model.vcov                 # ▶ full (ζ+β) variance–covariance matrix
model.stderror             # ▶ numpy array of standard errors
model.coefnames            # ▶ list[str]: names of all coefficients (ζ then β)

Entity-specific Coefficients DataFrame (coefdf)

The model.coefdf field provides a convenient way to access and report coefficients organized by categorical variables (e.g., by sector, entity, or other groupings). This DataFrame contains:

• All categorical variable values used in the model (e.g., entity IDs, sectors)
• Estimated coefficients for each term in the formula, stored in columns named <term>_coef
• Fixed effect estimates and fixed effect interaction with exogenous controls(if save = 'fe' or save = 'all' was specified)

Example:

# Using the estimated model above as an example
print(model.coefdf)
# id  id & p_coef     fe_id  fe_id&η1  fe_id&η2
# 1     1.007234  0.770445 -0.075198  0.905689
# 2     1.773353 -0.376699  0.452851  0.825657
# 3     1.368630 -0.827939 -1.033757 -0.512825
# 4     3.384603 -0.275443  1.348865   1.37676
# 5     0.619882 -0.419348  0.663217  1.108182

---

Simulation

The package includes utilities for Monte Carlo simulations using the simulate_data function:

from optimalgiv import simulate_data, SimParam

# Generate simulated panel datasets
simulated_dfs = simulate_data(
    params = SimParam(
        N=20,      # Number of entities
        T=50,      # Time periods
        K=3,       # Number of factors
        M=0.7,     # Aggregate elasticity
        sigma_zeta=0.5  # Elasticity dispersion
    ),
    nsims=1,      # Number of simulations
    seed=123      # Random seed
)

# Use the first dataset
df = simulated_dfs[0]

Simulation Parameters

The SimParam class accepts the following parameters:

Parameter	Description	Default
N	Number of entities	10
T	Number of time periods	100
K	Number of common factors	2
M	Aggregate price elasticity	0.5
sigma_zeta	Standard deviation of entity elasticities	1.0
sigma_p	Price volatility to target	2.0
h	Excess HHI for size distribution	0.2
ushare	Share of price variation from idiosyncratic shocks	0.2 (if K>0)
sigma_u_curv	Curvature for size-dependent volatility	0.1
nu	Degrees of freedom for t-distribution (Inf = Normal)	np.inf
missingperc	Percentage of missing values	0.0

Data Generating Process

The simulated data follows this economic model:

$$\begin{align} q_{it} &= u_{it} + \Lambda_i \cdot \eta_t - \zeta_i \cdot p_t \\\ p_t &= M \cdot \sum_i S_i \cdot (u_{it} + \Lambda_i \cdot \eta_t) \end{align}$$

Where:

• q_it: Quantity for entity i at time t
• p_t: Price (common across entities at time t)
• u_it: Idiosyncratic shocks
• η_t: Common factors
• Λ_i: Factor loadings
• ζ_i: Entity-specific elasticities
• S_i: Entity size/weights

Entity sizes follow a power law distribution calibrated to match the target excess HHI (h).

Output DataFrame

Each simulation returns a pandas DataFrame with columns:

• id: Entity identifier
• t: Time period
• q: Quantity (response variable)
• p: Price (endogenous regressor)
• S: Entity size/weight
• ζ: True entity-specific elasticity
• η1, η2, ...: Common factor realizations
• λ1, λ2, ...: Entity-specific factor loadings

---

Limitations

• PC extraction limitations: Only iv and iv_twopass algorithms support internal PC extraction. The debiased_ols and scalar_search algorithms do not support PC extraction.
• Variance-covariance matrix: When PC extraction is used (pc(k) in formula), the variance-covariance matrix calculation is automatically disabled as it is not correct. One should consider bootstrapping instead.
• Time fixed effects are not supported directly, but one can use a single factor pc(1) instead.
• Some algorithms require balanced panels.
• The debiased_ols and scalar_search algorithms require complete market coverage

---

To-do List

• Expose build_error_function interface.

---

References

Please cite:

• Gabaix, Xavier, and Ralph S.J. Koijen. Granular Instrumental Variables. Journal of Political Economy, 132(7), 2024, pp. 2274–2303.
• Chaudhary, Manav, Zhiyu Fu, and Haonan Zhou. Anatomy of the Treasury Market: Who Moves Yields? Available at SSRN: https://ssrn.com/abstract=5021055