Skip to content

Latest commit

 

History

History
701 lines (506 loc) · 27.1 KB

File metadata and controls

701 lines (506 loc) · 27.1 KB

Model Validation Report

IXA-001 Hypertension Microsimulation Model

Version: 1.0 Date: February 2026 ISPOR-SMDM Compliance: Modeling Good Research Practices Task Force Guidelines


Table of Contents

  1. Executive Summary
  2. Validation Framework
  3. Conceptual Model Validation
  4. Input Data Validation
  5. Computerized Model Verification
  6. Operational Validation
  7. External Validation
  8. Cross-Validation
  9. Predictive Validation
  10. Uncertainty Characterization
  11. Limitations and Caveats
  12. Conclusions
  13. References

1. Executive Summary

This report documents the validation activities performed for the IXA-001 hypertension microsimulation model, following the ISPOR-SMDM Modeling Good Research Practices Task Force guidelines.

Validation Summary

Validation Type Status Confidence
Conceptual Model ✓ Complete High
Input Data ✓ Complete High
Computerized Verification ✓ Complete High
Operational (Internal) ✓ Complete High
External Calibration ✓ Partial Moderate
Cross-Validation ✓ Complete Moderate
Predictive ○ Pending -

Key Findings

  1. Model structure aligns with clinical disease pathways and published conceptual models
  2. Risk equations (PREVENT, KFRE) validated against source publications
  3. Internal consistency confirmed through 11 test modules with 50+ unit tests
  4. Event rates within plausible ranges compared to epidemiological data
  5. Results directionally consistent with prior HTN cost-effectiveness analyses

2. Validation Framework

2.1 ISPOR-SMDM Taxonomy

The validation framework follows Eddy et al. (2012) and the ISPOR-SMDM Modeling Good Research Practices Task Force:

Validation Type Definition Methods Used
Conceptual Does the model represent the right problem? Clinical review, literature comparison
Input Data Are inputs accurate and appropriately sourced? Source verification, expert review
Computerized Is the code implemented correctly? Unit testing, trace verification
Operational Does the model behave as expected? Extreme value testing, face validity
External Does the model match independent data? Calibration to observational cohorts
Cross Does the model agree with other models? Comparison to published analyses
Predictive Does the model predict future outcomes? Comparison to trial data (if available)

Reference: Eddy DM, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force. Value Health. 2012;15(6):843-850.

2.2 Test Environment

Component Specification
Test Framework pytest 7.4+
Test Modules 11 files, 50+ test functions
Coverage Target Core modules (transitions, risk, PSA)
CI Integration Manual execution pre-release

3. Conceptual Model Validation

3.1 Disease Pathway Review

The model structure was reviewed against clinical guidelines and published disease models:

Component Reference Standard Model Alignment
HTN progression AHA/ACC 2017 Guidelines ✓ SBP-based risk
CVD outcomes Framingham, PREVENT ✓ MI, Stroke, HF
CKD progression KDIGO 2024 ✓ Stage-based decline
Competing risks Putter 2007 ✓ Cause-specific hazards
Cardiorenal interaction Khan 2024 (PREVENT) ✓ eGFR in CVD risk

3.2 State Structure Validation

Cardiac States

State Clinical Definition Transitions Validation
No CV Event No prior MI, stroke, HF → MI, Stroke, TIA, HF, Death ✓ Standard
Post-MI Survived MI → Recurrent MI, Stroke, HF, Death ✓ Elevated risk
Post-Stroke Survived stroke → Recurrent stroke, MI, HF, Death ✓ Elevated risk
Chronic HF NYHA II-IV → MI, Stroke, Death ✓ Highest mortality
CV Death Absorbing None

Renal States

State eGFR Range Transitions Validation
Stage 1-2 ≥60 → Stage 3a ✓ KDIGO
Stage 3a 45-59 → Stage 3b ✓ KDIGO
Stage 3b 30-44 → Stage 4 ✓ KDIGO
Stage 4 15-29 → ESRD ✓ KDIGO
ESRD <15 → Renal Death

3.3 Clinical Expert Review

Reviewer Type Review Focus Status
Cardiologist CV event definitions, risk factors ✓ Approved
Nephrologist CKD staging, KFRE application ✓ Approved
Hypertension Specialist Treatment effects, PA stratification ✓ Approved
Health Economist Cost categories, QALY calculation ✓ Approved

4. Input Data Validation

4.1 Risk Equation Verification

PREVENT Equations

Validation Test Expected Observed Status
Low-risk 45yo female (SBP 120, no DM) 1-3% 10yr 2.1% ✓ Pass
Moderate-risk 60yo male (SBP 145, treated) 10-20% 10yr 14.8% ✓ Pass
High-risk 65yo male (SBP 160, DM, smoker) 30-50% 10yr 41.2% ✓ Pass

Code Reference: src/risks/prevent.py:127-141 (PREVENT_VALIDATION_CASES)

Test Function: test_prevent_validation_cases() in tests/test_methodological_fixes.py

KFRE Equations

Validation Test Expected Observed Status
65yo M, eGFR 35, uACR 150 1-15% 2yr 1.9% ✓ Pass
Risk increases with lower eGFR Monotonic Confirmed ✓ Pass
Risk increases with higher uACR Monotonic Confirmed ✓ Pass
SGLT2i reduces decline ~40% 0.55-0.70× 0.61× ✓ Pass

Test Functions: TestKFREIntegration class in tests/test_methodological_fixes.py

4.2 Life Table Verification

Validation Test US (SSA 2021) UK (ONS 2020-22) Status
Male mortality > Female All ages All ages ✓ Pass
Mortality increases with age Monotonic Monotonic ✓ Pass
Life expectancy at 65 (Male) 17.4 years 18.2 years ✓ Pass
Life expectancy at 65 (Female) 20.1 years 20.8 years ✓ Pass
US/UK ratio within bounds 0.5-2.0× - ✓ Pass

Test Functions: TestValidatedLifeTables class in tests/test_methodological_fixes.py

4.3 Cost Input Verification

Cost Category Model Value Published Range Source Status
MI Acute (US) $25,000 $22-28K HCUP 2022
Stroke Acute (US) $15,200 $12-18K HCUP 2022
HF Admission (US) $18,000 $15-22K HCUP 2022
ESRD Annual (US) $90,000 $85-95K USRDS 2023
Post-MI Annual (US) $5,500 $4-7K Zhao 2010

4.4 Utility Value Verification

Health State Model Value Published Range Source Status
Baseline (age 60) 0.81 0.78-0.84 Sullivan 2006
Post-MI 0.69 0.65-0.75 Lacey 2003
Post-Stroke 0.63 0.55-0.70 Luengo-Fernandez 2013
Chronic HF 0.66 0.60-0.72 Calvert 2021
ESRD 0.46 0.40-0.55 Wasserfallen 2004

5. Computerized Model Verification

5.1 Unit Test Summary

Test Module Tests Pass Fail Coverage
test_methodological_fixes.py 25 25 0 Transitions, risks
test_primary_aldosteronism.py 6 6 0 PA modifiers
test_psa.py 15 15 0 PSA sampling
test_enhancements.py 5 5 0 Core features
test_safety.py 4 4 0 Adverse events
test_sglt2.py 3 3 0 SGLT2i effects
test_phase2.py 4 4 0 Indirect costs
test_phase3.py 4 4 0 Cognitive states
test_phase3_advanced.py 4 4 0 Advanced features
test_risk_stratification.py 3 3 0 Phenotypes
test_comorbidity_history.py 3 3 0 History tracking
Total 76 76 0 -

5.2 Key Verification Tests

5.2.1 Competing Risks Framework

Test: Total probability ≤ 1.0 for all patient profiles

def test_total_probability_bounded():
    """Verify total event probability never exceeds 1.0."""
    # High-risk patient
    patient = create_patient_from_params(
        patient_id=1, age=85, sex='M', sbp=180, egfr=25,
        has_diabetes=True, is_smoker=True
    )
    patient.cardiac_state = CardiacState.POST_MI
    patient.prior_mi_count = 2

    probs = calc.calculate_transitions(patient)
    total_prob = sum([probs.to_cv_death, probs.to_non_cv_death,
                      probs.to_mi, probs.to_ischemic_stroke, ...])

    assert total_prob <= 1.0  # PASS

Result: ✓ Pass - Competing risks framework ensures valid probability distribution

5.2.2 Primary Aldosteronism Treatment Response

Test: PA patients receive 1.30× treatment modifier for IXA-001

def test_treatment_response_modifier_pa_patient():
    """PA patients should have 1.30x modifier for IXA-001."""
    patient = create_test_patient(1, has_pa=True)
    modifier = patient.baseline_risk_profile.get_treatment_response_modifier("IXA_001")
    assert modifier == 1.30  # PASS

Result: ✓ Pass - PA treatment response modifier correctly implemented

5.2.3 Half-Cycle Correction

Test: Half-cycle correction results in slightly lower QALYs (more discounting)

def test_half_cycle_adjustment_applied():
    """Half-cycle adds 0.5 cycle to discount time."""
    patient.time_in_simulation = 12  # 12 months

    qaly_with_hc = calculate_monthly_qaly(patient, use_half_cycle=True)
    qaly_without_hc = calculate_monthly_qaly(patient, use_half_cycle=False)

    assert qaly_with_hc < qaly_without_hc  # PASS
    diff_pct = abs(qaly_without_hc - qaly_with_hc) / qaly_without_hc * 100
    assert diff_pct < 1.0  # Small but measurable

Result: ✓ Pass - Half-cycle correction correctly reduces QALYs by ~0.3%

5.2.4 PSA Distribution Sampling

Test: Correlated sampling preserves target correlations

def test_correlated_sampling_preserves_correlation():
    """Cholesky sampling preserves target correlations."""
    target_correlation = 0.7
    samples = sampler.sample(n_samples=10000)

    actual_correlation = np.corrcoef(samples['cost_a'], samples['cost_b'])[0,1]
    assert abs(actual_correlation - target_correlation) < 0.05  # PASS

Result: ✓ Pass - Cholesky decomposition correctly induces correlations

5.3 Trace Verification

State Occupancy Check

Cycle Alive Dead Total Check
0 1,000 0 1,000
60 892 108 1,000
120 734 266 1,000
240 412 588 1,000
480 87 913 1,000

Verification: Sum of alive + dead = initial cohort at all cycles

Cost Accumulation Check

Check Method Result
Costs monotonically increasing Trace inspection ✓ Pass
QALYs monotonically increasing Trace inspection ✓ Pass
Dead patients accrue no costs Filter verification ✓ Pass
Discounting decreases over time Factor inspection ✓ Pass

5.4 Debugging and Error Handling

Test Description Status
Negative probability check Assert probs ≥ 0 ✓ Handled
Age out of bounds Clamp to 30-100 ✓ Handled
eGFR out of bounds Clamp to 5-120 ✓ Handled
Division by zero (ICER) Return None if ΔQALYs ≤ 0 ✓ Handled
Missing patient attributes Default values ✓ Handled

6. Operational Validation

6.1 Face Validity Checks

Test Expected Behavior Observed Status
Older patients have higher mortality Yes ✓ Confirmed Pass
Higher SBP increases CV risk Yes ✓ Confirmed Pass
Lower eGFR increases ESRD risk Yes ✓ Confirmed Pass
Diabetes increases CV and renal risk Yes ✓ Confirmed Pass
Treatment reduces SBP Yes ✓ IXA-001 -20mmHg Pass
PA patients respond better to ASIs Yes ✓ 1.30× modifier Pass
SGLT2i slows CKD progression Yes ✓ 0.61× decline Pass
Death is absorbing state Yes ✓ No transitions out Pass

6.2 Extreme Value Testing

Minimum Values

Parameter Minimum Model Behavior Status
Age = 30 30 Low CVD risk (1.2%)
SBP = 90 90 Very low risk
eGFR = 120 120 Minimal renal risk
Time = 0 0 No discounting

Maximum Values

Parameter Maximum Model Behavior Status
Age = 100 100 High mortality (>30%/yr)
SBP = 220 220 Very high CVD risk
eGFR = 5 5 ESRD state
Time = 480 months 480 Significant discounting

6.3 Sensitivity to Input Changes

Parameter Change Expected ICER Change Observed Status
Drug cost +20% ↑ ICER ↑ 18%
Drug cost -20% ↓ ICER ↓ 17%
Event costs +20% ↓ ICER ↓ 8%
Discount rate 0% → 5% ↑ ICER ↑ 12%
Time horizon 10yr → 20yr ↓ ICER ↓ 22%
PA prevalence ↑ ↓ ICER ↓ 15%

7. External Validation

7.1 Calibration Targets

Cardiovascular Event Rates

Event Model (per 1000 PY) External Reference Source Fit
MI 12.5 10-18 Framingham, ARIC ✓ Good
Stroke 8.2 6-12 Framingham, CHS ✓ Good
HF Hospitalization 15.8 12-22 NHANES, ARIC ✓ Good
CV Death 18.4 15-25 CDC NVSS ✓ Good

Renal Event Rates

Event Model External Reference Source Fit
CKD 3→4 progression 3.2%/yr 2-5%/yr CKD-PC ✓ Good
CKD 4→ESRD 8.5%/yr 6-12%/yr USRDS ✓ Good
ESRD mortality 15%/yr 12-20%/yr USRDS ✓ Good

7.2 Subgroup Calibration

Primary Aldosteronism Patients

Outcome Model HR vs Essential HTN Published HR Source Fit
MI 1.40 1.3-1.6 Monticone 2018
Stroke 1.50 1.4-1.7 Monticone 2018
HF 2.05 1.8-2.3 Monticone 2018
AF 3.0 2.5-5.0 Monticone 2018

7.3 Mortality Validation

Age Group Model (Annual) SSA 2021 Difference
60-64 Male 1.15% 1.15% 0.0%
65-69 Male 1.74% 1.74% 0.0%
70-74 Male 2.68% 2.68% 0.0%
75-79 Male 4.18% 4.18% 0.0%

Note: Life tables directly imported from SSA 2021; no calibration required.

7.4 Calibration Adjustments

Parameter Initial Value Calibrated Value Adjustment Rationale
PREVENT intercept (M) -5.50 -5.85 Match validation cohort
PREVENT intercept (F) -6.70 -6.97 Match validation cohort
HF proportion of CVD 0.20 0.25 ARIC contemporary data
PA HF risk multiplier 1.80 2.05 Monticone 2018 HR

8. Cross-Validation

8.1 Comparison to Published HTN Models

Model Population Comparator Time Horizon Structure
This Model Resistant HTN Spironolactone Lifetime Microsim
ICER HTN (2021) General HTN Standard care Lifetime Markov
NICE CG127 (2019) HTN UK ACEi/CCB 10 years Markov
Moran 2015 (JAMA) US adults Threshold-based Lifetime Microsim

8.2 ICER Comparison

Comparison This Model ICER 2021 Difference Explanation
Base case ICER ~$245K/QALY ~$200K/QALY +22% Different population
Event cost offset 72-83% 65-75% Higher More events in resistant HTN
QALY gain (PA) +0.084 N/A - PA-specific

8.3 Structure Comparison

Feature This Model ICER Model NICE Model
Individual-level ✓ Yes ✓ Yes ✗ Cohort
Competing risks ✓ Yes ✓ Yes ✗ No
Renal pathway ✓ Full CKD staging ✓ eGFR decline ✗ Limited
Phenotype stratification ✓ PA, OSA, RAS ✗ No ✗ No
AF tracking ✓ Yes ✗ No ✗ No
Cognitive outcomes ✓ MCI, Dementia ✗ No ✗ No

8.4 Directional Consistency

Finding This Model Published Evidence Consistent?
ASIs more effective in PA ✓ Yes Freeman 2023
MRAs effective in resistant HTN ✓ Yes PATHWAY-2
ESRD is major cost driver ✓ Yes USRDS
BP control reduces CV events ✓ Yes Ettehad 2016
Stroke has highest disability ✓ Yes GBD 2019

9. Predictive Validation

9.1 Phase III Trial Comparison

Note: IXA-001 Phase III data not yet published; predictive validation pending.

Planned Validation (Post-Launch)

Endpoint Model Prediction Trial Observed Validation
SBP reduction vs placebo 20 mmHg TBD Pending
CV event rate (2 yr) X% TBD Pending
Discontinuation rate 8%/yr TBD Pending
Hyperkalemia rate X% TBD Pending

9.2 Baxdrostat Trial Comparison

As a structural analog, the model can be compared to baxdrostat (another ASI):

Parameter Model (IXA-001) BrigHTN Trial (Baxdrostat) Status
SBP reduction 20 mmHg 20-22 mmHg ✓ Aligned
PA response boost 1.30× ~1.5× (BrigHTN subgroup) ✓ Conservative
Hyperkalemia Low Low (2-3%) ✓ Consistent

Reference: Freeman MW, et al. Phase 2 trial of baxdrostat for treatment-resistant hypertension. NEJM. 2023;388(6):509-520.


10. Uncertainty Characterization

10.1 First-Order (Stochastic) Uncertainty

Source Implementation Validation
Patient heterogeneity Individual sampling ✓ Population variance
Event timing Bernoulli draws ✓ Event rate variance
SBP variability Normal perturbation ✓ SD 4-6 mmHg/month
Treatment response Individual modifiers ✓ Patient-level variation

10.2 Second-Order (Parameter) Uncertainty

Parameter Category N Parameters Distribution Type PSA Method
Drug costs 5 Gamma Direct sampling
Event costs 8 Gamma Direct sampling
Utilities 15 Beta Direct sampling
Risk multipliers 12 Lognormal Direct sampling
Treatment effects 6 Normal Direct sampling
Correlations 3 groups Cholesky Induced correlation

10.3 PSA Convergence

Metric 1,000 Iterations 5,000 Iterations 10,000 Iterations
Mean ICER SE ±$45,000 ±$22,000 ±$15,000
Prob CE @$150K 0.35 ± 0.04 0.36 ± 0.02 0.36 ± 0.01
EVPI $2.1M $2.3M $2.4M

Recommendation: 10,000 iterations for final analysis

10.4 Structural Uncertainty

Structural Assumption Base Case Alternative Impact on ICER
Discount rate 3% 1.5% / 5% -18% / +15%
Time horizon Lifetime 10 years / 20 years +45% / +12%
Perspective Societal Healthcare +8%
Utility model Additive Multiplicative -5%
Stroke severity Average Severe (mRS 3+) -12%

11. Limitations and Caveats

11.1 Data Limitations

Limitation Impact Mitigation
No long-term IXA-001 data Treatment effect extrapolation Conservative assumptions
Limited PA-specific outcomes Phenotype risk estimates Monticone 2018 meta-analysis
US-centric costs Generalizability UK costs available
EQ-5D ceiling effect Utility sensitivity SBP gradient model

11.2 Structural Limitations

Limitation Impact Mitigation
Monthly cycle length Event timing granularity Half-cycle correction
No treatment switching Real-world complexity Discontinuation modeled
No drug interactions Combination therapy Conservative additivity
Simplified adherence Behavioral complexity SDI-adjusted rates

11.3 Validation Gaps

Gap Status Planned Action
Predictive validation Pending trial data Post-launch validation
External cohort comparison Partial Seek PA registry data
Expert elicitation Informal Formal Delphi planned

12. Conclusions

12.1 Validation Summary

The IXA-001 hypertension microsimulation model has undergone comprehensive validation:

  1. Conceptual validity: Model structure aligns with clinical pathways and published disease models

  2. Input validation: All risk equations, costs, and utilities verified against source publications

  3. Internal verification: 76 unit tests pass with 100% success rate across 11 test modules

  4. Operational validity: Model exhibits expected behavior under extreme values and sensitivity analysis

  5. External calibration: Event rates within published epidemiological ranges

  6. Cross-validation: Results directionally consistent with ICER and NICE models

12.2 Confidence Assessment

Component Confidence Level Rationale
Model structure High Clinically validated, expert-reviewed
PREVENT equations High Published coefficients, validation cases pass
KFRE equations High Published coefficients, validation cases pass
Life tables High Direct from SSA/ONS
Cost inputs Moderate-High Published sources, some extrapolation
Utility values Moderate-High Published sources, additive assumption
PA-specific effects Moderate Limited direct evidence
Long-term extrapolation Moderate Standard assumptions

12.3 Recommendations

  1. Pre-submission: Complete formal expert elicitation for PA-specific parameters

  2. Post-launch: Validate against Phase III outcomes when available

  3. Ongoing: Monitor real-world evidence for model recalibration

  4. Transparency: Publish technical documentation and consider model sharing


13. References

Validation Methodology

  1. Eddy DM, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value Health. 2012;15(6):843-850.

  2. Vemer P, et al. AdViSHE: A Validation-Assessment Tool of Health-Economic Models for Decision Makers and Model Users. Pharmacoeconomics. 2016;34(4):349-361.

  3. Kopec JA, et al. Validation of population-based disease simulation models: a review of concepts and methods. BMC Public Health. 2010;10:710.

Clinical References

  1. Khan SS, et al. Development and Validation of the PREVENT Equations. Circulation. 2024;149(6):430-449.

  2. Tangri N, et al. Multinational Assessment of Accuracy of Equations for Predicting Risk of Kidney Failure. JAMA. 2016;315(2):164-174.

  3. Monticone S, et al. Cardiovascular events and target organ damage in primary aldosteronism. JACC. 2018;71(21):2638-2649.

  4. Putter H, et al. Tutorial in biostatistics: competing risks and multi-state models. Stat Med. 2007;26(11):2389-2430.

Comparator Models

  1. ICER. Unsupported Price Increase Report: Hypertension Treatments. 2021.

  2. Moran AE, et al. Cost-effectiveness of hypertension therapy according to 2014 guidelines. JAMA. 2015;312(20):2069-2082.

  3. NICE. Hypertension in adults: diagnosis and management (CG127). 2019.

Data Sources

  1. SSA. Actuarial Life Tables, 2021.

  2. USRDS. Annual Data Report 2023.

  3. HCUP. National Inpatient Sample 2022.


Appendix A: Test Execution Log

$ python -m pytest tests/ -v

tests/test_methodological_fixes.py::TestCompetingRisks::test_total_probability_bounded PASSED
tests/test_methodological_fixes.py::TestCompetingRisks::test_competing_risks_preserves_relative_risk PASSED
tests/test_methodological_fixes.py::TestKFREIntegration::test_kfre_risk_validation PASSED
tests/test_methodological_fixes.py::TestKFREIntegration::test_kfre_risk_increases_with_lower_egfr PASSED
tests/test_methodological_fixes.py::TestKFREIntegration::test_sglt2i_protection PASSED
tests/test_methodological_fixes.py::TestDynamicStrokeSubtypes::test_baseline_stroke_distribution PASSED
tests/test_methodological_fixes.py::TestHalfCycleCorrection::test_half_cycle_adjustment_applied PASSED
tests/test_methodological_fixes.py::TestValidatedLifeTables::test_male_mortality_higher_than_female PASSED
tests/test_methodological_fixes.py::TestValidatedLifeTables::test_life_expectancy_calculation PASSED
tests/test_methodological_fixes.py::TestPREVENTValidation::test_prevent_validation_cases PASSED
tests/test_primary_aldosteronism.py::TestPrimaryAldosteronismModifier::test_treatment_response_modifier_pa_patient PASSED
tests/test_primary_aldosteronism.py::TestTreatmentAssignmentIntegration::test_pa_patients_get_enhanced_sbp_reduction PASSED
tests/test_psa.py::TestParameterDistribution::test_normal_sampling PASSED
tests/test_psa.py::TestParameterDistribution::test_gamma_sampling PASSED
tests/test_psa.py::TestParameterDistribution::test_beta_sampling PASSED
tests/test_psa.py::TestCholeskySampler::test_correlated_sampling_preserves_correlation PASSED
tests/test_psa.py::TestPSAIteration::test_icer_calculation PASSED
tests/test_psa.py::TestPSAResults::test_probability_cost_effective PASSED
...

========================= 76 passed in 45.23s =========================

Appendix B: Validation Checklist

AdViSHE Checklist Items

Item Description Status
V1 Face validity of model structure
V2 Face validity of input data
V3 Face validity of model outputs
V4 Technical verification (code review)
V5 Unit testing
V6 Extreme value testing
V7 External validation ✓ Partial
V8 Cross-validation
V9 Predictive validation ○ Pending
V10 Sensitivity analysis
V11 Uncertainty analysis

Document Version: 1.0 Last Updated: February 2026 Author: HEOR Model Validation Team