Add global option to skip households on simulation failure #1023

i-am-sijia · 2025-11-20T18:25:38Z

This PR addresses issue #892.

Status: Draft - core functionality implemented, remaining: logging and documentation.

Summary

Adds a new skip_failed_choices setting that allows ActivitySim to skip households that cause choice model failures instead of being masked and forced a choice by the overflow_pretection or crashing the entire simulation. This improves robustness for large-scale production runs where occasional edge cases shouldn't halt the model.

Key Changes

Core Logic (activitysim/core/)

Modified logit.py to temporarily mark failed choices with -99 and track skipped households in state
Added functionality in state.py to dynamically drop data that belong to the skipped household
Updated simulate.py and interaction_sample_simulate.py to handle failed choices
Enhanced report_bad_choices() to log skipped households

Configuration

Added skip_failed_choices: bool = True setting in top.py

Model Updates (activitysim/abm/models/)

Retained household_id in chooser columns across location choice models for tracking
Modified shadow price calculations to exclude skipped households
Updated trip matrices to adjust household expansion weights: weight * (valid_hh / total_hh)
Added null trip mode handling in trip mode choice and school escorting

Testing

Added new unit test test/skip_failed_choices to deliberately trigger failures and validates households being skipped

Usage

# settings.yaml
skip_failed_choices: true

Access skipped household info via:

state.get("num_skipped_households", 0)
state.get("skipped_household_ids", set())

Final count logged at end of simulation in the activitysim.log.

Notes

Defaults to True for production robustness
Disables overflow protection when active to surface underlying issues
Temporary code adds household_id to chooser columns (remove when PR Access to global constants and removing chooser col filtering #1017 merges)

we need it for debug tracing and skipping failed hh

…exist

i-am-sijia · 2025-12-10T05:10:07Z

For components where we intentionally allow some choices to fail without being skipped, the allow_zero_probs option is already set to True, which exempts them from this new feature. Therefore, it may not be necessary to introduce this as a component-level setting. In addition, keeping it as a global setting helps reduce manual errors, avoid confusion, and prevent inconsistencies across the implementation.

Copilot

Pull request overview

This PR adds a new skip_failed_choices feature that allows ActivitySim to skip households causing choice model failures rather than crashing the simulation or forcing a choice through overflow protection. This is designed to improve robustness in large-scale production runs where occasional edge cases shouldn't halt the entire model.

Key Changes:

Core logic modified to mark failed choices with -99, track skipped household IDs, and dynamically remove skipped household data from state tables
New skip_failed_choices setting added (defaults to True) with overflow protection disabled when active
Model updates to handle null modes, adjust trip matrices weights, exclude skipped households from shadow pricing, and retain household_id for tracking

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
activitysim/core/logit.py	Adds logic to mark failed choices with -99, track skipped households in state, and conditionally skip vs. raise errors
activitysim/core/workflow/state.py	Implements update_table() to drop rows belonging to skipped households from all state tables
activitysim/core/simulate.py	Updates eval_nl to pass skip_failed_choices flag to report_bad_choices, adds trace_choosers parameter
activitysim/core/interaction_sample_simulate.py	Clips positions to 0 when skip_failed_choices enabled to prevent index errors from -99 values
activitysim/core/util.py	Always includes household_id in columns for tracking skipped households
activitysim/core/configuration/top.py	Adds skip_failed_choices boolean setting (defaults to True)
activitysim/abm/models/location_choice.py	Filters skipped households from shadow price calculations, adds household_id retention
activitysim/abm/models/trip_matrices.py	Adjusts household expansion weights for skipped households, logs final skip count
activitysim/abm/models/trip_mode_choice.py	Handles null trip modes from failed choices, filters skipped households from merged trips
activitysim/abm/models/school_escorting.py	Adds household_id to chooser columns
activitysim/abm/models/util/tour_*.py	Adds household_id to chooser columns in tour scheduling, OD, and destination functions
activitysim/abm/models/util/school_escort_tours_trips.py	Allows null trip modes when skip_failed_choices enabled
test/skip_failed_choices/test_skip_failed_choices.py	New test that deliberately triggers failures and validates skipping behavior
.github/workflows/core_tests.yml	Adds new skip_failed_choices test to CI workflow

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

activitysim/abm/models/util/school_escort_tours_trips.py

activitysim/core/logit.py

activitysim/core/simulate.py

Copilot · 2026-01-08T17:54:34Z

test/skip_failed_choices/test_skip_failed_choices.py

+@pytest.fixture
+def state():
+    configs_dir = new_configs_dir
+    output_dir = dir_test_path("output")
+    data_dir = example_path("data")
+
+    # turn the global setting on to skip failed choices
+    update_settings(new_settings_file, "skip_failed_choices", True)
+
+    # make some choices fail by setting extreme coefficients in the uec
+    # auto ownership
+    auto_ownership_uec_file = os.path.join(new_configs_dir, "auto_ownership.csv")
+    # forcing households in home zone 8 (recoded 7) to fail auto ownership choice
+    update_uec_csv(auto_ownership_uec_file, "@df.home_zone_id==7", -999.0)
+
+    # work location choice
+    work_location_choice_uec_file = os.path.join(
+        new_configs_dir, "workplace_location.csv"
+    )
+    # forcing workers from home zone 18 to fail work location choice
+    # as if there is a network connection problem for zone 18
+    update_uec_csv(work_location_choice_uec_file, "@df.home_zone_id==18", -999.0)
+
+    # trip mode choice
+    trip_mode_choice_uec_file = os.path.join(new_configs_dir, "trip_mode_choice.csv")
+    # forcing trips on drivealone tours to fail trip mode choice
+    update_uec_csv(trip_mode_choice_uec_file, "@df.tour_mode=='DRIVEALONEFREE'", -999.0)
+
+    state = workflow.State.make_default(
+        configs_dir=configs_dir,
+        output_dir=output_dir,
+        data_dir=data_dir,
+    )
+
+    from activitysim.abm.tables.skims import network_los_preload
+
+    state.get(network_los_preload)
+
+    state.logging.config_logger()
+    return state


The test fixture modifies shared configuration files in new_configs_dir, which is reused across test runs. This can cause test pollution where one test run affects subsequent test runs. The fixture should create test-specific configuration directories or properly reset configurations after each test.

activitysim/core/workflow/state.py

activitysim/core/util.py

Copilot · 2026-01-08T17:54:36Z

test/skip_failed_choices/test_skip_failed_choices.py

+    state.run(models=state.settings.models, resume_after=None)
+
+    # check that the number of skipped households is recorded in state
+    assert state.get("num_skipped_households", 0) == 943


The hardcoded assertion value of 943 skipped households is fragile and will break if the test data changes or if the model behavior changes slightly. Consider using a range check or verifying that the number is greater than zero instead of an exact match.

Suggested change

assert state.get("num_skipped_households", 0) == 943

assert state.get("num_skipped_households", 0) > 0

Copilot · 2026-01-08T17:54:37Z

test/skip_failed_choices/test_skip_failed_choices.py

+    final_persons_df = state.get_dataframe("persons")
+    assert not any(
+        (final_persons_df["home_zone_id"] == 18)
+        & (final_persons_df["is_worker"] == True)


Use is True comparison for boolean values instead of == True. The explicit comparison with == True is redundant and not following Python best practices.

Suggested change

& (final_persons_df["is_worker"] == True)

& (final_persons_df["is_worker"])

activitysim/core/workflow/state.py

jpn--

Mostly looks fine. We discussed (possibly while @i-am-sijia was out) that the skipping should not be unlimited; the idea is we want to skip problems when problems are few and probably not seriously consequential. If the problems are many there's little benefit of allowing a simulation to run to completion.

jpn-- · 2026-01-08T17:50:20Z

activitysim/abm/models/util/school_escort_tours_trips.py


+    # trip_mode can be na if the run allows skipping failed choices and the trip mode choice has failed
+    if state.settings.skip_failed_choices:
+        return trips


Rather than just being OK here, code should:

emit a warning that N choices have failed, and

check if N exceeds configured failure thresholds (e.g. not more than 1% of all choices), in which case an error should be raised.

Added the warning for item 1. For item 2, I implemented the check in core.workflow.state.update_table(), immediately after the households are being dropped to make sure it raises an error when N (or the weighted N) exceeds the threshold. Implementing the check in the state instead of in model components makes the code base clean, i.e., less code duplication. The global parameter of the threshold is added in core.configuration.top.fraction_of_failed_choices_allowed.

jpn-- · 2026-01-08T17:55:05Z

activitysim/abm/models/location_choice.py

+            how="left",
+        )
+        if len(choices_df) > 0:
+            choices_df = choices_df[


Should check here how many choices are getting dropped, and

emit a warning that N choices have failed, and

check if N exceeds configured failure thresholds (e.g. not more than 1% of all households), in which case an error should be raised.

Both the warning and the check have been implemented in core.workflow.state.update_table(). I implemented them in the state.py instead of in model components to make the code base clean, i.e., less code duplication.

jpn-- · 2026-01-08T17:55:59Z

activitysim/abm/models/trip_matrices.py


+    # print out number of households skipped due to failed choices
+    if state.settings.skip_failed_choices:
+        logger.info(


logging level for this should be higher than "info"; at least warning if not "error" level.

I've moved the summary of number of skipped households, total and total by model component, to cli.run.run(), which is a better home than trip_matrices.py.

jpn-- · 2026-01-08T17:59:18Z

activitysim/abm/models/trip_matrices.py

+                f"Adjusting household expansion weights in {hh_weight_col} to account for {state.get('num_skipped_households', 0)} skipped households."
+            )
+            # adjust the hh expansion weights to account for skipped households
+            adjustment_factor = state.get_dataframe("households").shape[0] / (


Should the adjustment to expansion weights be based not on the number of skipped households, but instead on the original expansion weight total and the total expansion weight of the households we dropped? E.g. if we drop a household with a giant expansion weight that's a bigger thing than dropping a household with a smaller one.

I've updated the adjustment to expansion weights to be based on the weight of households instead of counts. I agree with you in theory that it should be based on weights. However, I don't think it will make a difference in application. Here's why. In simulation, 99% of the time ActivitySim has the same sample rate for all households and the same expansion weight (calculated as 1/sample rate), this is especially true when ActivitySim uses an input synthetic population with weight of 1 for each record implicitly built in. In contrast to simulation, in estimation, the input households are very likely to have different weights, however, we don't use weights when estimating a model, and trip_matrices step is usually skipped in the estimation mode.

activitysim/abm/models/trip_mode_choice.py

jpn-- · 2026-01-08T18:03:04Z

activitysim/core/configuration/top.py

+
+    .. versionadded:: 1.6
+    """
+


Need additional setting[s] to set thresholds for how many skips are OK and when it's too many and should be an error.

Added fraction_of_failed_choices_allowed as a global parameter and implemented the checking in core.workflow.state.update_table()

jpn-- · 2026-01-08T18:06:46Z

activitysim/core/logit.py

+        state.set("num_skipped_households", num_skipped_households)
+        state.set("skipped_household_ids", skipped_household_ids)
+
+        logger.debug(


logger level here should be warn not debug

Updated to warning for all loggings of skipping households.

i-am-sijia added 7 commits November 13, 2025 11:53

skip hh in simple simulate

a0ec6a8

retain household_id by default in the drop_unused_columns()

8623516

we need it for debug tracing and skipping failed hh

skip choice for interaction sample simulate

397f250

don't log hh id when bad choice df is empy

fa08f11

add hh id to user specified chooser_columns when chooser_columns …

f26cc80

…exist

blacken

76fd833

update test utils to probs raises

96e73ec

jpn-- added this to Phase 11 Dec 2, 2025

jpn-- moved this to In Progress in Phase 11 Dec 2, 2025

jpn-- assigned i-am-sijia Dec 4, 2025

i-am-sijia added 6 commits December 9, 2025 20:03

[skip ci] shadow price calc without failed choices

cf98cd2

[skip ci] special case in trip mode choice

baa47fa

[skip ci] adjust sample_rate

6cffd9b

[skip ci] add final logging of num hh skipped

0e34b7d

unit test

5da9715

update CI

50034fd

jpn-- moved this from In Progress to Under Review in Phase 11 Dec 11, 2025

jpn-- self-requested a review December 11, 2025 01:03

jpn-- mentioned this pull request Dec 11, 2025

2025-12-11 Engineering Team ActivitySim/meeting-notes#52

Closed

jpn-- requested a review from Copilot January 8, 2026 17:44

Copilot started reviewing on behalf of jpn-- January 8, 2026 17:45 View session

Copilot AI reviewed Jan 8, 2026

View reviewed changes

jpn-- requested changes Jan 8, 2026

View reviewed changes

i-am-sijia added 6 commits January 19, 2026 13:32

track and summarize failed households by component

a6ed5cb

skipped_household_ids is now a dictionary

91227b6

print more loggings about trips missing modes

c986e5b

adjust trip expansion factor using hh weight instead of count

e6a8c1b

update state tables only when new households are skipped

ee52916

put a limit on num of households allowed to skip

5316890

	assert state.get("num_skipped_households", 0) == 943
	assert state.get("num_skipped_households", 0) > 0

	& (final_persons_df["is_worker"] == True)
	& (final_persons_df["is_worker"])

Add global option to skip households on simulation failure #1023

Are you sure you want to change the base?

Add global option to skip households on simulation failure #1023

Uh oh!

Conversation

i-am-sijia commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Usage

Notes

Uh oh!

i-am-sijia commented Dec 10, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jpn-- left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

i-am-sijia Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

i-am-sijia Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

i-am-sijia commented Nov 20, 2025 •

edited

Loading

i-am-sijia Jan 20, 2026 •

edited

Loading

i-am-sijia Jan 20, 2026 •

edited

Loading