North america build & proximal subsampling example by jameshadfield · Pull Request #124 · nextstrain/measles

jameshadfield · 2026-06-15T00:51:49Z

Tasks

Update changelog
Before merge: replace TBDs in changelog
Post-merge: run ingest-to-phylogenetic.yaml and update nextstrain.org manifest

Review threads

Original description

See commit messages

This generalises the workflow and makes it much easier to add new builds

The config changes are not backwards compatible

I won't merge this until the blog / tutorial / docs have been written, but 👀 would be great!

victorlin

Reviewed 7f46d96...95b7bf8

joverlee521

Did a first pass, will need to wrap my head around the config changes a little bit more...

joverlee521

Workflow changes look reasonable to me, left a non-blocking comment on potential direction for custom analyses set up.

jameshadfield

Thanks for taking over here @victorlin - it's looking great. I just updated the outbreak example's README. Apart from a rebase to clean up the commits I think this is good to go.

We already exported the relevant metadata to show the data use terms in Auspice

Removes the snakemake-hardcoded conditional rule selection and introduces a new wildcard 'gene_or_genome' which operates above (i.e. before) the build wildcard specific rules take over. This will allow more genome-based builds (i.e. a north-america 2025/26 build) which can use the same genome QC & alignment

to be defined in the config.yaml without needing to update a hardcoded list in config.smk Note: The intended error message (removed by this commit) was not working as expected - the helpful text wasn't being printed. I didn't track down why not.

Add some basic error / completeness checking

The parameters for the build were first implemented on branch trvrb/usa-analysis. Note that we don't run this (yet) as part of our automated builds Adding a third build prompted two config structure changes: 1. Supplying the actual command line strings to `augur refine` rather than using conditional helper functions 2. augur traits is now opt-in at the config level rather than hardcoded per-build conditionals in the snakefiles

This is to be paired with a blog-post / tutorial This was added as a new target within the canonical workflow, but subsequent commits will change this style as it needs PPX restricted data to be useful. The format of config.export has been updated to allow build-specific options

victorlin · 2026-07-20T20:38:29Z

Rebased, and in the process dropped 10315c9...bb30612 in favor of nextstrain/shared#78.

Into a separate config so we can (i) keep the canonical config cleaner and (ii) we can keep PPX restricted data -- which is important for this example -- out of the canonical config.

Shorter name for the same idea, already used prior to "changed 'gene' wildcard to 'build'" (46baf70).

It's good practice to set constraints for all wildcards used in the workflow. Move the helper function to config() so it is available early in the workflow, and update it to take just a build name instead of a wildcards variable.

Use slashes to organize builds by values in different parts. This is common practice in multi-build Nextstrain workflows. The implementation is flexible for use in other Nextstrain workflows. 'gene' is parsed out of the build name where needed, instead of maintaining a separate 'build_to_gene' mapping. A new 'build_with_underscores' wildcard and 'copy_export' rule are introduced to handle the name pattern expected for Auspice datasets. Paths for files read and written by the workflow have been updated to use slashes in build names as directories.

The example data isn't enough for the narrow filters used by 'genome/north-america'.

Follow up to <#124 (comment)> In the future, we can consider adding tests for other workflows and move these tests to a generalized reusable workflow in nextstrain/.github.

victorlin reviewed Jun 15, 2026

View reviewed changes

Comment thread phylogenetic/defaults/config.yaml Outdated

Comment thread phylogenetic/defaults/config.yaml Outdated

Comment thread phylogenetic/rules/prepare_sequences.smk

joverlee521 reviewed Jun 17, 2026

View reviewed changes

Comment thread phylogenetic/rules/prepare_sequences.smk Outdated

Comment thread phylogenetic/rules/prepare_sequences.smk

Comment thread phylogenetic/defaults/config.yaml Outdated

Comment thread phylogenetic/defaults/config.yaml Outdated

jameshadfield force-pushed the proximal-subsampling branch 2 times, most recently from b1e8944 to b9aa577 Compare June 19, 2026 00:09

victorlin reviewed Jun 24, 2026

View reviewed changes

Comment thread phylogenetic/defaults/config.yaml Outdated

victorlin reviewed Jun 29, 2026

View reviewed changes

Comment thread phylogenetic/defaults/config.yaml Outdated

victorlin reviewed Jun 29, 2026

View reviewed changes

Comment thread phylogenetic/rules/config.smk

victorlin reviewed Jun 29, 2026

View reviewed changes

Comment thread phylogenetic/defaults/config.yaml

joverlee521 linked an issue Jun 30, 2026 that may be closed by this pull request

phylo: Implement proximity subsampling #121

Closed

victorlin force-pushed the proximal-subsampling branch from 5e361c8 to c8eacb4 Compare July 6, 2026 18:31

victorlin mentioned this pull request Jul 7, 2026

Support pattern-matching syntax for subsample config #131

Closed

6 tasks

joverlee521 approved these changes Jul 10, 2026

View reviewed changes

Comment thread phylogenetic/custom-analyses/north-america-outbreak-example/README.md Outdated

joverlee521 reviewed Jul 10, 2026

View reviewed changes

Comment thread phylogenetic/custom-analyses/north-america-outbreak-example/README.md

joverlee521 reviewed Jul 10, 2026

View reviewed changes

Comment thread phylogenetic/Snakefile Outdated

joverlee521 reviewed Jul 14, 2026

View reviewed changes

Comment thread phylogenetic/defaults/config.yaml

jameshadfield commented Jul 20, 2026

View reviewed changes

victorlin self-assigned this Jul 20, 2026

victorlin force-pushed the proximal-subsampling branch from a5b0d12 to e4acc59 Compare July 20, 2026 20:24

jameshadfield added 6 commits July 20, 2026 13:24

multi-threaded align

97b6567

Update metadata fields

ccbdf4b

We already exported the relevant metadata to show the data use terms in Auspice

Allow extra builds

af1b0ef

to be defined in the config.yaml without needing to update a hardcoded list in config.smk Note: The intended error message (removed by this commit) was not working as expected - the helpful text wasn't being printed. I didn't track down why not.

Add config error checking

3571ab5

Add some basic error / completeness checking

victorlin force-pushed the proximal-subsampling branch from e4acc59 to 128edc3 Compare July 20, 2026 20:35

victorlin force-pushed the proximal-subsampling branch from 128edc3 to 95f2c63 Compare July 20, 2026 20:36

victorlin force-pushed the proximal-subsampling branch 2 times, most recently from 87a0191 to fdb8ef9 Compare July 20, 2026 23:13

victorlin reviewed Jul 21, 2026

View reviewed changes

Comment thread CHANGELOG.md Outdated

victorlin force-pushed the proximal-subsampling branch from a6b4b12 to 25d03d6 Compare July 22, 2026 00:10

jameshadfield mentioned this pull request Jul 23, 2026

[blog] Proximal subsampling nextstrain/nextstrain.org#1362

Open

2 tasks

victorlin reviewed Jul 23, 2026

View reviewed changes

Comment thread phylogenetic/custom-analyses/north-america-outbreak-example/README.md Outdated

Comment thread phylogenetic/custom-analyses/north-america-outbreak-example/README.md Outdated

jameshadfield and others added 6 commits July 23, 2026 12:40

Isolate outbreak example

a2bdbbc

Into a separate config so we can (i) keep the canonical config cleaner and (ii) we can keep PPX restricted data -- which is important for this example -- out of the canonical config.

Rename "gene_or_genome" to "gene"

cb6dd5a

Shorter name for the same idea, already used prior to "changed 'gene' wildcard to 'build'" (46baf70).

Update outputs in docstring

cb5d561

Set wildcard constraints for gene

f1fe10e

It's good practice to set constraints for all wildcards used in the workflow. Move the helper function to config() so it is available early in the workflow, and update it to take just a build name instead of a wildcards variable.

Only run global builds in CI

1b5259a

The example data isn't enough for the narrow filters used by 'genome/north-america'.

victorlin force-pushed the proximal-subsampling branch from 25d03d6 to 1ac7e75 Compare July 23, 2026 19:41

Update changelog

c6c403c

victorlin force-pushed the proximal-subsampling branch from 1ac7e75 to c6c403c Compare July 23, 2026 19:48

victorlin merged commit 7b04659 into main Jul 23, 2026
5 checks passed

victorlin deleted the proximal-subsampling branch July 23, 2026 19:50

joverlee521 mentioned this pull request Jul 24, 2026

ci: Add test for nextstrain run with outbreak example #137

Merged

2 tasks

joverlee521 mentioned this pull request Jul 27, 2026

Fix measles links nextstrain/docs.nextstrain.org#281

Merged

2 tasks

Uh oh!

Conversation

jameshadfield commented Jun 15, 2026 • edited by victorlin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Review threads

Original description

Uh oh!

victorlin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joverlee521 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joverlee521 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jameshadfield left a comment

Choose a reason for hiding this comment

Uh oh!

victorlin commented Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jameshadfield commented Jun 15, 2026 •

edited by victorlin

Loading