Skip to content

[labs] Notebooks break recurrently due to hardcoded legacy Bedrock model IDs #50

@ManojRamani

Description

@ManojRamani

Problem

Lab notebooks across all 5 modules hardcode Anthropic Claude model IDs inline. When AWS Bedrock deprecates a model (e.g. Claude Haiku 3.5 → legacy), every notebook that references it starts failing with ValidationException / access-denied errors, and every refresh becomes a chore of grepping through ~17 notebooks and bumping strings one-by-one.

Current legacy IDs still in use across labs/:

  • us.anthropic.claude-3-5-haiku-20241022-v1:0 (Haiku 3.5 — marked legacy in Bedrock)
  • us.anthropic.claude-3-5-sonnet-20240620-v1:0
  • us.anthropic.claude-3-sonnet-20240229-v1:0
  • anthropic.claude-3-sonnet-20240229-v1:0
  • anthropic.claude-3-haiku-20240307-v1:0
  • bedrock/anthropic.claude-3-haiku-20240307-v1:0 (LiteLLM-prefixed)
  • bedrock:us.anthropic.claude-3-sonnet-20240229-v1:0 (Strands-prefixed)
  • bedrock:us.anthropic.claude-3-5-haiku-20241022-v1:0 (Strands-prefixed)

~36 occurrences across 17 notebooks.

Impact

  • Students hit opaque errors at workshop start time.
  • Workshop maintainers repeat the same notebook-by-notebook bump every few months.
  • PR diffs for these bumps are noisy (.ipynb JSON).

Proposed fix

1. Single source of truth for model IDs

A labs_common package at labs/labs_common/ that exports canonical constants (US cross-region inference profile IDs):

HAIKU_MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0"
SONNET_MODEL_ID = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
# + LiteLLM ("bedrock/<id>") and Strands ("bedrock:<id>") prefixed variants

Register labs as a [tool.uv.workspace] member so uv sync (already a documented prereq) installs it editable — no new setup step for students.

Notebooks then do:

from labs_common import HAIKU_MODEL_ID
MODEL_ID = HAIKU_MODEL_ID

Bumping the whole stack = edit one file.

2. Drift-detection script

A scripts/bump_models.py that:

  • Walks labs/**/*.ipynb and greps for any hardcoded claude-* IDs in code cells and prose cells (prose won't auto-update via the import).
  • Optionally applies a {old_id: new_id} mapping in one pass.
  • Can run in CI as a pre-merge check so a deprecated ID never lands on main.

Status

  • Pilot implemented on labs/module1/notebooks/1_setup_and_basics.ipynb — imports HAIKU_MODEL_ID from labs_common, verified working via uv sync.
  • labs/labs_common/ package + workspace wiring in place.
  • TODO: roll out the same pattern to the remaining 16 notebooks and add scripts/bump_models.py.

Related: #48 (stale tool.uv.workspace paths — separate cleanup).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions