AutoAuto

AutoAuto is a collection of standalone Python scripts for preparing BNBU course, lecturer, timetable, and handbook data for CostDB. The repository is organized by data lifecycle rather than by a fixed sequence of numbered steps.

Setup

Run every command from the repository root with the locked environment:

uv sync --locked
uv run --locked python scripts/<domain>/<script>.py --help

Scripts never search for the newest file or assume that an input is beside the script. Inputs and outputs must be supplied explicitly. Existing outputs are rejected unless --force is passed.

Repository layout

scripts/
├── courses/       # Course extraction, comparison, cleanup, and import building
├── teachers/      # Staff API fetch and CostDB lecturer merge
├── offerings/     # Current, legacy, and draft timetable processing
└── handbooks/     # DeepSeek-assisted handbook extraction

data/
├── raw/           # Immutable files obtained from external systems
├── derived/       # Reproducible intermediate datasets
└── exports/       # Files prepared for review or CostDB import

tests/             # Synthetic contract and behavior tests

See data/README.md for the complete storage convention and DATA_DICTIONARY.md for file schemas.

Script index

Courses

Script	Purpose
extract_course_descriptions.py	Convert an official CDE PDF into the five-column CDE TSV.
find_new_courses.py	Compare CDE data with a CostDB export and build an 11-column import template.
enrich_course_import.py	Optionally normalize English names and add Chinese names through LM Studio.
cleanup_course_export.py	Clean course types and apply the latest department mapping.
compare_course_exports.py	Produce a field-level before/after report.
find_courses_missing_from_offerings.py	Find historical offering codes absent from CostDB.
build_course_import_from_offerings.py	Recover names and units for those missing courses.

Teachers

Script	Purpose
fetch_teachers.py	Fetch raw staff API JSON and a normalized four-column CSV.
merge_teachers.py	Merge normalized staff data with a CostDB lecturer export.

Offerings

Script	Purpose
extract_current_offerings.py	Parse the current timetable format.
extract_legacy_offerings.py	Parse the legacy timetable format.
extract_draft_offerings.py	Parse the draft timetable format.
extract_departments.py	Extract normalized department snapshots.
combine_offerings.py	Validate, exact-deduplicate, and combine offering CSVs.
merge_departments.py	Keep the latest department mapping for every course.
build_offering_import_for_missing_courses.py	Filter historical offering rows for missing courses.
resolve_offering_lecturer_conflicts.py	Apply a deterministic lecturer override before import.

Handbooks

extract_handbook_courses.py reads the second page of each PDF in an explicit directory and uses DeepSeek to create four-column course TSV files. Set DEEPSEEK_API before running it.

Workflows

Publish newly released course descriptions

uv run --locked python scripts/courses/extract_course_descriptions.py \
  --input data/raw/course-descriptions/cd-20260710.pdf \
  --output data/derived/course-descriptions/20260710.tsv \
  --raw-text-output data/derived/course-descriptions/20260710-raw-lines.txt

uv run --locked python scripts/courses/find_new_courses.py \
  --cde data/derived/course-descriptions/20260710.tsv \
  --live-courses data/raw/system-exports/courses/courses_export_2026-07-13.tsv \
  --output data/exports/courses/new-courses-20260710.tsv

uv run --locked python scripts/courses/enrich_course_import.py \
  --input data/exports/courses/new-courses-20260710.tsv \
  --output data/exports/courses/new-courses-20260710-enriched.tsv \
  --model qwen/qwen3.5-9b

The enrichment step is optional and requires LM Studio at http://localhost:1234/v1 unless --lmstudio-url is supplied.

Refresh lecturers

uv run --locked python scripts/teachers/fetch_teachers.py \
  --json-output data/raw/staff-api/teachers-2026-07-13.json \
  --csv-output data/derived/teachers/teachers-2026-07-13.csv

uv run --locked python scripts/teachers/merge_teachers.py \
  --latest data/derived/teachers/teachers-2026-07-13.csv \
  --live data/raw/system-exports/lecturers/lecturers_export_2026-02-27.csv \
  --output data/exports/lecturers/lecturers-import-2026-07-13.csv

Process a timetable

Choose the extractor matching the workbook format and provide the calendar year and normalized semester explicitly:

uv run --locked python scripts/offerings/extract_current_offerings.py \
  --input "data/raw/timetables/current/Course List and Timetable_Semester 2 of AY2025-26_20260109.xls" \
  --year 2026 --semester SPRING \
  --output data/derived/offerings/offering-2026-SPRING.csv

uv run --locked python scripts/offerings/extract_departments.py \
  --input "data/raw/timetables/current/Course List and Timetable_Semester 2 of AY2025-26_20260109.xls" \
  --year 2026 --semester SPRING \
  --output data/derived/departments/departments-2026-SPRING.tsv

uv run --locked python scripts/offerings/resolve_offering_lecturer_conflicts.py \
  --input data/derived/offerings/offering-2026-FALL.csv \
  --lecturer-conflict last \
  --output data/exports/offerings/offering-import-2026-FALL.csv \
  --qa-output data/exports/qa/offering-import-2026-FALL-lecturer-overrides.tsv

Combine files by listing the exact inputs. Shell glob expansion may be used by the operator, but scripts themselves do not select files implicitly.

uv run --locked python scripts/offerings/combine_offerings.py \
  --inputs data/derived/offerings/offering-*.csv \
  --output data/exports/offerings/all-offerings.csv

Recover courses found only in historical offerings

uv run --locked python scripts/courses/find_courses_missing_from_offerings.py \
  --live-courses data/raw/system-exports/courses/courses_export_2026-07-13.tsv \
  --offerings data/derived/offerings/offering-v2-*.csv \
  --output data/derived/offerings/missing-courses-from-offerings.tsv

uv run --locked python scripts/courses/build_course_import_from_offerings.py \
  --missing-courses data/derived/offerings/missing-courses-from-offerings.tsv \
  --timetables data/raw/timetables/current/* data/raw/timetables/legacy/* \
  --output data/exports/courses/courses-import-from-offerings.tsv \
  --qa-output data/exports/qa/courses-import-from-offerings-qa.tsv

uv run --locked python scripts/offerings/build_offering_import_for_missing_courses.py \
  --missing-courses data/derived/offerings/missing-courses-from-offerings.tsv \
  --offerings data/derived/offerings/offering-v2-*.csv \
  --output data/exports/offerings/offerings-import-for-missing-courses.csv \
  --qa-output data/exports/qa/offerings-import-for-missing-courses-qa.tsv

Clone missing 800X course records

When an offering uses a derived 800X code that is absent from CostDB, clone the complete 11-column course record from its no-suffix course:

uv run --locked python scripts/courses/clone_derived_courses.py \
  --live-courses data/raw/system-exports/courses/courses_export_2026-07-13.tsv \
  --offerings data/derived/offerings/offering-2026-FALL.csv \
  --output data/exports/courses/courses-import-derived-800x-2026-FALL.tsv

Verification

uv run --locked python -m unittest discover -s tests -v
uv run --locked python -m py_compile $(rg --files scripts -g '*.py')

The tests use synthetic workbooks and mocked AI clients; they do not require private university data, network access, LM Studio, or DeepSeek.