diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/README.md b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/README.md new file mode 100644 index 0000000000..9fb4625001 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/README.md @@ -0,0 +1,259 @@ +# Record: Lock-In Byte Mixer (LBM) — corrected val_bpb **1.067219** + +> **Originally posted as 0.979556. That headline was wrong** because the +> cond-PPM mixer divided total mix-NLL by SP-piece UTF-8 bytes (which +> include the 3-byte CaseOps sentinels U+E001..U+E004) instead of the +> canonical raw-text byte sidecar that every other CaseOps-lineage record +> charges BPB against (PR #1729 et seq). The corrected mean on the +> canonical denominator is **1.067219** (3-seed mean, std ≈ 7.6e-05). See +> the Errata section below for the algebra and source-line citations. + +--- + +## Errata (2026-05-02) + +Reported by @codemath3000 on PR #2138; thank you. The correction is +algebraic — same artifact, same per-token NLL, same logs — only the BPB +denominator changes. No re-eval was required. + +### The bug + +In the wrapped `train_gpt.py`'s decoded `_cond_ppm_mixture_jit` (lines +~3489 / ~3572 / ~3755 in the decoded source) and its pure-Python twin +`_cond_ppm_mixture_bpb` (lines ~3910 / ~3969), the BPB denominator +accumulates `n_token_bytes = len(piece.encode("utf-8"))` per token +(plus 1 for include_space). For the CaseOps-V1 SP8192 vocabulary that +includes the 3-byte UTF-8 encodings of the U+E001..U+E004 capitalization +sentinels, which inflate the total by 8.95 % vs the canonical raw-text +byte stream. + +The sliding-window denominator (lines ~3265–3271 of the decoded source) +is correct: it pulls per-token byte counts from `val_data.val_bytes` +(`load_validation_byte_sidecar` at line ~643), which is the canonical +raw-text sidecar `fineweb_val_bytes_*.bin`. PR #1729's authoritative +metric definition: *"BPB is still charged against the original raw UTF-8 +bytes through the exported validation byte sidecar, not against +transformed text length."* The cond-PPM mixer is the only point in the +record's code path where that convention is bypassed. + +Per-seed forensic evidence in this record's `train_seed{42,1337,314}.log`: + +``` +cond_ppm tokens=47851520 bytes=164594398 cond_mix_bpb=0.979491 … # buggy denominator (SP-piece bytes) +quantized_sliding_window val_loss:2.40982833 val_bpb:1.10119889 # correct denominator (sidecar) +``` + +The canonical sidecar count (151,074,309) is reverse-solvable from any +seed's sliding-window line: + +``` +total_NLL = val_loss × tokens = 2.40982833 × 47851520 ≈ 115,313,958 nats +sidecar = total_NLL / log(2) / 1.10119889 = 151,074,309 bytes +inflation = 164,594,398 / 151,074,309 = 1.089493 (+8.95 %) +``` + +### The corrected numbers (algebraic) + +Per-token NLL is invariant under denominator change (same artifact, same +forward pass, same PPM mixer state — none depend on the divisor). So the +corrected BPB per seed is exactly: + +``` +new_bpb[s] = old_bpb[s] × 164594398 / 151074309 + = old_bpb[s] × 1.089493 +``` + +| Seed | val_loss (nats/tok) | sliding-window BPB (canonical) | reported cond-PPM BPB (buggy denom) | **corrected cond-PPM BPB (canonical)** | +|------|---------------------:|-------------------------------:|------------------------------------:|---------------------------------------:| +| 42 | 2.33531315 | 1.10119889 | 0.97949078 | **1.067148** | +| 1337 | 2.33544783 | 1.10141497 | 0.97954725 | **1.067210** | +| 314 | 2.33564246 | 1.10142762 | 0.97962885 | **1.067299** | +| **Mean** | — | **1.101347** | 0.979556 (originally reported) | **1.067219** | +| **Std** | — | 0.000129 | 0.000069 | ≈ 0.000076 | + +### Where this leaves the submission + +- **vs current SOTA (PR #1855, 1.06108):** the corrected mean is + **+0.006 BPB worse**, so this submission is **not a SOTA result on the + canonical denominator**. The original "−0.082 vs SOTA" claim was + entirely the inflation ratio, not a real compression improvement. +- **vs sliding-window alone (1.101347):** LBM still gives a real + **−0.034 BPB** improvement on the canonical denominator. The mixer + technique itself — high-confidence sigmoid lock-in to PPM-D — is + legitimate, just much smaller in magnitude than originally reported. +- **The C2 correctness story is unchanged.** Both mix steps remain + convex combinations of two proper distributions over the same alphabet + (byte alphabet for byte_0; joint byte-sequence alphabet for the + remainder). The bug was *only* in the BPB-headline divisor, not in the + mixer math. + +### What changed in this record + +This errata patches: + +- `README.md` — adds Errata section, corrected 3-seed table, repositions + the writeup as not-SOTA. +- `submission.json` — corrects `val_bpb`, `val_bpb_per_seed`, std, + `eval_canonical_byte_count_per_seed` (164594398 → 151074309), and + `headline_metric_description`. Adds `errata` field. +- `run.sh` — header comment notes corrected headline. + +The following are **kept as-is** because they are forensic records of +what actually ran: + +- `final_model.int6.ptz` — the original quantized artifact. +- `train_gpt.py` — the original wrapped source (still contains the + buggy denominator). Decode and grep `total_canonical_bytes += n_bytes` + to find both occurrences. +- `train_seed{42,1337,314}.log` — original eval logs. The buggy + `cond_ppm bytes=164594398 / val_bpb=0.979…` lines remain visible + alongside the canonical-correct `quantized_sliding_window` lines. + +The fix lives on branch `cond-ppm-stack` of +`https://github.com/anmarhindi/parameter-golf-a` (commits to be tagged +once verified): the mixer now takes a per-token sidecar array and +accumulates that as the BPB denominator, falling back to SP-piece bytes +only when no sidecar is available (non-CaseOps callers). + +--- + +## Original technique writeup (denominator-correction applied) + +The **Lock-In Byte Mixer (LBM)** is a high-confidence-only byte-level +mixture of the model's softmax with a PPM-D byte-conditional state. The +mixer's sigmoid gate + +``` +λ = 1 - sigmoid(α · (PPM_conf - β)) with α=25, β=0.9999, order=5 +``` + +defaults to the neural model on essentially every byte position; PPM +only "locks in" — entirely substituting for the NN — at byte positions +where its context confidence exceeds 0.9999. In practice these are +positions where the language is locally near-deterministic given recent +bytes (frequent function-word continuations, common bigram/trigram +prefixes, repeated tokens), and PPM nails them at ~zero NLL while the +NN still pays a few nats. The α=25 sharp transition keeps the +mid-confidence regime out of the mix entirely. + +This is the **C2-correct member of the PPM-mix family** (PR #1835). +Where #1835's byte-level mix used `P_NN(byte) = exp(token_logp / n_bytes)` +— a per-byte value that does not sum to 1 over the byte alphabet — LBM +derives `P_NN(byte_0)` from the model's full softmax via the canonical +first-byte mask: + +``` +P_NN(byte_0 = b) = Σ_{T : first_byte(T) = b} P_NN(T) +``` + +The remainder bytes mix at the joint-byte-sequence alphabet via the NN's +chain-rule residual `P_NN_rem = P_NN(token) / P_NN(byte_0)` and the +PPM-D byte chain. **Both mix steps are between two proper distributions +over the same alphabet** — C2-defensible by construction. + +The mixer's hot loop is JIT-compiled with `numba` (added to +`requirements.txt`) so the eval pass over the canonical 50K-doc +validation set (~48M scored tokens) completes inside the 600s eval cap. + +**Data hygiene:** trained on canonical FineWeb docs **50,000+** — fully +disjoint from the canonical 50K-doc validation set. The default in +`prepare_caseops_data.py` was patched from the historic +`--val-docs=10000` to `--val-docs=50000` to prevent the train/val +overlap flagged by the CaseOps memory-leakage audit. + +Stacked on: +* PR #1493 sliding-window eval at stride=128 (each val token scored + from up to `seq_len-stride` tokens of strict-past context; + stride=128 is necessary to fit the 5×-larger canonical val under + the 600s eval cap). +* PR #1797 (@dexhunter) base + SmearGate BOS-mask fix (cocohearts review). +* `PARALLEL_START_LAYER=5`, `ENABLE_LOOPING_AT=0.85`, + `WARMDOWN_FRAC=0.85`. + +## Eval-set coverage + +The cond-PPM (LBM) runs over the **full canonical CaseOps validation +set** for every seed: + +* **47,851,520 tokens / 151,074,309 canonical raw-text bytes per seed.** +* The 164,594,398 figure that appears in each `train_seed*.log`'s + `cond_ppm tokens=… bytes=…` line is the **SP-piece** byte total + (canonical bytes + CaseOps sentinel overhead) — see Errata above. +* `cppm_chunks` are all-gathered across all 8 ranks before the + byte-level PPM-D state is advanced on rank 0 in canonical val order + — full-val PPM context, not per-rank. + +This addresses the eval-set-coverage concern raised on PR #1835. + +## Compliance — Issue #1017 + +* **C1 (causality):** sliding-window scoring strict-past only; cond-PPM + byte state advances ONLY after each byte's mix log-prob is recorded. + Marginalization at byte_0 is derived from the position's softmax, + which sees only strict past. Mix gate weights depend on PPM context + confidence ONLY (not the realized byte). +* **C2 (normalized):** byte_0 mix is a convex combination of two + byte-alphabet distributions; remainder mix is a convex combination + of two joint-byte-sequence distributions. Product is a proper + distribution over the realized token's byte stream. +* **C3 (score-first):** both NN softmax and PPM byte conditional commit + before observing the realized byte at each step. PPM state advances + post-scoring. +* **C4 (single L→R pass):** each val token contributes exactly one BPB + term. + +No SLOT, no n-gram cache outside the legal byte-level PPM-D state, no +logit bias, no ETLB, no pre-quant TTT (which is C3-violating). Standard +softmax over the SP8192 alphabet at every scored position. + +## Lineage + +PR #1394 (clarkkev) → PR #1530 (samacqua) → PR #1729 (romeerp CaseOps) +→ PR #1787 (nprime06) → PR #1797 (dexhunter, SmearGate fixed) +→ PR #1493 sliding-window +→ PR #1835 PPM byte-mix (this submission's C2-correct successor) + +## Reproduction + +```bash +git clone https://github.com/anmarhindi/parameter-golf-a +cd parameter-golf-a +git checkout cond-ppm-stack +cd obliterate_0p9 +bash run.sh # 3 seeds × (≤600s train + ≤600s eval) +bash build_submissions.sh # produces this submission folder + tarball +``` + +`run.sh` invokes `_pod_setup.sh` to install deps (brotli, sentencepiece, +flash_attn_3, **numba** for the LBM hot loop) and downloads +`docs_selected.jsonl` from `willdepueoai/parameter-golf`, tokenizes via +`prepare_caseops_data.py` with `--val-docs 50000`, then trains + evals +each seed. + +A re-run on `cond-ppm-stack` (post-fix) will log +`cond_ppm tokens=47851520 bytes=151074309 cond_mix_bpb=1.067148 … +sidecar=on` directly, matching the algebraically-corrected number above. + +To inspect the readable source of the wrapped `train_gpt.py`: + +```python +import lzma, base64, re +src = open("train_gpt.py").read() +blob = re.search(r'b85decode\("([^"]+)"\)', src).group(1) +print(lzma.decompress(base64.b85decode(blob)).decode()) +``` + +## Files + +| file | purpose | +|---|---| +| `README.md` | this record + Errata | +| `submission.json` | metadata (corrected) + `errata` field | +| `train_gpt.py` | original wrapped source (lzma+base85). Contains the buggy denominator; preserved as forensic record. | +| `final_model.int6.ptz` | original quantized + brotli artifact. Unchanged. | +| `lossless_caps.py` | CaseOps transform module (PR #1729 lineage) | +| `prepare_caseops_data.py` | parallel CaseOps tokenizer (data prep helper) | +| `tokenizers/...caseops_v1_reserved.model` | SP8192 CaseOps tokenizer | +| `train_seed{42,1337,314}.log` | original per-seed train + eval logs (each contains both the buggy `cond_ppm bytes=164594398` line and the canonical-correct `quantized_sliding_window val_bpb` line) | +| `requirements.txt` | unchanged | +| `run.sh` | reproduction launcher (header comment notes errata) | diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/final_model.int6.ptz b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/final_model.int6.ptz new file mode 100644 index 0000000000..5ecdec48cc Binary files /dev/null and b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/final_model.int6.ptz differ diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/lossless_caps.py b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/lossless_caps.py new file mode 100644 index 0000000000..98e472f824 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/lossless_caps.py @@ -0,0 +1,833 @@ +"""Lossless capitalization pre-encoding helpers. + +This module provides a narrow, reversible transform that only touches +ASCII capital letters `A-Z`. Each uppercase ASCII letter is rewritten as +``, where `sentinel` is a private-use Unicode +character that is escaped by doubling if it appears literally in the +input text. + +Example with the default sentinel `\\uE000`: + + "The NASA Launch" -> "\\uE000the \\uE000n\\uE000a\\uE000s\\uE000a \\uE000launch" + +The transform is intentionally simple for v1: + +- lowercase ASCII letters are unchanged +- uppercase ASCII letters become sentinel + lowercase letter +- non-ASCII characters are left untouched +- literal sentinel characters are escaped as sentinel + sentinel + +This makes the transform exactly invertible while allowing a downstream +tokenizer to reuse lowercase subwords across case variants. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Callable, Iterable + +LOSSLESS_CAPS_V1 = "lossless_caps_v1" +LOSSLESS_CAPS_V2 = "lossless_caps_v2" +LOSSLESS_CAPS_V3 = "lossless_caps_v3" +LOSSLESS_CAPS_V4 = "lossless_caps_v4" +LOSSLESS_CAPS_V5 = "lossless_caps_v5" +LOSSLESS_CAPS_V6 = "lossless_caps_v6" +LOSSLESS_CAPS_V7 = "lossless_caps_v7" +LOSSLESS_CAPS_CASEOPS_V1 = "lossless_caps_caseops_v1" +IDENTITY = "identity" +DEFAULT_SENTINEL = "\uE000" +DEFAULT_V2_TITLE = "\uE001" +DEFAULT_V2_ALLCAPS = "\uE002" +DEFAULT_V2_CAPNEXT = "\uE003" +DEFAULT_V2_ESC = "\uE004" +DEFAULT_V5_TITLE_MIN_LEN = 7 +DEFAULT_V6_ALLCAPS_MIN_LEN = 3 +DEFAULT_V7_ALLCAPS_MIN_LEN = 4 + + +class LosslessCapsError(ValueError): + """Raised when a transformed string is malformed.""" + + +def _is_ascii_upper(ch: str) -> bool: + return "A" <= ch <= "Z" + + +def _is_ascii_lower(ch: str) -> bool: + return "a" <= ch <= "z" + + +def _is_ascii_alpha(ch: str) -> bool: + return _is_ascii_lower(ch) or _is_ascii_upper(ch) + + +def _validate_distinct_single_chars(*chars: str) -> None: + if any(len(ch) != 1 for ch in chars): + raise ValueError("all control characters must be exactly one character") + if len(set(chars)) != len(chars): + raise ValueError("control characters must be distinct") + + +def encode_lossless_caps_v1(text: str, *, sentinel: str = DEFAULT_SENTINEL) -> str: + """Encode ASCII capitals reversibly using a one-character sentinel.""" + if len(sentinel) != 1: + raise ValueError("sentinel must be exactly one character") + out: list[str] = [] + for ch in text: + if ch == sentinel: + out.append(sentinel) + out.append(sentinel) + elif _is_ascii_upper(ch): + out.append(sentinel) + out.append(ch.lower()) + else: + out.append(ch) + return "".join(out) + + +def decode_lossless_caps_v1(text: str, *, sentinel: str = DEFAULT_SENTINEL) -> str: + """Decode the `lossless_caps_v1` transform back to the original text.""" + if len(sentinel) != 1: + raise ValueError("sentinel must be exactly one character") + out: list[str] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + if ch != sentinel: + out.append(ch) + i += 1 + continue + if i + 1 >= n: + raise LosslessCapsError("dangling capitalization sentinel at end of string") + nxt = text[i + 1] + if nxt == sentinel: + out.append(sentinel) + elif _is_ascii_lower(nxt): + out.append(nxt.upper()) + else: + raise LosslessCapsError( + f"invalid sentinel escape sequence {sentinel + nxt!r}; " + "expected doubled sentinel or sentinel + lowercase ASCII letter" + ) + i += 2 + return "".join(out) + + +def encode_lossless_caps_v2( + text: str, + *, + title: str = DEFAULT_V2_TITLE, + allcaps: str = DEFAULT_V2_ALLCAPS, + capnext: str = DEFAULT_V2_CAPNEXT, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Encode ASCII word capitalization with cheap word-level markers. + + Rules over maximal ASCII alphabetic runs: + - lowercase words stay unchanged + - TitleCase words become `title + lowercase(word)` + - ALLCAPS words become `allcaps + lowercase(word)` + - mixed-case words use: + - optional `title` when the first letter is uppercase + - `capnext + lowercase(letter)` for subsequent uppercase letters + - literal control characters are escaped as `esc + literal` + """ + _validate_distinct_single_chars(title, allcaps, capnext, esc) + controls = {title, allcaps, capnext, esc} + out: list[str] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + if ch in controls: + out.append(esc) + out.append(ch) + i += 1 + continue + if not _is_ascii_alpha(ch): + out.append(ch) + i += 1 + continue + + j = i + 1 + while j < n and _is_ascii_alpha(text[j]): + j += 1 + word = text[i:j] + lower_word = word.lower() + + if word.islower(): + out.append(word) + elif len(word) >= 2 and word.isupper(): + out.append(allcaps) + out.append(lower_word) + elif _is_ascii_upper(word[0]) and word[1:].islower(): + out.append(title) + out.append(lower_word) + else: + if _is_ascii_upper(word[0]): + out.append(title) + out.append(lower_word[0]) + for orig_ch, lower_ch in zip(word[1:], lower_word[1:], strict=True): + if _is_ascii_upper(orig_ch): + out.append(capnext) + out.append(lower_ch) + i = j + return "".join(out) + + +def decode_lossless_caps_v2( + text: str, + *, + title: str = DEFAULT_V2_TITLE, + allcaps: str = DEFAULT_V2_ALLCAPS, + capnext: str = DEFAULT_V2_CAPNEXT, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Decode the `lossless_caps_v2` transform back to the original text.""" + _validate_distinct_single_chars(title, allcaps, capnext, esc) + out: list[str] = [] + pending_escape = False + pending_word_mode: str | None = None + active_allcaps = False + pending_capnext = False + in_ascii_word = False + + for ch in text: + if pending_escape: + if pending_word_mode is not None and not _is_ascii_alpha(ch): + raise LosslessCapsError("escaped control char cannot satisfy pending word capitalization mode") + out.append(ch) + pending_escape = False + if _is_ascii_alpha(ch): + in_ascii_word = True + else: + in_ascii_word = False + active_allcaps = False + continue + + if ch == esc: + pending_escape = True + continue + if ch == title: + if pending_word_mode is not None or in_ascii_word or pending_capnext: + raise LosslessCapsError("invalid title marker placement") + pending_word_mode = "title" + continue + if ch == allcaps: + if pending_word_mode is not None or in_ascii_word or pending_capnext: + raise LosslessCapsError("invalid allcaps marker placement") + pending_word_mode = "allcaps" + continue + if ch == capnext: + if pending_capnext: + raise LosslessCapsError("duplicate capnext marker") + pending_capnext = True + continue + + if _is_ascii_alpha(ch): + at_word_start = not in_ascii_word + if at_word_start: + if pending_word_mode == "allcaps": + out.append(ch.upper()) + active_allcaps = True + elif pending_word_mode == "title": + out.append(ch.upper()) + elif pending_capnext: + out.append(ch.upper()) + else: + out.append(ch) + pending_word_mode = None + pending_capnext = False + in_ascii_word = True + continue + + if pending_word_mode is not None: + raise LosslessCapsError("word capitalization marker leaked into the middle of a word") + if active_allcaps: + out.append(ch.upper()) + elif pending_capnext: + out.append(ch.upper()) + else: + out.append(ch) + pending_capnext = False + continue + + if pending_word_mode is not None or pending_capnext: + raise LosslessCapsError("capitalization marker not followed by an ASCII letter") + out.append(ch) + in_ascii_word = False + active_allcaps = False + + if pending_escape: + raise LosslessCapsError("dangling escape marker at end of string") + if pending_word_mode is not None or pending_capnext: + raise LosslessCapsError("dangling capitalization marker at end of string") + return "".join(out) + + +def encode_lossless_caps_v3( + text: str, + *, + title: str = DEFAULT_V2_TITLE, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Encode only common word-level capitalization patterns. + + Rules over maximal ASCII alphabetic runs: + - lowercase words stay unchanged + - TitleCase words become `title + lowercase(word)` + - ALLCAPS words become `allcaps + lowercase(word)` + - all other mixed-case words are left unchanged + - literal control characters are escaped as `esc + literal` + """ + _validate_distinct_single_chars(title, allcaps, esc) + controls = {title, allcaps, esc} + out: list[str] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + if ch in controls: + out.append(esc) + out.append(ch) + i += 1 + continue + if not _is_ascii_alpha(ch): + out.append(ch) + i += 1 + continue + + j = i + 1 + while j < n and _is_ascii_alpha(text[j]): + j += 1 + word = text[i:j] + + if word.islower(): + out.append(word) + elif len(word) >= 2 and word.isupper(): + out.append(allcaps) + out.append(word.lower()) + elif _is_ascii_upper(word[0]) and word[1:].islower(): + out.append(title) + out.append(word.lower()) + else: + out.append(word) + i = j + return "".join(out) + + +def decode_lossless_caps_v3( + text: str, + *, + title: str = DEFAULT_V2_TITLE, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Decode the `lossless_caps_v3` transform back to the original text.""" + _validate_distinct_single_chars(title, allcaps, esc) + out: list[str] = [] + pending_escape = False + pending_word_mode: str | None = None + active_allcaps = False + in_ascii_word = False + + for ch in text: + if pending_escape: + if pending_word_mode is not None and not _is_ascii_alpha(ch): + raise LosslessCapsError("escaped control char cannot satisfy pending word capitalization mode") + out.append(ch) + pending_escape = False + if _is_ascii_alpha(ch): + in_ascii_word = True + else: + in_ascii_word = False + active_allcaps = False + continue + + if ch == esc: + pending_escape = True + continue + if ch == title: + if pending_word_mode is not None or in_ascii_word: + raise LosslessCapsError("invalid title marker placement") + pending_word_mode = "title" + continue + if ch == allcaps: + if pending_word_mode is not None or in_ascii_word: + raise LosslessCapsError("invalid allcaps marker placement") + pending_word_mode = "allcaps" + continue + + if _is_ascii_alpha(ch): + at_word_start = not in_ascii_word + if at_word_start: + if pending_word_mode == "allcaps": + out.append(ch.upper()) + active_allcaps = True + elif pending_word_mode == "title": + out.append(ch.upper()) + else: + out.append(ch) + pending_word_mode = None + in_ascii_word = True + continue + + if pending_word_mode is not None: + raise LosslessCapsError("word capitalization marker leaked into the middle of a word") + out.append(ch.upper() if active_allcaps else ch) + continue + + if pending_word_mode is not None: + raise LosslessCapsError("capitalization marker not followed by an ASCII letter") + out.append(ch) + in_ascii_word = False + active_allcaps = False + + if pending_escape: + raise LosslessCapsError("dangling escape marker at end of string") + if pending_word_mode is not None: + raise LosslessCapsError("dangling capitalization marker at end of string") + return "".join(out) + + +def encode_lossless_caps_v4( + text: str, + *, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Encode only ALLCAPS ASCII words, leaving all other case untouched.""" + _validate_distinct_single_chars(allcaps, esc) + controls = {allcaps, esc} + out: list[str] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + if ch in controls: + out.append(esc) + out.append(ch) + i += 1 + continue + if not _is_ascii_alpha(ch): + out.append(ch) + i += 1 + continue + j = i + 1 + while j < n and _is_ascii_alpha(text[j]): + j += 1 + word = text[i:j] + if len(word) >= 2 and word.isupper(): + out.append(allcaps) + out.append(word.lower()) + else: + out.append(word) + i = j + return "".join(out) + + +def decode_lossless_caps_v4( + text: str, + *, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Decode the `lossless_caps_v4` transform back to the original text.""" + _validate_distinct_single_chars(allcaps, esc) + out: list[str] = [] + pending_escape = False + pending_allcaps = False + in_ascii_word = False + active_allcaps = False + + for ch in text: + if pending_escape: + if pending_allcaps and not _is_ascii_alpha(ch): + raise LosslessCapsError("escaped control char cannot satisfy pending allcaps mode") + out.append(ch) + pending_escape = False + if _is_ascii_alpha(ch): + in_ascii_word = True + else: + in_ascii_word = False + active_allcaps = False + continue + + if ch == esc: + pending_escape = True + continue + if ch == allcaps: + if pending_allcaps or in_ascii_word: + raise LosslessCapsError("invalid allcaps marker placement") + pending_allcaps = True + continue + + if _is_ascii_alpha(ch): + if not in_ascii_word: + active_allcaps = pending_allcaps + pending_allcaps = False + in_ascii_word = True + out.append(ch.upper() if active_allcaps else ch) + continue + + if pending_allcaps: + raise LosslessCapsError("allcaps marker not followed by an ASCII letter") + out.append(ch) + in_ascii_word = False + active_allcaps = False + + if pending_escape: + raise LosslessCapsError("dangling escape marker at end of string") + if pending_allcaps: + raise LosslessCapsError("dangling allcaps marker at end of string") + return "".join(out) + + +def encode_lossless_caps_v5( + text: str, + *, + title: str = DEFAULT_V2_TITLE, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, + title_min_len: int = DEFAULT_V5_TITLE_MIN_LEN, +) -> str: + """Encode ALLCAPS words and only sufficiently long TitleCase words.""" + _validate_distinct_single_chars(title, allcaps, esc) + controls = {title, allcaps, esc} + out: list[str] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + if ch in controls: + out.append(esc) + out.append(ch) + i += 1 + continue + if not _is_ascii_alpha(ch): + out.append(ch) + i += 1 + continue + j = i + 1 + while j < n and _is_ascii_alpha(text[j]): + j += 1 + word = text[i:j] + if len(word) >= 2 and word.isupper(): + out.append(allcaps) + out.append(word.lower()) + elif len(word) >= title_min_len and _is_ascii_upper(word[0]) and word[1:].islower(): + out.append(title) + out.append(word.lower()) + else: + out.append(word) + i = j + return "".join(out) + + +def decode_lossless_caps_v5( + text: str, + *, + title: str = DEFAULT_V2_TITLE, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Decode the `lossless_caps_v5` transform back to the original text.""" + return decode_lossless_caps_v3(text, title=title, allcaps=allcaps, esc=esc) + + +def encode_lossless_caps_v6( + text: str, + *, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, + allcaps_min_len: int = DEFAULT_V6_ALLCAPS_MIN_LEN, +) -> str: + """Encode only ALLCAPS words with length >= allcaps_min_len.""" + _validate_distinct_single_chars(allcaps, esc) + controls = {allcaps, esc} + out: list[str] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + if ch in controls: + out.append(esc) + out.append(ch) + i += 1 + continue + if not _is_ascii_alpha(ch): + out.append(ch) + i += 1 + continue + j = i + 1 + while j < n and _is_ascii_alpha(text[j]): + j += 1 + word = text[i:j] + if len(word) >= allcaps_min_len and word.isupper(): + out.append(allcaps) + out.append(word.lower()) + else: + out.append(word) + i = j + return "".join(out) + + +def decode_lossless_caps_v6( + text: str, + *, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Decode the `lossless_caps_v6` transform back to the original text.""" + return decode_lossless_caps_v4(text, allcaps=allcaps, esc=esc) + + +def encode_lossless_caps_v7( + text: str, + *, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, + allcaps_min_len: int = DEFAULT_V7_ALLCAPS_MIN_LEN, +) -> str: + """Encode only ALLCAPS words with length >= 4.""" + return encode_lossless_caps_v6( + text, + allcaps=allcaps, + esc=esc, + allcaps_min_len=allcaps_min_len, + ) + + +def decode_lossless_caps_v7( + text: str, + *, + allcaps: str = DEFAULT_V2_ALLCAPS, + esc: str = DEFAULT_V2_ESC, +) -> str: + """Decode the `lossless_caps_v7` transform back to the original text.""" + return decode_lossless_caps_v6(text, allcaps=allcaps, esc=esc) + + +def get_text_transform(name: str | None) -> Callable[[str], str]: + """Return the forward text transform for the given config name.""" + normalized = IDENTITY if name in {None, "", IDENTITY} else str(name) + if normalized == IDENTITY: + return lambda text: text + if normalized == LOSSLESS_CAPS_V1: + return encode_lossless_caps_v1 + if normalized == LOSSLESS_CAPS_V2: + return encode_lossless_caps_v2 + if normalized == LOSSLESS_CAPS_V3: + return encode_lossless_caps_v3 + if normalized == LOSSLESS_CAPS_V4: + return encode_lossless_caps_v4 + if normalized == LOSSLESS_CAPS_V5: + return encode_lossless_caps_v5 + if normalized == LOSSLESS_CAPS_V6: + return encode_lossless_caps_v6 + if normalized == LOSSLESS_CAPS_V7: + return encode_lossless_caps_v7 + if normalized == LOSSLESS_CAPS_CASEOPS_V1: + return encode_lossless_caps_v2 + raise ValueError(f"unsupported text_transform={name!r}") + + +def get_text_inverse_transform(name: str | None) -> Callable[[str], str]: + """Return the inverse transform for the given config name.""" + normalized = IDENTITY if name in {None, "", IDENTITY} else str(name) + if normalized == IDENTITY: + return lambda text: text + if normalized == LOSSLESS_CAPS_V1: + return decode_lossless_caps_v1 + if normalized == LOSSLESS_CAPS_V2: + return decode_lossless_caps_v2 + if normalized == LOSSLESS_CAPS_V3: + return decode_lossless_caps_v3 + if normalized == LOSSLESS_CAPS_V4: + return decode_lossless_caps_v4 + if normalized == LOSSLESS_CAPS_V5: + return decode_lossless_caps_v5 + if normalized == LOSSLESS_CAPS_V6: + return decode_lossless_caps_v6 + if normalized == LOSSLESS_CAPS_V7: + return decode_lossless_caps_v7 + if normalized == LOSSLESS_CAPS_CASEOPS_V1: + return decode_lossless_caps_v2 + raise ValueError(f"unsupported text_transform={name!r}") + + +def normalize_text_transform_name(name: str | None) -> str: + """Normalize empty/None transform names to the identity transform.""" + return IDENTITY if name in {None, "", IDENTITY} else str(name) + + +def get_text_transform_control_symbols(name: str | None) -> list[str]: + """Return reserved control symbols used by a transform, if any.""" + normalized = normalize_text_transform_name(name) + if normalized == IDENTITY: + return [] + if normalized == LOSSLESS_CAPS_V1: + return [DEFAULT_SENTINEL] + if normalized == LOSSLESS_CAPS_V2: + return [DEFAULT_V2_TITLE, DEFAULT_V2_ALLCAPS, DEFAULT_V2_CAPNEXT, DEFAULT_V2_ESC] + if normalized == LOSSLESS_CAPS_CASEOPS_V1: + return [DEFAULT_V2_TITLE, DEFAULT_V2_ALLCAPS, DEFAULT_V2_CAPNEXT, DEFAULT_V2_ESC] + if normalized in {LOSSLESS_CAPS_V3, LOSSLESS_CAPS_V5}: + return [DEFAULT_V2_TITLE, DEFAULT_V2_ALLCAPS, DEFAULT_V2_ESC] + if normalized in {LOSSLESS_CAPS_V4, LOSSLESS_CAPS_V6, LOSSLESS_CAPS_V7}: + return [DEFAULT_V2_ALLCAPS, DEFAULT_V2_ESC] + raise ValueError(f"unsupported text_transform={name!r}") + + +def infer_text_transform_from_manifest(tokenizer_path: str | Path) -> str: + """Best-effort lookup of a tokenizer's text transform from a local manifest.""" + tokenizer_path = Path(tokenizer_path).expanduser().resolve() + manifest_candidates = [ + tokenizer_path.parent.parent / "manifest.json", + tokenizer_path.parent / "manifest.json", + ] + for manifest_path in manifest_candidates: + if not manifest_path.is_file(): + continue + try: + payload = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + continue + tokenizers = payload.get("tokenizers") + if not isinstance(tokenizers, list): + continue + for tokenizer_meta in tokenizers: + if not isinstance(tokenizer_meta, dict): + continue + model_path = tokenizer_meta.get("model_path") or tokenizer_meta.get("path") + if not model_path: + continue + candidate = (manifest_path.parent / str(model_path)).resolve() + if candidate == tokenizer_path: + return normalize_text_transform_name(tokenizer_meta.get("text_transform")) + return IDENTITY + + +def surface_piece_original_byte_counts( + surfaces: Iterable[str], + *, + text_transform_name: str | None = None, + sentinel: str = DEFAULT_SENTINEL, +) -> list[int]: + """Return exact original UTF-8 byte counts contributed by each surface piece. + + `surfaces` must be the exact decoded text fragments emitted by SentencePiece + in order, e.g. `piece.surface` from `encode_as_immutable_proto`. + """ + normalized = normalize_text_transform_name(text_transform_name) + if normalized == IDENTITY: + return [len(surface.encode("utf-8")) for surface in surfaces] + if normalized == LOSSLESS_CAPS_V1: + if len(sentinel) != 1: + raise ValueError("sentinel must be exactly one character") + sentinel_bytes = len(sentinel.encode("utf-8")) + pending_sentinel = False + counts: list[int] = [] + for surface in surfaces: + piece_bytes = 0 + for ch in surface: + if pending_sentinel: + if ch == sentinel: + piece_bytes += sentinel_bytes + elif _is_ascii_lower(ch): + piece_bytes += 1 + else: + raise LosslessCapsError( + f"invalid continuation {ch!r} after capitalization sentinel" + ) + pending_sentinel = False + continue + if ch == sentinel: + pending_sentinel = True + else: + piece_bytes += len(ch.encode("utf-8")) + counts.append(piece_bytes) + if pending_sentinel: + raise LosslessCapsError("dangling capitalization sentinel across piece boundary") + return counts + if normalized not in {LOSSLESS_CAPS_V2, LOSSLESS_CAPS_V3, LOSSLESS_CAPS_V4, LOSSLESS_CAPS_V5, LOSSLESS_CAPS_V6, LOSSLESS_CAPS_V7, LOSSLESS_CAPS_CASEOPS_V1}: + raise ValueError(f"unsupported text_transform={text_transform_name!r}") + + title = DEFAULT_V2_TITLE + allcaps = DEFAULT_V2_ALLCAPS + capnext = DEFAULT_V2_CAPNEXT + esc = DEFAULT_V2_ESC + if normalized in {LOSSLESS_CAPS_V2, LOSSLESS_CAPS_CASEOPS_V1}: + _validate_distinct_single_chars(title, allcaps, capnext, esc) + elif normalized in {LOSSLESS_CAPS_V4, LOSSLESS_CAPS_V6, LOSSLESS_CAPS_V7}: + _validate_distinct_single_chars(allcaps, esc) + else: + _validate_distinct_single_chars(title, allcaps, esc) + pending_escape = False + pending_word_mode: str | None = None + active_allcaps = False + pending_capnext = False + in_ascii_word = False + counts: list[int] = [] + for surface in surfaces: + piece_bytes = 0 + for ch in surface: + if pending_escape: + if pending_word_mode is not None and not _is_ascii_alpha(ch): + raise LosslessCapsError("escaped control char cannot satisfy pending word capitalization mode") + piece_bytes += len(ch.encode("utf-8")) + pending_escape = False + if _is_ascii_alpha(ch): + in_ascii_word = True + else: + in_ascii_word = False + active_allcaps = False + continue + if ch == esc: + pending_escape = True + continue + if normalized in {LOSSLESS_CAPS_V2, LOSSLESS_CAPS_V3, LOSSLESS_CAPS_V5, LOSSLESS_CAPS_CASEOPS_V1} and ch == title: + if pending_word_mode is not None or in_ascii_word or pending_capnext: + raise LosslessCapsError("invalid title marker placement") + pending_word_mode = "title" + continue + if ch == allcaps: + if pending_word_mode is not None or in_ascii_word or pending_capnext: + raise LosslessCapsError("invalid allcaps marker placement") + pending_word_mode = "allcaps" + continue + if normalized in {LOSSLESS_CAPS_V2, LOSSLESS_CAPS_CASEOPS_V1} and ch == capnext: + if pending_capnext: + raise LosslessCapsError("duplicate capnext marker") + pending_capnext = True + continue + + if _is_ascii_alpha(ch): + at_word_start = not in_ascii_word + if at_word_start: + piece_bytes += 1 + active_allcaps = pending_word_mode == "allcaps" + pending_word_mode = None + pending_capnext = False + in_ascii_word = True + continue + if pending_word_mode is not None: + raise LosslessCapsError("word capitalization marker leaked into the middle of a word") + piece_bytes += 1 + pending_capnext = False + continue + + if pending_word_mode is not None or pending_capnext: + raise LosslessCapsError("capitalization marker not followed by an ASCII letter") + piece_bytes += len(ch.encode("utf-8")) + in_ascii_word = False + active_allcaps = False + counts.append(piece_bytes) + if pending_escape: + raise LosslessCapsError("dangling escape marker across piece boundary") + if pending_word_mode is not None or pending_capnext: + raise LosslessCapsError("dangling capitalization marker across piece boundary") + return counts diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/prepare_caseops_data.py b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/prepare_caseops_data.py new file mode 100644 index 0000000000..2deac46c10 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/prepare_caseops_data.py @@ -0,0 +1,196 @@ +"""Parallel rebuild of prepare_caseops_data.py — multiprocessing tokenization. + +Bottleneck of the original is single-threaded SP encode + per-char byte +prefix-sum on val docs. With N workers via mp.Pool we get ~N× speedup. +On 28-vCPU pod, 16 workers cuts ~12h to ~45 min. + +Same CLI as prepare_caseops_data.py + extra --workers flag. +""" +import argparse +import json +import multiprocessing as mp +import os +import pathlib +import sys +from typing import Optional + +import numpy as np +import sentencepiece as spm + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from lossless_caps import ( + encode_lossless_caps_v2, + DEFAULT_V2_TITLE, + DEFAULT_V2_ALLCAPS, + DEFAULT_V2_CAPNEXT, + DEFAULT_V2_ESC, +) + +BOS_ID = 1 +SHARD_MAGIC = 20240520 +SHARD_VERSION = 1 +SHARD_TOKENS = 10_000_000 + +_LOSSLESS_V2_OPERATORS_CHARS: Optional[frozenset] = None +_worker_sp: Optional[spm.SentencePieceProcessor] = None + + +def _make_operators_set() -> frozenset: + return frozenset(( + DEFAULT_V2_TITLE, DEFAULT_V2_ALLCAPS, DEFAULT_V2_CAPNEXT, DEFAULT_V2_ESC, + )) + + +def _worker_init(sp_path: str) -> None: + global _worker_sp, _LOSSLESS_V2_OPERATORS_CHARS + _worker_sp = spm.SentencePieceProcessor(model_file=sp_path) + _LOSSLESS_V2_OPERATORS_CHARS = _make_operators_set() + + +def _byte_counts(transformed: str, piece_ids: list, pieces: list) -> np.ndarray: + n_chars = len(transformed) + prefix = np.zeros(n_chars + 1, dtype=np.int64) + running = 0 + ops = _LOSSLESS_V2_OPERATORS_CHARS + for idx, ch in enumerate(transformed): + if ch not in ops: + cp = ord(ch) + if cp < 0x80: + running += 1 + elif cp < 0x800: + running += 2 + elif cp < 0x10000: + running += 3 + else: + running += 4 + prefix[idx + 1] = running + counts = np.empty(len(piece_ids), dtype=np.uint16) + cursor_t = 0 + for i, piece in enumerate(pieces): + surface = piece.replace("▁", " ") + span_len = len(surface) + end = cursor_t + span_len + if end > n_chars: + end = n_chars + original_bytes = int(prefix[end] - prefix[cursor_t]) + cursor_t = end + counts[i] = max(0, min(65535, original_bytes)) + return counts + + +def _worker_process_doc(args: tuple) -> tuple: + """Worker: transform + tokenize one doc. Returns (doc_idx, token_ids, byte_counts_or_None).""" + doc_idx, text, is_val = args + sp = _worker_sp + transformed = encode_lossless_caps_v2(text) + piece_ids = sp.encode(transformed, out_type=int) + token_ids = [BOS_ID] + piece_ids + byte_counts = None + if is_val: + pieces = [sp.id_to_piece(int(pid)) for pid in piece_ids] + byte_counts = _byte_counts(transformed, piece_ids, pieces) + return doc_idx, token_ids, byte_counts + + +def _write_shard(path: pathlib.Path, arr: np.ndarray) -> None: + # 256 int32 header = 1024 bytes — matches kevclark/parameter-golf format + # and the load_data_shard expectations in train_gpt.py. + header = np.zeros(256, dtype=np.int32) + header[0] = SHARD_MAGIC + header[1] = SHARD_VERSION + header[2] = arr.size + with path.open("wb") as fh: + fh.write(header.tobytes()) + fh.write(arr.tobytes()) + + +def _iter_docs(docs_path: pathlib.Path): + with docs_path.open("r", encoding="utf-8") as fh: + for idx, line in enumerate(fh): + line = line.strip() + if not line: + continue + obj = json.loads(line) + yield idx, (obj["text"] if isinstance(obj, dict) else obj) + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--docs", required=True, type=pathlib.Path) + ap.add_argument("--out", required=True, type=pathlib.Path) + ap.add_argument("--sp", required=True, type=pathlib.Path) + # Default 50,000 matches the canonical romeerp/parameter-golf-caseops-v1 + # split (docs_val=50000). The original default of 10,000 left docs + # 10K-49,999 in train AND in canonical val — an 80% leak. Flagged in the + # CaseOps memory-leakage audit (caseops-memory-leakage/, 2026-05). + ap.add_argument("--val-docs", type=int, default=50_000) + ap.add_argument("--max-docs", type=int, default=0, help="Stop after N total docs (0 = process all). Limits disk usage.") + ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 8) - 4)) + ap.add_argument("--chunksize", type=int, default=64) + args = ap.parse_args() + + print(f"loading sp model: {args.sp}", flush=True) + sp_master = spm.SentencePieceProcessor(model_file=str(args.sp)) + print(f"loaded sp: vocab={sp_master.vocab_size()}", flush=True) + print(f"workers: {args.workers}", flush=True) + + train_out = args.out / "datasets" / "fineweb10B_sp8192_lossless_caps_caseops_v1_reserved" + train_out.mkdir(parents=True, exist_ok=True) + + val_buf_tokens: list = [] + val_buf_bytes: list = [] + train_buf: list = [] + val_written = 0 + train_written = 0 + n_docs = 0 + + def _build_args(): + for doc_idx, text in _iter_docs(args.docs): + if args.max_docs > 0 and doc_idx >= args.max_docs: + return + yield (doc_idx, text, doc_idx < args.val_docs) + + with mp.Pool(args.workers, initializer=_worker_init, initargs=(str(args.sp),)) as pool: + for doc_idx, token_ids, byte_counts in pool.imap( + _worker_process_doc, + _build_args(), + chunksize=args.chunksize, + ): + if doc_idx < args.val_docs: + val_buf_tokens.extend(token_ids) + val_buf_bytes.append(0) + val_buf_bytes.extend(int(b) for b in byte_counts[: len(token_ids) - 1]) + if len(val_buf_tokens) >= SHARD_TOKENS: + _write_shard(train_out / f"fineweb_val_{val_written:06d}.bin", + np.array(val_buf_tokens[:SHARD_TOKENS], dtype=np.uint16)) + _write_shard(train_out / f"fineweb_val_bytes_{val_written:06d}.bin", + np.array(val_buf_bytes[:SHARD_TOKENS], dtype=np.uint16)) + val_buf_tokens = val_buf_tokens[SHARD_TOKENS:] + val_buf_bytes = val_buf_bytes[SHARD_TOKENS:] + val_written += 1 + else: + train_buf.extend(token_ids) + if len(train_buf) >= SHARD_TOKENS: + _write_shard(train_out / f"fineweb_train_{train_written:06d}.bin", + np.array(train_buf[:SHARD_TOKENS], dtype=np.uint16)) + train_buf = train_buf[SHARD_TOKENS:] + train_written += 1 + n_docs += 1 + if n_docs % 10_000 == 0: + print(f" processed {n_docs} docs train_shards={train_written} val_shards={val_written}", flush=True) + + if val_buf_tokens: + _write_shard(train_out / f"fineweb_val_{val_written:06d}.bin", + np.array(val_buf_tokens, dtype=np.uint16)) + _write_shard(train_out / f"fineweb_val_bytes_{val_written:06d}.bin", + np.array(val_buf_bytes, dtype=np.uint16)) + if train_buf: + _write_shard(train_out / f"fineweb_train_{train_written:06d}.bin", + np.array(train_buf, dtype=np.uint16)) + + print(f"done. docs={n_docs} train_shards={train_written + (1 if train_buf else 0)} val_shards={val_written + (1 if val_buf_tokens else 0)}") + + +if __name__ == "__main__": + mp.set_start_method("spawn", force=True) + main() diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/requirements.txt b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/requirements.txt new file mode 100644 index 0000000000..c34c7ad0a7 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/requirements.txt @@ -0,0 +1,11 @@ +# Runtime dependencies for the cond-PPM submission. Most of these are already +# in OpenAI's RunPod Parameter-Golf template; we list them for completeness. +torch>=2.0 +sentencepiece>=0.1.99 +numpy>=1.20 +brotli>=1.0 +flash-attn>=2.0 # optional; FA3 wheel pre-installed on RunPod template +# numba is REQUIRED — accelerates the cond-PPM byte-mixer Python loop ~7×. +# Without it the eval falls back to pure Python and exceeds the 600s eval cap +# on the canonical 50K-doc validation set (~48M scored tokens). +numba>=0.59 diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/submission.json b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/submission.json new file mode 100644 index 0000000000..a194dac537 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/submission.json @@ -0,0 +1,46 @@ +{ + "track": "10min_16mb", + "date": "2026-05-01", + "name": "LockInByteMixer_0.979556", + "author": "Anmar Hindi", + "github_id": "anmarhindi", + "val_bpb": 1.067219, + "val_bpb_std": 7.6e-05, + "val_bpb_per_seed": { + "42": 1.067148, + "1337": 1.067210, + "314": 1.067299 + }, + "diagnostic_pre_ema_3seed_mean": 1.080004, + "diagnostic_quantized_3seed_mean": 1.092764, + "diagnostic_sliding_window_3seed_mean": 1.101347, + "headline_metric": "quantized_cond_ppm", + "headline_metric_description": "Lock-In Byte Mixer (LBM): high-confidence-only byte-level mixture of NN softmax (proper byte_0 marginal + chain-rule remainder) and PPM-D byte conditional, gated by lambda = 1 - sigmoid(alpha*(PPM_conf - beta)) with alpha=25.0 beta=0.9999 (very sharp gate; only mixes PPM when its top-symbol probability is near-certain). Run on FULL CaseOps validation set (47,851,520 scored tokens / 151,074,309 canonical raw-text bytes per seed, from val_bytes sidecar fineweb_val_bytes_*.bin per PR #1729 convention).", + "artifact_bytes_max": 15758573, + "wrapped_code_bytes": 51885, + "total_submission_bytes_max": 15810458, + "compliant_under_16mb": true, + "seeds": [ + 42, + 1337, + 314 + ], + "hardware": "8xH100 80GB SXM", + "eval_full_val_verified": true, + "eval_token_count_per_seed": 47851520, + "eval_canonical_byte_count_per_seed": 151074309, + "errata": { + "date": "2026-05-02", + "summary": "Originally reported val_bpb 0.979556 used SP-piece UTF-8 bytes (164,594,398 per seed, includes 3-byte CaseOps sentinels U+E001..U+E004) as the BPB denominator instead of the canonical raw-text sidecar (151,074,309 per seed) used by every other CaseOps-lineage record per PR #1729 convention. Corrected algebraically: new_bpb = old_bpb * 164594398 / 151074309 = old_bpb * 1.089493. Per-token NLL is invariant under denominator change, so no re-eval was required; logs and artifact are forensic records of the original run. See README.md Errata section for source-line citations and per-seed table.", + "originally_reported_val_bpb": 0.979556, + "originally_reported_val_bpb_per_seed": { + "42": 0.97949078, + "1337": 0.97954725, + "314": 0.97962885 + }, + "originally_reported_byte_count_per_seed": 164594398, + "denominator_inflation_ratio": 1.089493, + "credit": "Reported by @codemath3000 on PR #2138.", + "fix_branch": "cond-ppm-stack on https://github.com/anmarhindi/parameter-golf-a" + } +} diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model new file mode 100644 index 0000000000..fffc8bb306 Binary files /dev/null and b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model differ diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_gpt.py b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_gpt.py new file mode 100644 index 0000000000..7b31fbb553 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_gpt.py @@ -0,0 +1,2 @@ +import lzma as L,base64 as B +exec(L.decompress(B.b85decode("{Wp48S^xk9=GL@E0stWa8~^|S5YJf5<5wf0uw4K&n@VT6Qap3Y@@2YR==kt^V3b|z;4_%p4p-u~Q2v~X2+~vf(+18k!VGi-te~DEcJAJ=+DZRe<7gEe>JqSSwW>47lL0GV26$Q;+~vn5pr%|c7Ic0Tw2`TURu8+{rkHnH!c21y58{N^EAE`YOQ9AQo?JK4%uvSyOo)=!og(L&w%L%TG}FoXweI<4BQC&2(FP_`WsoG-?PK$rs2$xg_&4PdYCjst->B_sCNox)_#t;Sp}k${gD68sZ7wat#Po0h1dEAnpy2cjx5UZi)znIK7ENgpW7?~dEIU3if7s|JyNT(f!C?Dt9$NEYp4Q;6*)oC9v{^RG0zIl)y(gSOjz5!i;_hGmdU_k_&=t>%o_QY95I%C?SF?C~HQfsSm<7JE_n_6ZCsFGXACW9!GQzIJc(2I$;QJD_7mt{&86rp$yeVOpQ&T%llS!?ujlMg+F|eo_51PCT9vC9B*!am%$>YwV~gyq+etFOEJYv0{_Tz-}elED>j_Ar)@rBv1Mb334%Cg3y@hHqvMyTKG#T-rg$Muv5j-#>E{>DmFL7Nwng6TfSa07yRUc-H*n^%^}?O7yF`4Q8&5k8NIg5=sX8f96`Pf8PX{BP9s*M*&ktb`PCVJ~X#D4y}$$kr$$_3!yYzHZ}~WPoQ7`ys9`=n(6j_xT$#p7X$$DTl7E^gwyp176r4>i`BDn-!SV?q)(Lxdq8UPy}`(i9}_p*~8NIur7}B!8Ry~tFyt_nUobEfp@;;A&Cdet^;t9BnJPcrKzo)W2ADLqS$qM*_&wRO}JP00jm(@5t(t}-;)wALL`O0vO@~65HRpu(b0q!{{)H}E{iw!D%K6i_mjjD?JPt)QqrnK(uL4EtfL`rIx`of1$r!pjW_!YT8EaJz{J=PV4d)r6=MGVPgIgoPQ-P2*FFPQe0x}gU)rkamzbC_0vGmIQ|z_d-sX-mS(KpH2Z{pNvV=An@eZISg&KS+NuxFfxhkCutq+$Y({RmwIV8(&KHh~N+#_pPXQSF;H>m=oLS8V4jO&jv+)cQcsI*cok`|aC(_@czAqJ}XUBQI~n@oB245$hw%Ge4pi#Bc?l@3hYSArEZslzG_VoC7#(MO-Gs^RQxxU6NevILTWigE(FX*dTEry*Kpfz3Zs8)>c>(ZvjJ;{XPFN9Z-=0d>h7=`u5h*tHQL2W~tn1eW2ZjCVDSQniX333F|Oy;IkOj9cHb{sP4gGoxPmBO0y$c}pO2qE^~quv48O_6*j@uan?;mebH`8fpZh6akY%zJA7lquE5*LKEgcX)ag`K(XRXbYc1-fFke4mmMRMg?=Fho2GU$R(hss|h8M8AQE|=hM81w93%*MoZdnB0Ay6cB}$5nF;gH}PkFT$OlNQ*X!$dMlRh3k{3ZA|{|&9xo%6G;jlCiwVJ89D~z1QMTn(E!KEch%0e|8CXtbHPO5;%(21Od4}?9))`56kcQuv~#tVAs^{uQ0)#y7Mwbi$2CyATtzYnBc-=XG1eTHmO_fw!+i=!D$$;|okgr!+=sl$yXtI5ZHdn$jSpNHJ&w4%;$`Blk>EcX>PJ@>$HZX)&JdW5%Eh0nVvO3T(^C*FMCYo5``-{$QbkH)knCvW~Hsr?V|#yqc6Kg-#BN$3GU5Ct%!pwnq(-u-l$_3f}wYRaawk1MdJFyGiBO+1(LN?dFQ2YOo&kgBMgRqHO0KUn=KsTt85D3yZ$KY?8@5({+0Yg9M0T~PE*ao8{KtOxxr1lcm0yB1x=W`D;;l#ITWHkvT55cTfci+E5v0{)=p*Xx6(5#Hjj_f1plpyZQ<1wGpvZrVc<6o#R%mQqKlT3-(g{LpJ%#1r3GkpFq{F8l22a+1h*nJg-1XgM%R`G4Z$$XBpza*DYzg63Sv9^Vpgf}h7Hy{8@+4#9JCVj6Q>P5)HmtJ{aqMSG{>2A^WwlL=p=aiXs^RN%WCVRI+@q(sXyvEdaI>Jyf#%7>3n@stgjjy>wBs#G0d+GlHs?9}p|6KxKFCW%^annW)9_AK}p1Ui@7CmbUA<8o@8H$zi@}doE30SFwNpeLFJc5Z(g5B=(z3&mct6IUktmPHuw$_5=KX=;rZ36!oM~6<_>*J*_p`z-3ejAA)!U;=3zU8%#jpuOkIDj>{YH)Ln`%&_l8@ld}!JYr!m?c{}3Zz)@cls|Ekdj^vSJM-wW?63(c5^yu2hHV3^-FHV-SHgm>jEzgUtb}_Md$T;p7N=5z=f6siKpe^NK(dASFTB)8p^FBDYeUB*$q_8J=PgM5Mo+;0(8&9!l(fgHJoDuF&vWOqBwQ(7+y56OO<_8HM@XLB5Zpx|=qtcs@%-dH-=IC<%?~+%C@$&>qCqQFyC}*W12kvTjg*dke%rBvJ?^`HopW!;2#147a9ox6A^K&F*hZg78nR|6Ln%tG?rlsLB0%ArFL<0MJ4o(v9V5^Coll%*>d)cKu5y)wuVT`)Td5m=LoC6$ogy`hM`YQL7Bir8E$ZGS-xuMre=a3ZfOzSeaz_$ytOFs%z4iE!BZLw`gHCcFRR^(Fd!ntf`23PaEMsp{)U5}XJ26Rvu&8g0!0ve?1SCn)dQS&j!a0x5NOi!xi*_lZDWykS=!_&=x@qg+tQu}eaT81_eny;TR%VUHxOxH;$kcUIZTCP#x3C?VV=tI81RDuh-bKP50M^(1UQLXs?$%^rTyd?aQEaT#vXRbc0*u;F&YM#P=b=klD`Dx1EyFd7%$6FCr5K56b7G0PEU}rjR=$uyAzvDpFSV^8g)C4d8`^bVxAXKo-IIu&AaoE^v3bgA(_MWDhH?+lz%EYp*3MI6#9TQjj~lIPDQ4n7vf*u24$OXtY?#uqr7Gd+V6^aC^X2uyyR^JUAo`jPfBFqGFsn1pMtF+;lCz6Iz0uFW^{`<6|yU!y^FAP(Ryqpox>J@%QeaZe=GW0Bc)P|?OtW4fh{--FdpU9(S8E(Lo5Z2yT0D$VA`m0q717aWp`yUM!Sm^prReA_R0y{f4L_8(WrKtM~Z&0yhafa-hk;VbVy?#U&ujjJ?^-r}0k(&(F&-z~WTSPVdh^&LQy<9y;|UqFaxv{%83ofDQZddS%`No^kpC3Sjyx%6@30rg?)-bj!Lggm8X3gC$7mk7{CTsXYh)y?2nXY*S!U8rbU{VW6?VM9!^Yp0Uq%k}18hQ{RD>o7v+o3=ID(dcKp=(AUrp86f;I8+*uySdFeLoyVhXytaKIOeh^@baBeKo8D!yHQ9qePhWO^{lS5nd@@qX_#o(^;x$BL4A>aJyICQgPl+3u~7j6I1saeBI{tP8mAwWFSv^7hkLZQe(E0}kMygiV5Kxf#F(|2EYd|oEIC#k((c;{rKFaIpaL~;`xr)Pp%b>CK8y!<2y@9HtPE^1$NLgHpzlIm@2~0Rd|$iE!MSL)1*T`o6#ta@&%<+pUZ~H6P${fjW6df3%dpWmgSRHMge^GaP$tE|hH{DNm-QLBl+k%jTMV*}@%8kaZh?uUw|sFC70co|FEwUK{e*|KyV=diTy+sx#j`H3wg@@C6amaDq@cWQLnsV8obpQY@cw7Z3iVzPv}KIj7^04rfCvPlz&{Qmig)J!R&O^biVXYb}Jwp|G?Gqtf6hteJ)c+RvhEs%z$T-o@SVm2w^sBl6^y`rE)3^H21P^?)kP&r?#$Ca-$D2Zz(ub;n?@WC9Y54mmB>>o@0*oXUC^M_~iB${0fBZiGZ?5%S83GI&mZ!)YU#|3YsO8#0A<;MOQ9J1>T-&|;oj3oG97x_|WAGg!61l4lF4l6kQ4amD_A!kUz<Sdln;`et%x`yoNvVKRJ^u%54a-R4@V|JyvjX#YKXK2mFbp@CU1HzmP7anMZ>$`@~1$j$u?Kxs)yPztM5~(L1c^Vl*MXNA&R9UHtHIj=^dbP3Ucj#`tz1f5%xF5OaigL?RTGe6Hy2y($rSsJ#;5V!(BH9P~yEn2PHS8*X3~zI7OBoc}!Ac-$2FxFFPx=<;D>IY$CZr|)B^&p8ThxX(Aa_etLg+j3%MjhzG;IEnyW@90*`7!;(UhXpC#miK73T#}`+x->gd|@U=}zB)(9_bkxu?5zb1LS43H(D4N0p5hzGNMZ*T*#-TUGN`GwlU4NO1hRoBa@M^AK+#zu!14p`}GN@5+{P9w(!>0De^T39Ae$Ekn=*c2)MxEpsZZGMK*M!MdxT@-{>+F(0DE46bkslrY*~2b4&0cH-TE}yUSE~U-UHstzz7%buL1KqFwN;{sFFgsT6ynTU~qS`3lF1^N>s68cMUxEdpv5D2_6N-9(F9xhc$v1vVh$?t`%>8M61(Gzq(OcT5&;AeYnZSiFRJ{W^pby=IU#WgZ}$O=hHkckXMw)F}UsJgyN&X4E2IQXEa}@~4;Ybo_M^m6|}gjNIRv-Tot=)Iz%!@SvX}a6DW{E`SkanoxJ7v@#`Af*WEopFeSkQ@}u#)pFRIqZQndI$x{IS_-0dL1WX8OJ<|Zry|b7IT^+N?dv-s;d4swf++B7q})WQ`O;CLl#2l&v>ZQoLs~k#e^&k6Z#?UcA=PPXnOo8)1ZNP+@CTWL@PxA1ErNJLz7~SF9ep82YgwOikF3eXK=M&1_d9ofnSy%h?u?g(#u$F?c*XW1P|?Q5)5s08QfGTNn`dUcY$8$gCv6K2*YqQ!$nE@5Ck%DWI}fRZfhET4bw0{O+j8X}c)?pL}sYhCIrn0i+?4LO$4y4zhC|)af)Kgl>S+N?q^N*T#~PY8CJ1PwCtk-ZhU{knI-UKB$9)I=0PvZW{R_S%qO%9QAK=wu}7JyhJ{Y#9g65hIjU$%|ItjFqo%JWKxJVru#2jb4C>YCTT)p^DT7*T9O&Kg=LWZjv!Lip(`lZm8M5bc!AjW%jv1EKz%ucF6$6}%JU8f(}Ek4W5v=6HbvzSv+;B&Y;0g>Z>jrf{UeLi$_!D{4;9iOo^FhY0;?>1Of{2PYL`W@C2bES1Mp8j^@Y&E$=?6bZ%(5iwwg*;qCTyzzt6-`e-a$Jne&RT|6R#XE7fUe+jnbgB$%e;AKo+`)4etcm6m_eEE*YFzyqMxNxnmsEc3UjzqKId;uBjQ?}J}aPRyggHKA}p`bO7Q>01o3t*(VPVH$FO1Z{6f2Yh}lA1`HBbGx~LNKJ6dBXfkNOjA1T;U${ZP5mMs_@}&Zgw3NUr<+Dw%G}?rl{{2_NZ7F8*?VFI5LMa5U&R?J||CIBR1N(jy5;w@GR_pCC7?4bMsPi7clf)fF+1Sh!ZFAXawlaPVP+6Fg#GV&~8hKfJCgRN18)UJ2;L272wZ^?wKLmPBCNXWFHmkM}BcSpusAxdEs*Gw0=^I8zzD&^#1o~|*2`&9W{{OEcdAtC>EJV{;x#3(txJ?KwoGP<>}n$pTtezPE-+P0dBfoW*MrIzJSFh^s85(q%&8RJ$(m?>b0f8Bj_Q^FN4bn~Rei9+WDWz2Z5f%_nC{VX>2?g$3Rf(8BAKb&7mQ!gw**Y_tJ~0-hPFEw|4Jurm$sXir3PcR_otuKEUst79$2!-&;LEJB)<9o9X!%6{Tv`rVLFgXE(pKM$==Ke_op+caj_@Mwt05r&!^m~qoQ*{48lXnS7FfOgZ|r#!*@vlmR)2-NCaINM%1HYc2^L96`7o8)tx47JK`go32Cas!Rx?FarzXWR=(^f3L_)YsKT%uI<~@Ez5qw{=u7K>=2MCp`HR5#YhO|dlS{QqFEG%j{BAG7g3bMEVMkl7EfapkSXg>;j8x4J!jvVy<_tF-+$-*DT+Eq2xM%fZEh}?tnBwTC4%AU*Xrkhp|HS{ocTVlFJqhZKe6}2WDs^d6Wqtxun)(Dtgs&tbgJHsqURms8*7w3o;yJkL1XNu0}8z!H|y*zCM3<%=1H9~uYwiBsvD;%#IT=F-##SNMuAPv}D*6sk}pO@g-A&0;9pP0rjPG^*=d}yA|pQz6#zgs`EP4-NWotl~&v`6&sIOd&PvxznUC<&Vfb2yF`isdTNjW;O#Z60#gOJ`oYFR5O18z|E6jdFz&xw@ou+Xe@diqqy_-YuQh6j@YU;`SK;?)WsTREv^&HS>kJ{+4fQ(=8@Z%DM4N?TVC#p%xGXAR-WRCQCMf5UYsYDdz47ymXh=^rB*e2(}5Q5L?7jY>E@m_B--J5D&LOY{sVvIGm(6sS{yDHQ?u$|x>~CGZU5Zom=rx90^ZIy(E6eQ@d0?9dWcf{frJ`hmf9qVXc2N%O76l=4vNA!BUSB?6)AXN3fFuQoOHfsP?J&S^=>f^WJaj#b(RLMs)7N6pvP-}LlsY4167+i=?MbxXa_@fYB>Uzf@KdAI|PfaYDCQojeVBV^7^>wmlQ*+$-g`^W^6&+Y#ofhkTlV*e5_`&01TsxOf7VWK=^`#eiaV3H!F(UgaLoahCN*x^OtD5OKw?Y(`2piWJ)uVvZWIVwd)?Z+G=GghimIjv=^fn@FyjIPqltM}Khx)G~`f3h2VMYKz++xDYs28~lXOL#y!Wce?0CIB2|?+=R)4UCh(J6<8nVqEVn{EZWc;Dk=@bam3UzX!U|0bPY-DzM!XoDVJ$|5D}5T(tvey$#OLl8lkYj9!35ha?Bv$fW^3HfDR*|b)yiCWokOrau53l>qkcAw$NJU+6gk^R`cj^)07JRn&DDEBF)8X$Q7JA6-ruKrnon?mp4m9u3v0_iU2*vh>oS&dOb@qyPthzYxj@6%CMOn>G#Q*U}0xUsmWP~Z4efw2IEJ6i1_h=ZNM1geM$)tNZdaJ2b*9XKgGblffD)BZ>IC_$y+2#)k(S-@!-caVLeG5-dnN9bTT;w%WGm}@vDWD7ZCkeV{Q3e87y&NXy&EhVm9XF#b&&t7RdLEDw|Qu;$y{15~uWh2|0;%54d%XfrO8-*#&q;>^}o2KIGj<cDhG;D8TXs9sw+aCR2kaD?&2oH$wH%0RK#-hFH32R~iUFin$lx%2`lB*X#bm%@7J{G01}gcFo#1$fN?FqFpK&oy%-|7Oc!8p7xo(6ly86m1MfD&gg1A<{c*T%$Qi5^GywzUFe7v^C^M~FLU+VLkcN+t{-Bx>SYM-x`(SAw)J$lf4fPx?UckX6c98>#QI);A|Njvq2#i%HDcT^;5(DRbU;xrK4}49@YM>ZI0|qI-MCN%&%UvaheMJ=^19vnbdPX1wRZYC&IF5z7~ZDcZh{^~&3%h_2NJsj`OqoJ-fS2>K{x+KzsR$!6|ZwxQ4aO?ol_Eh`e?Z~fN@YZT}iK6JcbByQJ4!-5}(5UPeN6pyv8`b{|$*Ym=+Z4b*=bI6|P7}#}0A$g?cQ~z;ZZlI0{vSpP9Ldz0o9jK}ji9qo9;>LoWD6wM6y7wC=j2;!2|i7vU5)2YmF;bNEDb`-x5znKfmKj2&h`BJqjw*$Fj_g_>BD|5g>wzEZIXMZkPxC5UF(IQ`CWawvjb1oWed-2q=uKp2Is844g|twq+JCh)BU)NNb~{d|9PWqz$<%mh*n!c7XleeF(V69cfo9Cr3ftw7o5S^6#x8stHH0@*Yv*f_X5UOB^;xs!dK7ZM?E^I4Xp&9-X*MBklB0WglGkh=3Rhht4bE*Nc=1;%g@Vo!p(}LYJJgdGHdkf{U-ycnemFoqA7=$h-H47fL|C{U|%w4|c+REQEd9RLTWM(%M19QSb8-oMCr-GSi-|BT0RzpTTLesWbw>awjupG7PL+snTk5mDSCs(TQJjf~S2*JwAQuWs@PfP?vJ)^}NrV6_*X)(td=Md&8@_U{0{!JgW${vOJxRwq5X-{4|92)kHh)b-U3O-g~70Vy$sCPV2hbaan3alH>Te{*0s}DvVMVL((!E#h(UaaT6U5?6Z7!mE$R*8i0rfYo$_Ho&L8DN-miECr@Lsb6}kI7gNfVJP8;Eeo{=>Q8u!F_9ZnB-j9H>&$qamm4Xj=~bB99nr@JQI?#qDfNlUA*0|Yy)M=k!q_6bqz@~asy3f^W4={rr((iRlh(Du^pP~Jl(BJk;e-ZZE=v)?d=@-@p9}a`oSefuUVpnxMM91nEMHd22$`t+o>v6P8fH7s+OhPO!9>eLe&xI%hMvyiWAymN^0QR`5!^>e>fn6E4PXs5`cneNfN!K2=1nl*Ldahq2TcHL!NcH$ILef~y&EuW2*gZ!p(lMCVC4r;LX)Z>cQMHA%AFQtpBUDhzMDv+trv@vnudOlw~gDjopyoy^zj*2<3v%5O?k%}I0=!%8c3!6R!B&(zj=zHZTMO**<7Vt9r)-em$za^xO#S|Eumjo#EhWF64`PyShCHPL9M{3&vKrqc+L00XH%PbE#EF=(EiJ(C>QSW^s^D>Vr&oN&N`#t{gtT(U9$s+WU2s!JyLnWjOhho3K3IDVE(7-Ni&<{D4Fj-qFNNC`4`!f*5^_Q1wOmzZJl-J+iB<=(`0k?eR(59=(-9LO4sqf`1`B=W~r!vn#yI$OH&>hDv>Ioyb3DTSz0npK`!1AD8T9N%7Bp=SCRJSpzW%>Tu15!5GF{U~FBEtl&+wIXbWpQ(D^2ShvuBSJ=x@UcY~#049O-v-P8Mszcw&jatJ3wCHg#d$dic3v!c|iMIqb4P_Iy!>x^Vp6>%Hs!+grL5l~jutPd#E*`TXVgw>DCf$yh>N@{FMdn)hrDsZ$Td4ApqFW)eH}<9&D@POTD84*tGh3Df+yb5raIq$o;4XI=Anr3cmcc`pdwW|RP@Fzlf4n+)L(K{fMJbE)wz(M2)DqH`oL7#BI=4?8^8s;N~K44{c-7OxzR@H6NfMDtwk1SW1C@9kv4erFpnCQrGHKgt<9=PcqaG>yFb|ermGu9$nI^BZ}|kT)Vo`*ez+EnN8D8nP#YXQqFagmn#!z!b3TDy+X|_zuK`E>b>nT-ES|h>%&agN++V5gaUzu5ElzjIcSCu#JcO~;RZ=?&bevc7$ue6tqnDp@yYEUVVlv?7&~TbWKR9hlbg$7?p>%kFO>(>^l9R~f(czf=*M<$z740+YR%cvA?55#kr>DyG4T^tTJX`)-XRKT;vacPI4(SmgqRl>rb$lP;K8aiaL?O7s%NnCEgJUJM7448d_NfoKwONz`}YSp{tXg%nq*Th21h^d`||b1yvk%7+Te#PNr%zbkd;Kz)}2+&sUSOS=!(erl&5zNrOHajz*%0l3%yBKh_xp8mpL)Aiaw)90BsSAO`r3K#}&aP3Yc>pwz)!hn;XXqX`?I(ov8O>)sXTSA}bGsoJ+1~X$qh?9a?^kRId{{MzVhbkf^S{9F~5`g~>a4XLNK+d(a{_4eLybG9|DJ3Zg)tKhNH+*$#z8MFJO!mU1TA-vGOfq1`%es21UuAt`et;#ec`JW1=Q<6PZVk`C_QI~k$W>DiOFGzbO0;F`U`>}_u|VoR(8ZH^5ynX1f_swZh6i6E??b%uqQLwsNn&JSN+TS5$`{D*5Bf@tSh~&0#tBx!XDuJ`E{j)aPpo=&q@-wl+`Fet)k5qO1jZYb@{T(+F6R87t0@@;>;78=zx2fr7Tajc%s;50_r$&O5pnqAkWwHEpwWR2K)y3weyQ`d@^`8QKulns0=tK^LNZK9oT*BEbRk%bSl9P{c`jrF3LmALbHzK1oj3(G2ntXcV?TH@A165JaZC0=*;(s@fRGHgd&^ds}@Q$b)PYusd0{vh^6J5O$9Sl%Dpn(;bQG2^hxt`qrSy*~M?615erfPD-esRldnMFa3Z_)8{zA;?-P+r!T;Uy?kpB@uud&vb^ooBEko<+HVB7a1@n;(qg#rj+Rk*r>hXC&dpzo2W7Io`roCFAQdu-(qSQtkbYlPqVgDoUx6{MD}b7<*Z)vXRzr5zGB`7lUVC2xXbuxj|pme8w~D^Fep#5Y9(Dq_RL_>dFnH07~KUg{~rKlW0Low&rz3*i&?ik#iwSB#&ycYC$UXc{S2P)@Jd4!Ng~0~evMMq0^+I?C!$Vu9m$kPy{<1kohx;IB>=!+QgSk%VJ2UZn197gM&fXBq1iegk%a%RC?^0{6UH5O{2UPNv+?0O=&#uF+!ZBUmxSGX#S0skv;v6{6N`fzN-Z(DAjIJmUTR(_8*zpLk;ZYj*jiI$--BH-ffg3V+FOMwTBcQ8mV8P-0%A&`2P0LsA}$sQdkTnAl4UCCLjL*DA{SPE#REZ+c%6i5#;ufQkh~iH{y9S1!Nq9hAXClXjkjSrlR?Ca1^+H+!MO0Qq$XE%}7toQu6__TXg-N5V~?z`cP{N|H=X)sS}OATqiKvb5(zj_FPXD-Y`|#*HGMO~wU-hB_ysvJ7t~C3T5qVz{w{Bt7POX|H+z7KKazRjPbrfCW!WNqgGt_}Oj+DR154yc#j3E1#{*N>cgfrVDFPlu~9J1|xxh5>75Pz>ioTZc^mhznWC)DfkK0q_93o*|gvke2lcSQox!{oUMt8W?N{;rmnQG5+24@RO6nYFdB4?4~tKccIubS<%-wyd}>!UM$;3emO%W9;qfw#FpS93{vj~P$EfBUtguZMTX@QjsL)*2HV+3A8PAemQiHf6$Qrm%&5UMJ5Z?J4f;~(zI;58Q#BMYYC8FMO-8^Gp6v@!CYlkjy(Q`Qc3%)E}#L39%3EZP@cjh-Mp(Rma@W8_;N68mUSNvN%tV-hC8R%D?E!5f-aS^^UKGHbGEDgB{=^8268S4Ps=M%~;L`h5U>2P%Esxnb{&W@mJ=p_?3*c5Ku^8F_V*&I#iW%UAvKjNAJq8Xqn&;;6(@DBBgdV-n#6+(zP)eULjrgPo?O~2!32AgG;Kvm^Z_x4W?u)3kjYTdK819*rvSu&t|_KFda>)1=>N}?>&Ug^qM44MbhuM)FH?gbn*7(_vn(@efq7HIAi3oiB05yOaV4@iN-tLGjo$=BYu%aTg|;~y0MD&7O@Ts)wRA;ci&;&4{fV>#Rmm4WjGBH=l?VBfniH7&#&2{alT=RtP45_DAD(>4Xz%ETP(f-+cp(I%gDH&FH`>#2_5p+n>cg6$cMy$_Uhx=!SQu*+xcp(i%dk!)ms>`jMfgr6Vk#>tUvkcEQGf$^T>JnoI-EC!|UjMfviT!>=wvI!w9vCUGI_Su|@xS)ff>JlV)QLic`yeG4GYtq7a-jx=J;0G?kT$C)LG2v528wmG1W$>Yd;7(Y2EU7bSJnK6h_bjapkqN!va0CVEZ*NkCg{~RQ`xL)rTg0}R$2dADjBNk)RNrSEZ~wBPn)No!;ST0=Nzjl&WQ|=Jw8M#`-87i-)7V}m{}{W32ge?zg^^}isriVGc&G4o3}Bx-8RjVrIoPr|!zwDPGtkrHxPVd8ddrs-kEnU&N5tw|poD0Td|1Z`AICgJTU81Uz}Afu&2%y_sIsD7IP8H+cO2FO45N{}O)7wJ%kSD}Z{ok|{_E8$H-C13VW*zm9Dx=DaWGZZMz5(NSuasBM2ARW^Jwi!^dB_k?#C4OGt@qb6&PW{&f$w%nS#ULRn4uHZGj|kEPQ{EugZNL@PQd3{_7V{Tm#UwEwPaU;sO1`zK-r$!98O72eXE=K_r-dy$0<$X0*1FI3f~jB0z$|fx+#T0Cgkr`=}8a;2WxE-2##F{-8+-#H=7xYP|haYIVq2pd`N~3r(Cs8DM&4(eqP%3`3`nz2R6?E7+_vqn)cr~oD-#|B#Lm+-Bzr?nQo*WR|66;3x88-Mw{qb>JX3ZSAX^Q=tj0IXN!)8*9xvEs-vHcP5+Z^N!#i52L@oW-{t&4lYyx9e|;Wr|04Z?{z$|P`N2bn_f(`1f{rlDBcbdHDkcd$W>!LNP}adL69jo~*f)w9Y|Sq1NFw(~aic%8tL!w*P_0V~J>&Wsg3&$K2Q#={@azr-@(ke3`nCTF^3Fs%YG2qgvnxT+3c@>Q_RHEbLW2JHbq$HwLe4XhZl`X#*vhTL?2X6W1y|F##%tTYN*l^jGG``(2{dx-Uu%*AEV3uhAmX&o@MWxHSe!^5-oE>3SK`6vOHhd@jnIey!k4Mk?@Ox{pRcXNmN(%(m*^6Qno|DTNnWIqIhIleYgIeegDUs#Zt9>yp{Qpv=KI|Dp=tqu4diU8QZpDnankg$IVXil}0^aOkcX{K~YZBeF#h#YNQ~XYnH1K(7bjq)yoTd>5rRN`Dch(a9!E2j>#{NA~dtx<`$7g~=q(jM1i)8hcw1MDa8n@12a$AD=ND3tB1kT4~*0hA|nb*Gkc8q_bM{CW|3JJ~nZqy*ar0t7{SyE{9K+Da)p;<`-sDSoGn(81KD;2}HgICwKIMkCFy^M=X$dW+2deeBxH@i_=J4i!w;^s;}uZ;rh6izORGEUdg5zFAa;>E{&E*{Fu7Z>)qq;x=dV%ea+!s@naCPpT_yU-)hmp!Lo&8w}<@d}AXyB$dg;p^Lk_>itI>8^64NBFHIC!B^Vt%!t$ch=AW8^7S8L+U>Jx^QGC`lkk^)M5s4xIX$CG3@_qCBBG#eV|7Xo#b^)#gyYfxFYSLZ7t878a^`<{qV2tlFWIL?G>l`)2BxK49@1wGDXHn!?(CIc3kF=cS`{NTyn5?0@Lq$CzNG$H}Ny`w%Tp|(($yCLy_@TuM{3JZfGWfOw52BA)uD!-0;H~_&Qq}z4fz9)}ypV73G(1W}%6AD$Gv58i@dq)O23~4(=m&Lc{KUGupu65czQFtsFxD*=5b(1*68W87#2w7QbMmFKATF)X^^QyVcJYQl{(hX~1R%yNN&!Zdv>Gt@APtP{-w<^TTl^a0wVbAyRIYSvYO>a~GkY5+Ny5=a-M)3u54VfxG@R`vCZR@`T_T^ns#xc;d;_?z?yVy(mw3eQ8IolE^i!vr8_bZ{GY1o(!*opc$`@uyC%76NI&nC<@4XuO@10J)$m=x#LM4k-r&lyMqg_gR6g-9$sQ`PgJ^*wy>?lI)x?7Pa~X}s5d~{PLugV8k}LlkM1FksdL;zPI=ZMkF^#JLKSi;Xz^tNf*R^m8!ly!!gM})Q>j(7$mSm7J1l6%Mr14lJl^jwttGV|bsgsxPWotTwm4L>?AO;X87q+{7-@18~5$8H|-*VSfCjpyvA0+bF(E{#@>mIErg@3g)AGmx;F;zP+aF&Mqr$hm+w&F4+eEwd7la=aG#>5+q^*m87d`lwdF!P4&%F-{&dQ3MGxJv@C+xQLVxO>U73ssUl%{g|4?KT=y4NUb*1dudz+1QMC&IN^8mPiGNwjzR^w(wEKqR8o)j_B=G8T4`)F#j}$i`G;1iKVPovGn^6D#n>XBsjo_Mv>N-116*YHl5GE_&AM3K|aA3*-1}=NUUxgcb-7Ci*fQqjgijA5~306NI8d#hk>OzUuHk^uB9C?n5E|$Zm^3L}Y{n21P1*%A1OZAR$8dr6Q0^)`+~^lTq2&T2ul$=|*{nt#E-H;Pa3-O5tOx!~m*1f+V5XHiwiC($B+!C4@_3F;mAhX=a{ME`EO3l~&+&O3t*Q5x^FFL98WEGFX#Q_=6sOnO+RNi6(+xJu8Lv4ZkwX9UQvQ)jsJ{$Pr)nZK#jX6-W~xnYk6R;Bo4txXviP`i@ofY((6Ew#t58Eqn0)&4ZN6%1zUeCdeYEiM~>K5-;iGzq0{G=(LxZwNSgi7roz1^LJ7OhNvo02WyN`=gxrZ?Z!$(S_Y4^p5jN%8P0e9tPt<`2}6MNvA}j~XB#x#5h)9Z+E-(yRct>$2>R8pk+x!swu6Sk?a7?vv0QhKu}F=ax;+n=NSOreGL7oK{mK>%lChes<~DdT^|Rvbd3|_|;u=1TsH9WjIelrQg=YwKv?mp_bP1a!L9cr!I&A#5F2@$04gvlvYlwlfTMc!=V;J6^lhl9Xot%xl?PtwP$UW5<6##ugya3aJ7-QnW#YG^eJ@6BQeNfZI1OxJ4wd!cSfJ}v$6~}%&psRD?UOpp$xSjO3Hasvd%l|)suQjlPDwo*-)(@nuASEA-yFc*wSnu)}dGb_rYMPf2bsEzGp=v^adBmLdcFN>SwTPg)ws6Ym4#-V~nvINPliazX>F<@TNyN~FfZv34p|O3FwvN`~UGKc0(0jisjlf$cidq^+sbTk8pym4G(C7vcK^z{a#FR*ma&&4P`o2iY7=dd#-Jgf#yBJM||@UR4|9Y2MB9`7!86>#;$=)v(Yd`?u2B+0fyw!Z!tHhRnQ6WtvL->cE0Uo|=u;k*fKe`x7d0}PM*6ly+fK337dhvi%7I`rKiCyo~PwkI$2{#@LeRt&BC(o1;uz=v+$Mtw7;WQ_vr0L6rUz>4ErZO|@)6}ct|jv3RygA0uWhCmp;@7MdbSEtG(I~&F)m)%eV4_oQoU@QuUcZr9}?QAuA}ZDRB*yIX}PXI-_S3aLDi_=@}3uL6Mr=!SdkYQ3tertmNm)H&*zV`&JB~I@F}T*xndD2?52aH@jY(RMy1O2;dGa6;K}~z9B7$)Sm2|@ok(CLm62%ADWmYA4%6@L&lN*yW(Ue+RvN(ss*ho%=tyNXbMI9tN*V;8hEI%lZI3_3AE;$0TdTfc!5#eY0=+EoTOP~aL!%i#YgJ*2%k%^R4dEn9dACy<)xsq0X|^H|85T^2}~XR_#Z_*R&jQB4(~32NYb`&^q6&b!(8k%VT;-L@ah38uZf3+dqk5t584$n@K(0Ae#QwoKn8|xQ^d6R&B<@-dk}(_!)ul(jYti8KMTia7^Ahfu8a9@~l`v7&b~sNDRhS22?%ennTH?YQnqj>atrBewEYJIIqn383zBgAoDDDxV-}}eeehtFkj%$J%Fc5N1w;K0M!vI#?b6k~Me2EOrhoq(wakP8*TA0%9Ro9(wM9n-5i)7jyVQN+^^Qh_)#MiopLLF(c3wH|ZulL$w7-P2op1AVE=y#Q`;0bH+&qonz(l&8o@R(ZBa7`oe`ZcQ~Z9OBAD^=NKEIB|YMRvCPvw5U8Rj?>_q3pTu5#!@tM09;Gu0`ZE*kbO?^~mX?GJDX_Z(y<@-vz(5cW7Kti$mGzx7|!5jpKt~6wjgd_QE00_QN{Gj%6C{w>qJ!$4lE5@t44;`08W8V|3e!NfTUKbPJRM>WrOLL`y^pCs#ipLAH7x%QRNsf{S9M=ja)qr1o?n0ax*;ke^9-Jd$0}>HMN{pDJ|g2qb&$GodjkN$e!?d2Y01k@-{N~NeEP<2A~ld@(>}<5NwMgAae6bh0WcT7&YgYAJrkV#op{|{RI9pdRiw)g_*%UfskJN6i2R22m3bEnLqsE(QbF;0T=+(A0c^cHgof3}v$z~jtZ7g)3lZlS8mKAap(&(VQR3DKi7sz!-nd?JR>FJNx!mCeeW)0yT<{vYU?|I6I&xuinz|Wl9x<4Fqh?-&N&soZ?xcT&nt-I&@G`LgXbEsTBQ}jXJ!4$^)yj;BaClcpU5_-0^+m;eNG>}=~#cld2zVOr{QWs8h4ZFGm$9k;tLh_*t%2J;X6i@`jwiI$2qvx5mAYqM;%YMa=>4RzxAb{u`wDuULwhJKQs`+6z?z}@aaS#f#@LMn25D6wWJ%(h0sQpQAErIg>8YPCsTrtISUaSBnWfPYKNg>D#kP`ch4hJFsdDCM{f61gOIFY8*(Sl5ix!W+y0^kumqnV?vLO|JbOQzodfG1c7iL$e{KFxE3D5JoDr^>e=MpBuzqF?)JZvY;Q1GS+iw=H)0noG>j-*lBr#G^>(=;G&Phgy6m*?`GKz$3rpEhF4W6WUny15KUYB;#`g3RU|jq^2w>W8U}vb9s5@mFl-lrm#aq}*C9+hraQOIc&9F^BY|hu0=vG)Jz*DH9vRS5s3ft7OY4UJICLWB;(&;~xR(P}=fn(dncQ8nTo^3rgR8AhTk2vneD)ObJHVjxeu@m3I}CpZW+{sY+upL4NQ&o1t-m^#_Xfc}Y{9;F7JX7mjc}u4pt-~trxp53!nTM0wFyLHtd=sPD@5$q!xma>Lj(@I9lheRVh4to+w%)XobU~kKM4nz_$s72fAWQx|qG6{3#i{-W>VKea{D~-(6keZwWZZi^O2ie#(U14n7{5ziPrnqIg1woht+yrgDxImU@cD5?s}t>x%_HJD#t^5?Ejn0z*V$c&ny{gsV#@MZ!&xGa+`IW}crBj63M+pXwG@A8x2;lbu6ML)!M;6|xANM}1+ELw7<6k*Mh>qsU1@6|H^oYM}qR$Yk7P*U9poRMDIzKt=M$g{Bx&E8CN@f=kb_tYz2@#>7X)Dmo&uvJ!IP7*z>11zD6>fAP8tMw&&N;$C8to%ZFn`13kPrU3R~86{FX32eLid;fJ|fe#XW>iUnaE-bi|QN^_vGg%_C_s^8tpd3n-F#8)|_&b#8<{SlLc&-cB&Tmp4jky}VDr6=ZW9@lo80K0zoVwSCUvjfRt3U;poL4gZM6YL>ipLIVIsO>*1z_zBQ4x7FSf{AiKJirUPY-$$eAoyw+RRlN>R^iir6M5b9^@Re{2!t*ZT57RdX1%f_%I62$uXLBth4vhn8%-bwBQHZ|>!^5WM0tK#kpSA4+@<+xJvQ_AJV4*R(0PbzA9g|W(waPq~^=5z1_0AKVck7Chr6#}in?EsYkI37qXA-DnTio4ES`=Xok7Zn)6-fM=idp14+(1Zuq7p>KPO4}6L7n?sbc($^!vm}`WkqprdT2B$W?T2<0lGbQf-lm-|~yHXHnskP$cp%<7vDyB_zDuFO>mAv}{@m+sMp^E}X^YnprFDqgZj5q$eZKIC2AG5_^tJN-ks+Gr?j!8>hI-6T;jG;H%*9`MpwiP|QN-Jo@i&OIe1=kgX3iQAD>aZ?tZQLsgqc{t@>h_nZ_?Tt#~335~vae4#nE93c&^9ryqqsM*~bh?HI2sCscT-vR|5ANtW%o_YDBs5+invh$Mtr^lf46+S~dq)?C6f3I+Er%O2-A6%;b${D(L|21^QCqf2moQM}j2&YXR@I&6oXKE%Gr3&Xhnqs7IWq`k55ejHA9B~j6=G&Z7|GC-sxxU7UCJFI`)sf!+KHP&rqrmfe=LJzvmEsIFeFyWPIr79lianer5$`+|9rJjoyBJTE%n$!7wN5sjWmtO*PMi6L!ch@D>K)CS8`f~nvi{|@%^g~b!Ez2YnmCZ15**`kP;e9o7$%bmm}q$6-4%_<(GG+(EAs7SJLFXbkc%MzjoV1B7Nx$k^a?m3sko-WlXIVr+j;zXa#Var=X@-f!#}cxNz>S<;6!E$WcjLs!IrgNmU%3AG)#SWSv$6jEZn?16U`7{C?9D)(pNDI%byAPwTpe4W#i^SkR`v-Br#&584!hZjNG(3Kh)EC#f1T3ANSycjfk97ffSa-&&#hVOtaXiqu&zBw0sE5DDK+eclaVetmMNl7BKigH|xwYvaR!rsPnOhe4jY`*An}o9$cOY=fz=a;du>Rm7)pzY0k~6~eFRd`_Td4dD>xh*DSGZ0fi32th1)ltg=az{H_CM8RM!V(f6u3IE04sM=S9nlJ(WTnnFi-SO`=&the80`C@OgUycxU$N>1JX-5}Q}*IV_gwyGy6GZ-nQy5tu|gric>y4aU*+))6qgaXgt;UGSZBHN(}yLwWQh9snqQy)$o8-y#QeLTk01;c9!RWX@Kuv{$FoJ{IX{LQ4)?F#KJzliLLaU>(U_B90Ox6>1hQI@2e8$^(v$e+9Z}Jg@=LpRrJ(i({3}wK>i{Pe!mc-WeewfdXku?72F4ZChzM4`SlEC0CxA1fT-oo;5Ydo;13x|JX_E#!VvzG3LBS~T5g2zF0sFC8!$z5xIE>-xf%#G?hd%tU+e>4O*{j*4Fk#H^Z{OXpz)m%JqvSwf!%nVn3&`X2HUbw2o>+q)KTdd?$g3muf|PF9Hqu9-f>wGHkFV}eqFEkjNS%V-Q1_kb~aokz^8K{(nt4y2sJ}foTESjGm7PaCrj?bF%oo(zmU>y^|N@zR5uu*G&NE-?RqXzxyb6Kjnj3?(|!kQxhhtm6P<%<6pbrBb{$ypWm3$9ijM@H{k(f*N$2~}KdoOvoj6{m@1d4mJdp&x$oq0A7Hdr>axJB}(%?#2|eG#F(HVhAL|11GofAx`Hcu!-iROn;HN1}x?IBi--;T-?~tV89^7`D0rGLw4qstl~hov$cJY-l~+{$NI~Stq3jpBH*_^H=2#)GR*PvQF_G8!N}kSe?^%ziGa&R`P;}Qk)tZpP^mx?})vO5t5)_=}z{ak)@5X01heBUQ-g~X$Z^AdK}+F($9%3BXrI1kN@5nbEHy|t)$seoYGVp>SH`d{uijQ%Z=%&Dx&e$#9S$~;srMe#?C*OAbfq=3FnT`Yt8V`G_T%guTK4qIy)_v7cC=w8@NKbS1NdFVbpJ-N}n5HgD{tsE0zOW-O{&(H@9ZeRp$VuGFT?&7oEn~8u1%y#B*+lyJwE{v2PmM_LLo<7d*w8JF~nm-8GOJP){#)sz{u3*ivG~EMPX_0EKU$yBa`e&+oAge?y!(Kxzulov=8Fct?E8L64nPvYyUESdv_WcMm?=|}yx|0wCZW)Oy!OI)$yeGImNokW&9eET7o8S_|m>o)b(L6bsm+{+?!x(-DiG@0T$OZLd1ze;bDgS#{S~ZRsKM?xfl#$vt^vH%bI1qU=fC!wwkBu3|`KW`k?@L6HSux{5ta^@H>q-vR{U!^>Uy5;&r~bv$s$fEa#Qo^Le^fTncEe*A4dP}hf#voSA)sZbbQ7uXfc=Q82!q0SG&r)4?P%iZxhj$?z$&XI9!npph)fTexE9_*MKY+S!rhc!{@|JxNsIZM^G`%coFLQqMPAy|HLb4aT5zWk3*>vfO`!Fz;L1H>>{7S%-aXeiMwRYGY3ncecj;t(Z!w)khcgeSyDp{;J^^Xt`L)VLjDz+R^hk#rN`9l^?wlX2xXSCuZDK7?7{2M*lGhR)&*nZTK5J|BaGR8)ASTrZ+GeP*Mo<(dW?Le?#{=RS5`S*Q4aCW3mkzQ&5dbR{jze5vwRfz_CaJJ+>pp)A;M^1nJjw5iaRpn^!0)I;I$i?)48vi`dSwmbrr)r_m9>1m>lH4Ysl-3og)P-dA;iVo%hAEj(+1n&GHT7i-tl)rv5$zOiSZ8)jvRW_gb%XE_MK{@x$GKdIp33VOfdBEHM*RhE}S4)Q{j0xjvO=UL68aP3Yun+aD>yf3<|^PoUC|T8l%pa58phv=14)CoLwrl@V)*U(n(RS5IXEdh#cheu)_~Q}E9*iZ^8d~(YF#_>koBhMsFETpb)BM*jdG95Qb2@ceRZn8Odc}Gad6l&oVd}cD0_q=rucF7;ZZ^z%vUs?rq&xUqjBgv3LCS1i5~^7EYzqfmj3oQdl^(taFu5K^pS6p>WTd~dGGjV8_PxOB)q0`qkn(v!sTo%DyvNgX(g0ICv&o0`bvmEpPc%EF%?lJR$gMxh~16>!lDFxwr;$f5kUxh$9;8x(O~$dsW{#7Dp*2*i}_@S2|l@2)!yfQJN31w3d`Ql|>6ZK`2@=wTv-XEbqt&vbY~g)E<;QlNsO5}yvlevwSm$zq3NBnaj)B95=z^f{P({0!3uqnGV{Lg!|f=V-%J@^A>c+9`agV<~K?@cJfcPK#(qn_6foN&vkX9tX<&fVJIHXAmb7^@3U=zNDs5V{yBJ2o>suuepmn0z!6Y_Ry`V``fV4OGpBKfxo{h+(DL22XYX(LGZ{FoZ=5_Q_^$(6+oWagl}LsTEWPh!m|ZFy&YIPl&^N9AYPyFLlX&HF++fPrHrJAq}k2M;=nH5X5CRUH_YwW;LrtQ%c&RP27nJnR$`_s^#`Uh4LaQeZ24KYaRi*g=uH3a*4k>iJQI8I>^hy8_4emlKS5uBrpZrxi5;AvY#dmbQSZs0ItKeyjVd~`B@6|!bKhhhZrbhoCF)J@%R(lpoub~b;=LPXz#(XzGiVzl-T`~#8(YGfb=aLDSrthT3vaPy;#ecIk>T8iJ`Ig_^L$ItFe?p17nQhhv*VDOgg!?q%GE55il`y56n^b8-(>nHvd9xVIPA6=u8=X6B-Kgtx1j(%k_Luzg(-xiAwHZ6+B7nWl|Nt|GK^xAX13ir?<8nObF4H8_$-*~T~NREV*X%^OtVo?A!dHkoPYyIIVTjjyP7RAiE$86hM!|>Ji`R4Z%l-?mcWE2R|d#ejQhq3w2xY^ELK>ZL=$GvCvwD`dvC}w;lYUqJf;Pr76M)p(>ZGu{u&Tc2C)BX8C+q4VM6qM757*0O=H7x4s+nZ3i%ZGk#0#xfqs265?XF1oI>5;I!YR*kYLXW%&eo|$|{>rZFP8S&9)V1v{&Ongg_el#OorV%QM*`hw$yyE`7|1#xS&-2E(K%%e^Di#x^9q@Ux{&f!^i*J&Iv{vC+^8kQFt2f3DkJZ6%gppZR3oXp%eKbrFw?rm~R~mb<-V2~fM~a19x~lbAuFbcxNtl=Y*ejr>vDZAXLuLXUsrh6ZNQjE{9y2HUaM)lV=dfG@2Q+a1MrNS6p_`By`)QuPu*D6;wQ4*qlBUrL;INvh+r1BsVD&VSHpdjDI|b2xiE&?sBP|3gYnd4=Zfog|lGUPK4wqSkfE?see<`(iBXEL(RYF>2okJaGmRiVkGS+7pma$tAg_6h}&%h&5jh7W0@86k{ZEs_FA)BTR50Sf$7QU5Y`Tl13tBh-g;QKF4`V3nzbD?z1eWunPu{JD0Z??6{ptialza&Wr2UgN`7b6r0%&_=WEyXfwS<ln)mCj$-wrp8E9VC$#l>qi>Kio;s&{^x-0Juzuy;E#OjstO1dSdyBA^n&2#0I6`%eO&xX{Jhv-E%-U3G5mXf9PqvcBq>m3u8U;!yjY}eHu^sB7L*8sT`Q55sDb`EbO3_}5me59B|rm~=fr7{~_#8ts9!?0g+JzdDI#qd{yDk?FB?T0R@O7>;3e{bD04yl&ptvAzHdVveKsABdO0&|?QM{iLU!Yt;h`2S3nF))~%uo3NR&F-8cRbKFhb-J(+Ldf<{uJ@U1S>K?g7E0uZ1EIW-ZkSGTEp!>y8q4zFv?VaxBW@BLTuI69hd}3QK9Op5$Th)e@wJGl>%;(or5X3&{EsY1CEs)gT{o`Jk0KSycxqAX(X|xsft}RVpw?qduC+;r#Lkbwm&-JGPG%UDrB1C82o>#bu9|l)>&SA7UX-oNB*RS|qZ)*AzJA0S5wFb$TkEpB2Hf2zlhX__r!on%WZWOYcCj$6WD&Jl(kh13foj{KfTQoV<~&IAxLp=B2f(5?w~XRSe&Vmc0Vl*e4-gr+-Giw{{I7n%1xIu){Wkirv&k7?wt*h@g2{u#H9e{<_fjUUI?ZXasR~;9tdu(j<^i&`rT4T7TD238HHIU0U{GruAEoQ>+^!zN7C{HJ!X1G+*ol2hX`rX3Qw~C5b0C=u_JTdUBmD<2M`$31m*-WZX1LTq+p@MQH^Gdn+%2WpKn@LYA`orgtqSG+nc7vF{;wnt?IaBr&%!4Q4W%*3Pn3U`6f`{dxUPs$CiEP1h})N;b8yPE>6i90B97)=teF?9ak#rkB?HWufH*QY_M!qe;<&$~ep9dy;MKr~q&;W@oqT-lIGXNxuZn3^?+8#u%f7-q5kQ2nHi3#cg*0ujlS$#A5Etn>ra^Fl+<3CXNZB=OFIfFI>9640i4d{Ot>~?L@kJ%DMcRt4vkSSUevz-w*A3C*>}~vergv+4CLhEkA`)IoWLko~OxZAaXFHUvNCKK2()O-(*Ij7DV*O3*nEo$$P-i9$H7O^yN5QU-@Tg*|kI`{ia7w=Khge6#F)f5YXZZv^(kdxL&?Pco)P0A(X*SeGr%jkL8xqU$#!j`~Edh8yg54yRMj7z@!wttw1(JlF1eOu+XbQ73*68_4S=Jda<4JU(*T=5T<dJxf0s=A=6#*yj4?nC^agDx1fPF|*Q_+IRwurNzlWrfCP*`_#@6p*o8yl2JuIt#@Zhy}}sWA9B+1Sv>#^`SbBvCDlV_Q=78L2BDpytkdAZgse|zKelTGGj@{)BY^@LYQi_FEYy|2s>=Uoj3mW-x_#41SP1w7d*xVgKujummDhZt9_D(+1`14MeCOS4F;-PZ>RaE-&e2IZ21H+#bj!YOyr9V!gZ!BP)dxFQ!jkpHt0ES0$kypiouzjzap)67Jq{iF#6elyF|&F-!tDO2Uj~_gAv0p!Azr4c>)&w`l04>rZvTrp#4@dat;WzllC?J6SiE8kK?qc@t`5)Auz~@xz&ZA?f$+ZQtysbcds6Bw+Yf0l0imI^BKD}?6?cPe;Z1*Vv+zwjF`OvoHN9>mC#6}wMMs*Cua6_COP9xx^>9gX~1NwIbUP*oiYy79tOj$(nYTZ9znSCg-364EH-jI*yaZj{VR&hnwpFuf^MX?Z})JAn!8DHJA&+Skd-?Y@WaWm#{dy(^UJg3R=4uFvIrpo;{Wx|p@4?Y`(mbeZvOflfs-P1cj~xqjgvXMX-L|Alq#N_3H6H_((a59i*9M#D1vWFDF|eS#5SOhGg&f!mfzyk;zCSiT%WtCHG+vwYfI%RR*9p@9}o#)e*IgiC+HH~Ur$41Sqhy^J8~}(;bdN5WJZmGl7=kkN1I!)nf_YY={bQ|CI`+Vklb=_A~5WLkLBk+zvOf=I^k+{T>YuY;5)-iIW#dl8_9i&^o7-#5!)g#T73JQUw}#kX`jUZg1;S3)N}UGc05S0%~nOAN-rf$g{PQ4SdEB40iG8c#7hF|TEwpoem;-)^tyOx_4kclNmO>6(m;Y$!YlcVDz!~rtjJjPW&@qNxO&4v&3_2f9y^^xqX=n`=q^HhP(|%Li=n(rZHCUe?=}_#OODRmmefd+2?8^o6N%_1^M#X@iKpUG{y4()mM~ue?wZns=736?H=b%4;bmHt9Kbk^H(6m9W3knT?M3$Rz~}B%ok3ulf8cSzpZ>g~+cZOncCXnMkq3Ri>$RAYRvgli2G4jYKu6?%MPhdVYqo@lap>HG;g+LMGotZ>m_x7E#p8m_J3|?e$c9lWlmDQj0c}mhO6@7ae_#4Omkr2k@3*zm4RGF46K^MC&BJBAFCWF~`{>i?7V-y35kvAFtrx=%0oI89kbiZruyHLCjVjpRB8aV#TKmH@UEV`Bte_Y4uSj>74wGgTK(l-2m^PD<)0njDhP)0R&^EH_COB34Van&n!u{0aE(YZ^^u_;3y)&T!43Eo?@qh%;jpe6M(%x?~v--v%aF`H1pWk+p$bXXoT=1MYiv#2x5_bMCKQsuKB1QQR+Z8tDZp4z9e>Ty)9TJ%Lc-SNLnAo~PGoGuJv`t8e)Cn4cClrP$)v67v3WmvaQ-xSXl6?BV{%#wX65n!kI%E7nq%Wtoj5FkPPf08)ZytAiF+9lkGqZlH<)m^yw@TO_Tb~>m4Xi9Z{UIY@@eA?pZ&IfU62>$3uZl(2h=|(d~~P#t><+^Oi&~mT|UYPG6x|A9RtVO?c=qJ|%Q@9-mE{`pTm0cmfE?bb$@pK^!h7JJ%o69Y4C^;{+H@rP!ab6m6&ZboHFhFAu&O#&+<*&~`5MK~`tL?Td2LMFf)0kGZT4?D`YWiKqTqu{=||g!2#B6z6{l5UC#aVgMPO4i;UjY`CNCb#}E)QwN-v&z}%{5Nr-{*0-Jq!U1Ph2KFOOEmv$T+#@QA`ghMarz|O7dZZzMR6Gxe3_IY=dO}2$&&F)sd_|iJtt_j!^_uk_gYfknM#j8or}W0GfmeXCMIMokH|hn@C_L)3dHWF8!b&9M6*_JO|%q4SR{Z*Mg{Am6ZDnC~lVLgctb%e0Z7nJ!8pR_=O_(PJ`Tf>8~2bGfy)(jeG69y{1QELn#Yr-N${5?Wd_?i)^Z9PBVR9B4%Kqyv*>*0XjePCuVB0sZRp@q~=)}bS1_Vf3s;8+RIuvlad|K%MB)8!Mk_WLU*oS);XMQh_JMWeDr`GhfUN5ZH%?(cv=~Q{c=X+06}}aCIB>b;N_QJCtgO_HsQPO+!n7A#zVrWgWDsasyVub!-diCRPVxf4S6q0I!x4e>>qVny+14IT`XClP$~NZm}i2Hmr_yvgFF_9=|{_J-D~e{akD-OTa`=R%Bi{P84W)nAmyic4|~|F1&rzIV19ajkxDT?WCIkS;$9d~>>HzL4IvUhGP0x4YV+onsdxeFGE^5>W%WRdWHUSxEYbqbS#!pP{Mt7nBbUnUlJF`xpdmcjUK`vmqYM_BM<9Mg|>&u^93ys!Akp?q3Hn8q(;B~%2*83rgkExNT_Lb=C*BjGX<_=gBaNWp?;edmr$|7(Yfm(3k%o+dn?Nne$?*NNPBB_!kP#_6(tc}TvLQK*LYZNe5Vk*ZFrsxFTGTL<+_E#7r3yTP@r8L|BO?Lc++D|psQtM?{t{DuSj)R_C{Upk6|6r3z44ax76ij4`7vqef!UYbNd*gYO7s?KP}zLKtoFxsdd7#kqDsrq(#St~y){$KO5XU!wV)I$CH^Co@lHHr74r#~-8jyzZoDX%aljI;CHGG&SE^zXUt<%~`NFeWT|`Xv)(|y1B?T2O4NXvH5kwYN{8LI8wR_{)K&z^Z*50o*b8M_n1~K6CUnJc*`N2sDA0*d{-4wvRg4?cnl(9visw%{_FV_)8HC?maEqS$}&+d@3-8R=Z~Z9oFD5RX`h|;4T#7#``}g6_MD{~f0B5|3Xhh?(4*08N7whKyNI5-X2Y7eVlsss6gvPTdHJMaqA>Hhh_caVOBH(nff~y8kViIP5ghCSMO7N7^ae*U*hPhTu^z#&40~aL$LIA*iJPPcrlh6OZW>*G=7Ck|!tOM$Kzwc`K-B7@mL$T6(O`MH%(f>cC7(dLdjd?wTRZ@@(1G*Q={9z=(Mj_I;!`>e+U#{H6pxhQKmqSY2r7h;|%HoU*eTlhAkDbfJd}yLp`V>SLA*8)FW^&eL-U5urWnUjbs+Sh%aa{5egG36%wbrx{~<=8&0M{mcb8U$QmYP_J8?rx%cY!)?n>T`sBksZBIX1fvh#!cfY*4Gw9_SUUrX2$JH>njWEC45KL6N#k_*c~q%+=u`oe;oE;BazOp;YT6U>$y%vlPxdOMG;rij=3cyUUjG1$R%NtA@QCYI0?R|PY@0R7T+m!9kA+r04lk>{Z51&&cpJlnSqWYvXb3Q8+AWP=lA)*UToVLC`;%$Z%cm{u1eT0C`kfqLYzL3q91_+$o^kw&Es(#yM9&YGoOw8^{+1oOp{1fK%BDOiT2Hp1*@sB>(ph(g8Ih-y!~hyBK&Es-K4TA&IX4&_%ndF`EN>NE-Cr>fvw@D&@EQg+q4&f-QOWYsf?MFpC%3vd31HK0=H@&p}3q<)*t!iU&^8y?oODSFsa$g4|l6PUY~F$DZ$@*Q%9^vnn_a8UIBXgHWI>)g8@jI1>EIBld=&Os7xM4bJ0*T&8$^gr4tsWo3&$N%e{Bk%mJUObY?3ZC1p;iCCpL`YoiLxjlJk01Q~1!o|5TB)tjsmAe@eZZlwuk`Ia&)4C^)j$xP@?w+Bj%!Sd$E@|~*+p{5`j(kzk%;&o+%PA)Bn_9xXNyIN7DuFT}>ggS3)D=o?A>yz|Uplf=qY3|D7WBH;SVyef|H<)m`t;-CV|JR|XjXbV<)`qczz?P>M+5p3qk))KcNQ&ftYs5VYH=Wgm(9^@>>i&{2=$}r8r38v9OmRVxdW0+nzae#X}su_7PY_z@2CFUKwH#skRqTWGtri8!u4tyqEEBWI2PO%}BG9CFtbcjHEx@QQSKbnrQzS69pY{2`$Eco+l#E)zh$zEuTJd!#U$Yl?P2ayoGHWmA}`Ik8Y9k_}Rcvy|MbtCKpnq;Q$5uKcCkr;b&sm;XN-A5IJ9^D#4(eT&6O^`o24tl!CL7gf`;@G;UgmNKYVro;V>u~E8y7pFMPoLYyt_#!*V1bSA6>1$5aG)uFZct=c!v2YOre;x&Z?LX0kMVXX9NmN!cH@F?nEuDeJmSuHX(4hWllYo)pRXju_Oc4^s8>-oRAg~+SA_qlQ9O5N+7HO)7>sv!TM^~3|>FAI#}8Qvsw8R(tMQ|#{}MVZ8Rob01t{V9F$u23$#I}ZHw4Gre>|FvMvk37J8TMFLR!Uf+zODJCKOr#)Sz|SHS*emV?Owm)|kaYj*E*_Z1TE8eDx)Ctr%4nN-9FB>j!V1s2;aBmvD}7nY#yV+L@228nJtccZKJ{D02Js1)Pw^s4|eV!eJI!ub41LKWmOxur;>6$%;q7|X|sX3RpIj5EchNQ>pJGDO{C@kh-P2-w|-IWspkT;2*#+n9g30Y8CZ4fnkYBhlb5cL6HXfS^fL=N7kWkO(2AUklm^F3Wjfd>foaN+EUp{2=lCj1y0k`ItR)S$`oYV9sa{3;rLv4LA4wYvy}}h5xyAM51XjQ_n9Eo)1u%dB+s&~DTrVyy=o^P_fKv{ofnzj@mmfjvvJpN&V?dsx(n+;e?$nVLEZ9dV;6(uD9APDe=Wg=4C(d;f3i5YxXPXp*HH)WQQRl&5(7b4A+gONm$oS`u*{D>OTq`;eb^k*a5}*7ky=EJ1s6FsI|zik>R;NCp}c`^wnjK^u0Iqaje9C37ffSH)?8_1R?m%GlVQmQZm7s|WD>#sneEwW*0X_*)|Cj@@r52mwt>p+NQN@m_7rNG0V(AgvE^z3mC=+!FUOE|xk=0_k8hj$ZbtigqSVzXal2>|Hwp9`FL^T9@o2Xnwt7^jB@Doh{d%P_e=*IMV6=?SE%o`2(XCVoQ(a8U|aVk7zX9(koU$auT2Cn7xi=50z!98}yqhz|_79sjmqOs~$>n{ct?ySs;uN!}GQ-kTh;wjc_5PqC{C&$QrsZ=FyL;FDHRvwNU_f3}C1@k0&en$p4o{oGUI|90G@{xa}+V3D#hE>@Px6u7W)*@`wlgp%RAjVeHi)of%IvDEwzM{Xd8l>SH)@LZf_IB5DyJ$!TI!g7c)eDK3`p(rd#61fnrBeiMOOfEckH0@Z4N@8*fRM>(3?%{S!_E`_saz>0VgW_qS9%#g-!14;*LRxDs1^Ffz+!<&UaFW~5a{(83HP_O~`iNj?$G`L}OZ-k{h_d%);)JzLmP^C=&r(VzvGEHzAWSd)D@PSfrs_@(bdjYkQ?ZEKDmBW3=VUh8*XzvKA$LHdJ)=oXF3s0v6?CakpVO+W4iySKHQ^#sPtTgv`O`@PNO4A|_hWep4WA`Xf1YR$_yxUhIC`G|_b)WGA4e6LXcBfHNg+ySV_L$xsIh>~vUKQ)`xSLXiEHG1`(b$w@Jl^&ub2d#&+c)f*%xU^mqSjyZN(-fh*pE_JklA!YRzlQeov7U@f=Tw@5N!M;w{*q`cRk%SNK(_;dP{}DtVlbyid!eA7ud|JTg%VXEMTIR873s{-3F?{!14pNOk;e#gJ#CEMwM$^X;ue^ik3f<0M5>WIi9Eb+^p<8CqT_6lG)h$rBE)L@dbmoEsu?=LInezxFW&4m>$3BJb^$2bWcgl#8x0c*T)o1o((_4qcXhl0?SR?gERKu(59cmRU0TP{V+J8Q4rM%Re`ACxUA6)r8Yfu(oXjeKVCGt>&laB^2tR1DO9yF;-6e4gjPHEl$Kr6CnCd()wj!O+;B#01hAdsDWXq)(j~*@7%IGl&{~xjvxx2ovKp`94+h>vdM8o2f$Pt<0*z(P;UN5D{svh1)eeH|St!8+QwU}Rvndbt0~oqZP|*cJJ%;k={M}g+h{&k7GhRB4JJ%d!7?c0f6ofuI{6>Yrdlx`kVkV&Xq4I;N`@DtY>)-|Fl}X?T9SNpTrlvP4owi(1uS#dHK-F-$w0sE9&Md<%kq@=$1|gZ*p6`J)DJ_E6XM+sfB8i1e)toc!{6JcH;Br~f19BgVoRgDVYl9X}&_2hDVzjMgxLo(+t{0Mf$gNj(tebI=TMI%606N*fX_85khF!7QJ?4#Z#aJBXg<)-RD5BrxPazkz{3{*|5Zzf>-tY1WXRHZz3Z~%)>S!AEnVux4!OUbQR^V6G&IcY)B94I@JrjNp6uGUo21dJrb$guP9Xx3v`Wrbi%SR6p(=Ztb6qHsD#97Xyye#*)zfX{~U13YQy4KGROi%L@&ys`$tn0D0h7|{b$yL%FPP?7Tk>m1Ih&5)ThY!CnrQ6tX%?|v@G8*bqE_0(~)TLGFCJC7H$1OC>O6s60Rhk#m79JSTdd>b2+9LsqFQ+cwhkN!hc;WV6>TETim85I$~#FWPFr(;yVyKj9oWdsq3vwbx(KcP`WY`5jpT%c%p>BCZeu#J}KnfRakt0_t%rFp@O=81e=z#?wiDku9YD=}*8?B4R5L%3uSm$fMSSBZcqwel0rn5=0sc^LMeFY3+R!vtVMSkd&i+;us8$`M6IaWdf_&eLTZ2sD4hUizBsJV$J9Z%RC<{5}bA$PmX!RZP$YeI|Eu=JrF}4yqr2JV86&(R1%-_9%*xcA=!J}xO4n0=u_6$is7|vZ=Wy$49<>ijQlEo{nU7^iiqYz#s9bQ>_I&R=cEVoiK*k98C>cdo)q6d>1ulvC==?&;q_T|JX0AJpBMfrm-F8+Nb>}pE@nNhlt)L{tc1>|bYTJq8ECj^}Q?-rxdi$o996LQ5Hl>O24!pHYBcpVqZwbWYl{ZSNDM0Ne~$z>x!5y3o>5lmWT_}p|I9;cRg~j#w$M`#--)s@YS*#)5d0sifvif4_fvh)Ekaa$I2Wa)0DO4MC0#;D-lUICvF;wd^sz(2Vr6*{k_vQ<4%WRQVmg0AJxkdJz2C2!A!@%4{gnS%N%EPeUT;#bF#S%+(4uOZ>DvCOp|9(hb}yZ1NO`N{5aa@jx96?N|02nifkDqK60&Hmc@0f)}<#&#OP-06LV1RQ1x5vB9Hipc4^qEv{~b`?%KQ5RTR9sev|K2x%Y^_WBOLyw-)yZ7D+EoiYj&L;g2%EIE*{X$=Vg+k0;Amj_KWUKaX$i%fL%in#7m;8FfZ`RG5FWK9~+<3qxmBpS{wb!et&`>8dnE=%ac5-}Bw)6Xc8NXSw#?&Vt@4#kgRK25(hfRCw*49dEE%?*|a0?dx1dprl3`NDmtkE1h#ifC&1w+RnzKR~G?h@dgJJK?jo#Z1?ZigO#WD@m6w{qvscN^AQc^vIaGG7MW>)wYjnY80;@Dy2n#fyRru;vw`Sd0YZTQJK#`ZOBFJP#LAxizVR$kTDd!sE+BC!qPSv*Iq^?B_CZ2o1kRGFCX&&SnIHncbNR&hkEhClH|QB3pjCKwPkkhfK4cCvrVBpqK?A%_IHKN2b$jeM9CI}zV*TEG*2mUipy9>Z=8mX|deMqo1RbT0I4F$h96*Udgu4GfxWcH^c{CyWco&f3X0#G?tipL79D|4Uk{7QS^SpEr^p8mLFN?)1x>QTI#;+$K#vMnw`jm)I|22Y*{~Lk+{3e*sTM31eze@p{TypR_K}~!R#tBryR^(THwUZ-Z@zfGpXxhqj4DoiH!+(i`|K$4VU#Hetl6`_R(jk@=2NlQZ#6802Nw`Q%8I8@lBU#ewC$V2eeb5|3m?RD)_sT%6WR9S_4Zs=psLH|9SGduT&mkjx52%O=-qd-(HXveGBaQ9;B)Zx?SV+DZ`00x^^CIWNYe%T%sw;4Y*Q%*TnL;jBSiUSB;Bdj(A;=!oCc3`Ake6j)CuC>3ywy?zo&<>==K}tzb^~8Oqw0SOtrM*ePi^66?PPUk7c1-V-H;rL0LLc#Eq+Aq)Yqu&!SKIQ|oYjVXMI#n)w>k7S%5gcx@lG&m9TDRY|V3X0UBa)Z&aVz<(#vZb799sZx~B&TuzAfm&C9jJH+y_b~a3>HSX(qw%7FxU+Z(F>A3;64p=j*ZYF3Z;JLty3vCZcvHc{ks7ub?%|tV){U#e*UZZ8kl80MR;+^aGm6j9AntX8ol@|*4isdH&XB0I@Ek8F&GAtkR)7G{YnL^+D5Gd392=OHfl5)Ga$W=2dsgaCa!B-n4`Fjt0$oMKYY-N^@x!Ba#vYizhK3VDLEC1<*7kS>(!!?diha>Qn%=bFK{U)XNLk+oh;I8^%!xH;HB}3BaB9*EE6aAwXBDmj~t1R1gNsFFP!JT$TO)qQnOrt34<_4^vmP0e$m*ReDKrkSR8y;EO2sy?DQDSVzDbbJ#!wEKKOaXf@=>}@1zbb=Ei$IFCGv#SiuOX;-3dvab;h-T+?FKd4u2VbF<>O4TnTl|Ntx2>y4X^H#m3Z_bdKpI4@5_QM3nHl5A`#tNLw{&F?CE%$SFzbLKssjs1KzLs=yJx*f6@*Rdv;9(wGy7~PhS*LwMZDj4%;;q1hGNDa}Ef)M!`x5wN1t1km$YEenn;`LGw<{@UL-xsut}r+m3UT4z}R;qyxB=crEZh5)ICBW5Ww<@pkaXIaO0pFe6=u0d6;@iY$&${P`05*@YDAjwtdEKC#N}>9x}2AKz2=Nb#R>ua7;<-%C8--#+qJTJ!qQcq~fRSr`W@AHc!*HM`?8V1$mIL--exa1Bzw7tw|_%dPcV1q}-dXIn5;Od!0kcbBGMGP{w=+S%+Z#L_ovqv+LI*9vE%#U(h{&J|g@oi!)7V;LevRz}tsjr>%k%|g@lM$2D=J;_<6LG(u7cago#WKPl!20RnOu9Hai5qScStGJk^RB@skq+(@45==IWCFaQljuyp%ZF&H`2ck#_O$>0$wwc!93{W^;uYuVwI%CViY(YpL*EUKUWDTj3Crdbn)t}1@O7EFkyedx!q>h#mm@Tp+rts7fcG{~l&pPeI+6d4lgMBR^?q1}Lbu;@i)i#tqdT*rI?6%dO)EZH)-xdgw(P5N>LGI#OyTfBxiNXN6Ysiu2A_EqytI3h$5%IBQXDSASsJZsB49tZ?}pdcgH5y;O2cwQdPdO(?B26U0mPTu6XK<0aR@A{otYK8)74j&#NDyd7m4CM}AUEBC`lqmmml%}hW2ScP4V3>c?Y0tYc}>NiV;6q5R7DO$fC67{P7qC4*1x9g>d3w{{Ij{CiEWI!~^T(eI?=nCT=lrPtm!`rowwnH|P2FG!E5Q}d$mT8a+g4b!$O=&PerFuqvAZ_$~%6N)|=v;76dY)K%!->J1gOxnO_~XTjdYBU45aWYu`!URJ3(t$ym2XJ`=9*YjzoQjV+$vaBJv;6ZWPV0uzA$~&h2DmdYHR`J5wxeFs64o(<&i&vH|rLE4RgckcQ|Lhy{xpIbZoT{>o6uBIlDad;LjlRM}lSlil7nsZjgFzj4wsH%!Mr}vuTm<6bNX2s7-Q^wjhmnx%#r@uFHZOY!;5=PIs%F6wvWlTd)Gb?!u(a@XF5_rr>3bCEyKebH}Z~N!-n?;0EEzM5-5K2mZrMw+=aCATgD^lmk%-DZi4?8`lVurHUl9uWv;9C_*oicw@ZB1H?21cJ5Xu4!g(iU+h{XX_MF%MA?$IU-?=+4hZ(v8pAo?w;jA**}6hI2|mqCsS=s9ir_%OU`4ISbdIeebcOd*;>Vte41NK9;VU@-IF491n6%76|2$%-^6>nW^yl{aq9hGlghT!j_dxla66o|;vV_W|q(dAi+yzkSX6n(nwOAyEKT*D>4w3>lo1OQ=k4hNTEzA5L?at@U@m5)zowL(Xo-{7q1>n1j6q)=&lXijcifxk1s4P}{b*|Z7uzc(Y!jOzo(zc!iJEvtpqEkZ{_K@tHrOJm_bYbG@z`|x#Zm2j>UY-t6WEF*+~;!W4~UUN}3fXrdMfHOx#O%ABAr!LyR&1^UW7|U7ZKRDIN0U=GOhTaGE9vF4ylV%6pJ`-!iGorILTPU)?#965|TBvM%z3?nVhGcz>ev15*vU^-JPT>CMfO&He`{im#`s1vt~${o3bBx2T%#PQZO$C{jf0iU(jb?O174dpFNkgZ;W2#wL4|a-3bg3Y+NHUFf?9YRXDWvd3Fz>vFk>YjSbI0vqA)LBm58^)XDa4|^MD`BjG0aVT%(-09U4tYI1#4CY)QX<(#1S{xgcXiqv>jthmI*M5>#u+#}a-;J}k=1ePsI&h|wtuOqinqU~g6F9g3abP0de_AM)MvpPlfcw)3D@R4u}P2p?=VF}kJ3OD#2|xh7;Xhv+sbxq+bVsb}fa{ecR|L;R$SGRyVGD{_h8w#%M#c^OgK7)`4S{mET?_?v^~8DS1Nkl%|IS`gj#CH?V^s}Zh;uDb^omn$YCkral$j$-|0hAxHY%hgbAYYzw233AzN6e$nUr_Nv-asqL%v=Usve>Lw1D7(R+Gt~;Dxl_>b;4H>WeJE^UI;x;bXxS>YE3JB_wxG;Hltd`e2)f=bF*Y@R%cOk6jVuENQU+yBa;s%zU~SFJt@!0C3CQh1>NLH9LiIQJ7-;V1CXOWSyH8=*1Y>*ZL5)g#johXAphjTm^UjE-&=rvG5C*wiEaF%qtiM_TDu0e8WO%xCJ)fV)b-~T-{@B!$cVWd5ZcFD9Mi6S#=YQpZKFQL@M1MMINMRK&-4+QXmnr#Hy6#`u6O?%+yQzEs*Zrp04a0ftwN7FTQUk3Vos1Sri_@I$VSS-Zw@}zi&dSL7gELBr1PFrKP(_Cw>=RN5T}3$X}+@KGI1jj$Xb9!Lj9IzW(k+GjeN8)6xz6iS46s@wWJO3KxrDlUNNcT{tS7Xu30>w0+w991w(Q$!&xWIU{{j&$y8FTP%FSai>zG3?F~sv2{(t2k}-w6Sh<$}EicYz;(l=svxBfVe~$%v76bwU!>is8Jyfhau#{1Bt()L03qfXe+pO4C7t5($#7#;l1jFVMO}r00VjPDT9OQ#a_Zi>wOQVTg+&o5gzv#`!|AoA#gpYH+MUM*#s}T~l*(5^qV>ujlCr0Y#4b*tdGjG;nqd8PJ&l@6nX1>PG?axp-aQzd)#hvcJP`gfkh5*g?P$@c`gV(IO{)S_ga_)-pjl>M8NqyCwevbv$|brH5ph`^L$Fly0zDVgdE*Zry~mjr%B1|%lXm6e(bdvwl+!bua%MX!RUWCX9;O=kTBfY3BK(RA`TwPEZ0K!y(>nZ+L)BtEBSrL604WJsptpU4MAVb2rJhj!aY(T}O!+HU}_1>NXxB#r}lgn1WChkaKIs6@vj8ojT1~#rBP+vIiDbS|;XvTaI=y0_G#jE;b5tf6ivh8)prUW~=61cjy!WbjSuPDPG%ReWm+mx2}`A06avmI3jKN`N#*GZF=;FS9`4Evp~?f`*clspwz15x8{wfHZ8f(Zito2mG|mm>!GNVwWmcLOgt^ciF3Hn^i-L7=R?xF%GK9)p+YjHRTdU0UlAq%vBYQl0ssI200dcD"))) diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed1337.log b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed1337.log new file mode 100644 index 0000000000..d4a7a245a2 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed1337.log @@ -0,0 +1,233 @@ +W0501 23:00:19.834000 292968 torch/distributed/run.py:803] +W0501 23:00:19.834000 292968 torch/distributed/run.py:803] ***************************************** +W0501 23:00:19.834000 292968 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +W0501 23:00:19.834000 292968 torch/distributed/run.py:803] ***************************************** +Hyperparameters: + adam_eps: 1e-08 + adam_wd: 0.02 + artifact_dir: + attn_clip_sigmas: 13.0 + attn_out_gate_enabled: False + attn_out_gate_src: proj + awq_lite_bits: 8 + awq_lite_enabled: True + awq_lite_group_size: 64 + awq_lite_group_top_k: 1 + beta1: 0.9 + beta2: 0.95 + caseops_enabled: True + compressor: brotli + datasets_dir: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved + distributed: True + ema_decay: 0.9965 + embed_bits: 6 + embed_clip_sigmas: 15.0 + embed_lr: 0.6 + embed_wd: 0.085 + enable_looping_at: 0.85 + eval_seq_len: 2560 + eval_stride: 128 + fused_ce_enabled: True + gate_window: 12 + gated_attn_enabled: False + gated_attn_init_std: 0.01 + gated_attn_quant_gate: False + global_ttt_batch_seqs: 32 + global_ttt_chunk_tokens: 32768 + global_ttt_epochs: 1 + global_ttt_grad_clip: 1.0 + global_ttt_lr: 0.001 + global_ttt_momentum: 0.9 + global_ttt_respect_doc_boundaries: True + global_ttt_warmup_chunks: 0 + global_ttt_warmup_start_lr: 0.0 + gptq_calibration_batches: 16 + gptq_reserve_seconds: 0.5 + grad_accum_steps: 1 + grad_clip_norm: 0.3 + is_main_process: True + iterations: 20000 + ln_scale: True + local_rank: 0 + logfile: logs/cond_ppm_seed1337.txt + logit_softcap: 30.0 + loop_end: 5 + loop_start: 3 + lqer_asym_enabled: True + lqer_asym_group: 64 + lqer_enabled: True + lqer_factor_bits: 4 + lqer_rank: 4 + lqer_top_k: 2 + matrix_bits: 6 + matrix_clip_sigmas: 12.85 + matrix_lr: 0.026 + max_wallclock_seconds: 600.0 + min_lr: 0.1 + mlp_clip_sigmas: 11.5 + mlp_mult: 4.0 + model_dim: 512 + model_path: final_model.pt + multi_exit_aux_weight: 0.1 + multi_exit_enabled: False + multi_exit_layers: 4,6,8 + multi_exit_mix_lr: 0.05 + multi_exit_mix_steps: 80 + muon_backend_steps: 5 + muon_momentum: 0.97 + muon_momentum_warmup_start: 0.92 + muon_momentum_warmup_steps: 1500 + muon_row_normalize: True + muon_wd: 0.095 + num_heads: 8 + num_kv_heads: 4 + num_layers: 11 + num_loops: 2 + parallel_final_lane: mean + parallel_start_layer: 5 + phased_ttt_num_phases: 1 + phased_ttt_prefix_docs: 2000 + ppm_byte_conditional_alpha: 25.0 + ppm_byte_conditional_beta: 0.9999 + ppm_byte_conditional_enabled: True + ppm_conf_threshold: 0.9 + ppm_enabled: False + ppm_gate_mode: binary + ppm_lambda_hi: 0.9 + ppm_lambda_lo: 0.05 + ppm_mix_level: byte + ppm_order: 5 + ppm_sigmoid_alpha: 15.0 + ppm_sigmoid_beta: 0.8 + ppm_subset_tokens: 5000000 + ppm_token_conf_aggregate: mean + prequant_ttt_batch_seqs: 32 + prequant_ttt_beta1: 0.9 + prequant_ttt_beta2: 0.999 + prequant_ttt_chunk_tokens: 32768 + prequant_ttt_compile: True + prequant_ttt_enabled: False + prequant_ttt_epochs: 21 + prequant_ttt_fedavg_weights: True + prequant_ttt_grad_clip: 1.0 + prequant_ttt_lr: 0.0005 + prequant_ttt_lr_final: 5e-05 + prequant_ttt_optimizer: adamw + prequant_ttt_weight_decay: 0.0 + qk_gain_init: 5.0 + quantized_model_path: final_model.int6.ptz + rank: 0 + rope_base: 10000.0 + rope_dims: 16 + rope_train_seq_len: 2048 + rope_yarn: False + run_id: cond_ppm_seed1337 + scalar_lr: 0.02 + seed: 1337 + skip_gates_enabled: True + sliding_window_batch_seqs: 16 + sliding_window_enabled: True + smear_gate_enabled: True + sparse_attn_gate_enabled: True + sparse_attn_gate_init_std: 0.0 + sparse_attn_gate_scale: 1.0 + stoch_depth_max: 0.02 + stoch_depth_schedule: linear + temp_cal_enabled: False + temp_cal_lr: 0.1 + temp_cal_steps: 50 + tie_embeddings: True + tied_embed_init_std: 0.005 + tied_embed_lr: 0.03 + tokenizer_path: ./tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model + train_batch_tokens: 786432 + train_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_train_*.bin + train_log_every: 500 + train_seq_len: 2048 + ttt_batch_size: 64 + ttt_beta1: 0.0 + ttt_beta2: 0.999 + ttt_chunk_size: 48 + ttt_enabled: False + ttt_eval_batches: + ttt_eval_seq_len: 2048 + ttt_grad_steps: 1 + ttt_k_lora: True + ttt_lora_lr: 0.0001 + ttt_lora_rank: 96 + ttt_mlp_lora: True + ttt_o_lora: True + ttt_optimizer: adam + ttt_weight_decay: 1.0 + val_batch_tokens: 524288 + val_bytes_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_val_bytes_*.bin + val_doc_fraction: 1.0 + val_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_val_*.bin + val_loss_every: 4000 + vocab_size: 8192 + warmdown_frac: 0.85 + warmup_steps: 20 + world_size: 8 + xsa_last_n: 11 +train_shards: 191 +val_tokens: 47851520 +model_params:35945671 +gptq:reserving 0s, effective=599500ms +warmup_cu_buckets:64,128,192,256 iters_each:3 +warmup_step: 1/20 +warmup_step: 2/20 +warmup_step: 3/20 +warmup_step: 4/20 +warmup_step: 5/20 +warmup_step: 6/20 +warmup_step: 10/20 +warmup_step: 20/20 +loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +loop_warmup_step: 1/20 +loop_warmup_step: 2/20 +loop_warmup_step: 3/20 +loop_warmup_step: 4/20 +loop_warmup_step: 5/20 +loop_warmup_step: 6/20 +loop_warmup_step: 10/20 +loop_warmup_step: 20/20 +0/20000 val_loss: 9.0070 val_bpb: 4.1155 +1/20000 train_loss: 9.0081 train_time: 0.0m tok/s: 11363660 +2/20000 train_loss: 13.0447 train_time: 0.0m tok/s: 11050278 +3/20000 train_loss: 10.2767 train_time: 0.0m tok/s: 9898471 +4/20000 train_loss: 8.7516 train_time: 0.0m tok/s: 9461543 +5/20000 train_loss: 7.9322 train_time: 0.0m tok/s: 9181326 +500/20000 train_loss: 2.7540 train_time: 0.8m tok/s: 8160756 +1000/20000 train_loss: 2.8037 train_time: 1.6m tok/s: 8128962 +1500/20000 train_loss: 2.7439 train_time: 2.4m tok/s: 8123551 +2000/20000 train_loss: 2.5105 train_time: 3.2m tok/s: 8124044 +2500/20000 train_loss: 2.6525 train_time: 4.0m tok/s: 8126867 +3000/20000 train_loss: 2.5891 train_time: 4.8m tok/s: 8129491 +3500/20000 train_loss: 2.6738 train_time: 5.6m tok/s: 8132298 +4000/20000 train_loss: 2.6863 train_time: 6.4m tok/s: 8134392 +4000/20000 val_loss: 2.5100 val_bpb: 1.1469 +4500/20000 train_loss: 2.4782 train_time: 7.2m tok/s: 8135604 +5000/20000 train_loss: 2.4559 train_time: 8.1m tok/s: 8138615 +layer_loop:enabled step:5274 frac:0.850 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +5500/20000 train_loss: 2.2726 train_time: 9.0m tok/s: 8036608 +5984/20000 val_loss: 2.3873 val_bpb: 1.0908 +stopping_early: wallclock_cap train_time: 599632ms step: 5984/20000 +peak memory allocated: 37566 MiB reserved: 42790 MiB +ema:applying EMA weights +diagnostic pre-quantization post-ema val_loss:2.36431212 val_bpb:1.08030167 eval_time:11098ms +Serialized model: 135417533 bytes +Code size (uncompressed): 247422 bytes +Code size (compressed): 42815 bytes +GPTQ:collecting Hessians from calibration data... +GPTQ:collected 67 Hessians + 67 act_stats in 4.1s +Quantized weights: + gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight + gptq (int6)+awqgrpint8+lqer_asym: tok_emb.weight + gptq (int6)+lqer_asym: blocks.mlp.fc.weight + passthrough (float16): blocks.attn.attn_gate_w, blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, parallel_post_lambdas, parallel_resid_lambdas, skip_gates, skip_weights, smear_gate.weight, smear_lambda +Serialized model quantized+brotli: 15757591 bytes +Total submission size quantized+brotli: 15800406 bytes +diagnostic quantized val_loss:2.39181531 val_bpb:1.09286843 eval_time:12202ms +cond_ppm tokens=47851520 bytes=164594398 cond_mix_bpb=0.979547 alpha=25.0 beta=0.9999 jit_seconds=115.7 +quantized_cond_ppm val_loss:2.33544778 val_bpb:0.97954725 +quantized_sliding_window val_loss:2.41030118 val_bpb:1.10141497 eval_time:208140ms diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed314.log b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed314.log new file mode 100644 index 0000000000..4a6e833ed8 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed314.log @@ -0,0 +1,233 @@ +W0501 23:18:42.069000 309465 torch/distributed/run.py:803] +W0501 23:18:42.069000 309465 torch/distributed/run.py:803] ***************************************** +W0501 23:18:42.069000 309465 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +W0501 23:18:42.069000 309465 torch/distributed/run.py:803] ***************************************** +Hyperparameters: + adam_eps: 1e-08 + adam_wd: 0.02 + artifact_dir: + attn_clip_sigmas: 13.0 + attn_out_gate_enabled: False + attn_out_gate_src: proj + awq_lite_bits: 8 + awq_lite_enabled: True + awq_lite_group_size: 64 + awq_lite_group_top_k: 1 + beta1: 0.9 + beta2: 0.95 + caseops_enabled: True + compressor: brotli + datasets_dir: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved + distributed: True + ema_decay: 0.9965 + embed_bits: 6 + embed_clip_sigmas: 15.0 + embed_lr: 0.6 + embed_wd: 0.085 + enable_looping_at: 0.85 + eval_seq_len: 2560 + eval_stride: 128 + fused_ce_enabled: True + gate_window: 12 + gated_attn_enabled: False + gated_attn_init_std: 0.01 + gated_attn_quant_gate: False + global_ttt_batch_seqs: 32 + global_ttt_chunk_tokens: 32768 + global_ttt_epochs: 1 + global_ttt_grad_clip: 1.0 + global_ttt_lr: 0.001 + global_ttt_momentum: 0.9 + global_ttt_respect_doc_boundaries: True + global_ttt_warmup_chunks: 0 + global_ttt_warmup_start_lr: 0.0 + gptq_calibration_batches: 16 + gptq_reserve_seconds: 0.5 + grad_accum_steps: 1 + grad_clip_norm: 0.3 + is_main_process: True + iterations: 20000 + ln_scale: True + local_rank: 0 + logfile: logs/cond_ppm_seed314.txt + logit_softcap: 30.0 + loop_end: 5 + loop_start: 3 + lqer_asym_enabled: True + lqer_asym_group: 64 + lqer_enabled: True + lqer_factor_bits: 4 + lqer_rank: 4 + lqer_top_k: 2 + matrix_bits: 6 + matrix_clip_sigmas: 12.85 + matrix_lr: 0.026 + max_wallclock_seconds: 600.0 + min_lr: 0.1 + mlp_clip_sigmas: 11.5 + mlp_mult: 4.0 + model_dim: 512 + model_path: final_model.pt + multi_exit_aux_weight: 0.1 + multi_exit_enabled: False + multi_exit_layers: 4,6,8 + multi_exit_mix_lr: 0.05 + multi_exit_mix_steps: 80 + muon_backend_steps: 5 + muon_momentum: 0.97 + muon_momentum_warmup_start: 0.92 + muon_momentum_warmup_steps: 1500 + muon_row_normalize: True + muon_wd: 0.095 + num_heads: 8 + num_kv_heads: 4 + num_layers: 11 + num_loops: 2 + parallel_final_lane: mean + parallel_start_layer: 5 + phased_ttt_num_phases: 1 + phased_ttt_prefix_docs: 2000 + ppm_byte_conditional_alpha: 25.0 + ppm_byte_conditional_beta: 0.9999 + ppm_byte_conditional_enabled: True + ppm_conf_threshold: 0.9 + ppm_enabled: False + ppm_gate_mode: binary + ppm_lambda_hi: 0.9 + ppm_lambda_lo: 0.05 + ppm_mix_level: byte + ppm_order: 5 + ppm_sigmoid_alpha: 15.0 + ppm_sigmoid_beta: 0.8 + ppm_subset_tokens: 5000000 + ppm_token_conf_aggregate: mean + prequant_ttt_batch_seqs: 32 + prequant_ttt_beta1: 0.9 + prequant_ttt_beta2: 0.999 + prequant_ttt_chunk_tokens: 32768 + prequant_ttt_compile: True + prequant_ttt_enabled: False + prequant_ttt_epochs: 21 + prequant_ttt_fedavg_weights: True + prequant_ttt_grad_clip: 1.0 + prequant_ttt_lr: 0.0005 + prequant_ttt_lr_final: 5e-05 + prequant_ttt_optimizer: adamw + prequant_ttt_weight_decay: 0.0 + qk_gain_init: 5.0 + quantized_model_path: final_model.int6.ptz + rank: 0 + rope_base: 10000.0 + rope_dims: 16 + rope_train_seq_len: 2048 + rope_yarn: False + run_id: cond_ppm_seed314 + scalar_lr: 0.02 + seed: 314 + skip_gates_enabled: True + sliding_window_batch_seqs: 16 + sliding_window_enabled: True + smear_gate_enabled: True + sparse_attn_gate_enabled: True + sparse_attn_gate_init_std: 0.0 + sparse_attn_gate_scale: 1.0 + stoch_depth_max: 0.02 + stoch_depth_schedule: linear + temp_cal_enabled: False + temp_cal_lr: 0.1 + temp_cal_steps: 50 + tie_embeddings: True + tied_embed_init_std: 0.005 + tied_embed_lr: 0.03 + tokenizer_path: ./tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model + train_batch_tokens: 786432 + train_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_train_*.bin + train_log_every: 500 + train_seq_len: 2048 + ttt_batch_size: 64 + ttt_beta1: 0.0 + ttt_beta2: 0.999 + ttt_chunk_size: 48 + ttt_enabled: False + ttt_eval_batches: + ttt_eval_seq_len: 2048 + ttt_grad_steps: 1 + ttt_k_lora: True + ttt_lora_lr: 0.0001 + ttt_lora_rank: 96 + ttt_mlp_lora: True + ttt_o_lora: True + ttt_optimizer: adam + ttt_weight_decay: 1.0 + val_batch_tokens: 524288 + val_bytes_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_val_bytes_*.bin + val_doc_fraction: 1.0 + val_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_val_*.bin + val_loss_every: 4000 + vocab_size: 8192 + warmdown_frac: 0.85 + warmup_steps: 20 + world_size: 8 + xsa_last_n: 11 +train_shards: 191 +val_tokens: 47851520 +model_params:35945671 +gptq:reserving 0s, effective=599500ms +warmup_cu_buckets:64,128,192,256 iters_each:3 +warmup_step: 1/20 +warmup_step: 2/20 +warmup_step: 3/20 +warmup_step: 4/20 +warmup_step: 5/20 +warmup_step: 6/20 +warmup_step: 10/20 +warmup_step: 20/20 +loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +loop_warmup_step: 1/20 +loop_warmup_step: 2/20 +loop_warmup_step: 3/20 +loop_warmup_step: 4/20 +loop_warmup_step: 5/20 +loop_warmup_step: 6/20 +loop_warmup_step: 10/20 +loop_warmup_step: 20/20 +0/20000 val_loss: 8.9980 val_bpb: 4.1114 +1/20000 train_loss: 8.9988 train_time: 0.0m tok/s: 11400849 +2/20000 train_loss: 12.8933 train_time: 0.0m tok/s: 11017358 +3/20000 train_loss: 10.2639 train_time: 0.0m tok/s: 9812210 +4/20000 train_loss: 8.7044 train_time: 0.0m tok/s: 9398051 +5/20000 train_loss: 7.9156 train_time: 0.0m tok/s: 9139944 +500/20000 train_loss: 2.7515 train_time: 0.8m tok/s: 8157629 +1000/20000 train_loss: 2.8009 train_time: 1.6m tok/s: 8129699 +1500/20000 train_loss: 2.7436 train_time: 2.4m tok/s: 8124870 +2000/20000 train_loss: 2.5065 train_time: 3.2m tok/s: 8124699 +2500/20000 train_loss: 2.5183 train_time: 4.0m tok/s: 8127439 +3000/20000 train_loss: 2.5295 train_time: 4.8m tok/s: 8130078 +3500/20000 train_loss: 2.6747 train_time: 5.6m tok/s: 8132414 +4000/20000 train_loss: 2.6917 train_time: 6.4m tok/s: 8134671 +4000/20000 val_loss: 2.5144 val_bpb: 1.1489 +4500/20000 train_loss: 2.5668 train_time: 7.2m tok/s: 8137276 +5000/20000 train_loss: 2.4599 train_time: 8.1m tok/s: 8139150 +layer_loop:enabled step:5274 frac:0.850 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +5500/20000 train_loss: 2.2687 train_time: 9.0m tok/s: 8035077 +5982/20000 val_loss: 2.3862 val_bpb: 1.0903 +stopping_early: wallclock_cap train_time: 599560ms step: 5982/20000 +peak memory allocated: 37566 MiB reserved: 42790 MiB +ema:applying EMA weights +diagnostic pre-quantization post-ema val_loss:2.36324243 val_bpb:1.07981291 eval_time:11168ms +Serialized model: 135417533 bytes +Code size (uncompressed): 247422 bytes +Code size (compressed): 42815 bytes +GPTQ:collecting Hessians from calibration data... +GPTQ:collected 67 Hessians + 67 act_stats in 4.1s +Quantized weights: + gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight + gptq (int6)+awqgrpint8+lqer_asym: tok_emb.weight + gptq (int6)+lqer_asym: blocks.mlp.fc.weight + passthrough (float16): blocks.attn.attn_gate_w, blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, parallel_post_lambdas, parallel_resid_lambdas, skip_gates, skip_weights, smear_gate.weight, smear_lambda +Serialized model quantized+brotli: 15758484 bytes +Total submission size quantized+brotli: 15801299 bytes +diagnostic quantized val_loss:2.39170326 val_bpb:1.09281723 eval_time:12134ms +cond_ppm tokens=47851520 bytes=164594398 cond_mix_bpb=0.979629 alpha=25.0 beta=0.9999 jit_seconds=112.9 +quantized_cond_ppm val_loss:2.33564234 val_bpb:0.97962885 +quantized_sliding_window val_loss:2.41032887 val_bpb:1.10142762 eval_time:205380ms diff --git a/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed42.log b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed42.log new file mode 100644 index 0000000000..9e88f46a37 --- /dev/null +++ b/records/track_10min_16mb/2026-05-01_LockInByteMixer_0.979556/train_seed42.log @@ -0,0 +1,233 @@ +W0501 22:42:03.077000 275866 torch/distributed/run.py:803] +W0501 22:42:03.077000 275866 torch/distributed/run.py:803] ***************************************** +W0501 22:42:03.077000 275866 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. +W0501 22:42:03.077000 275866 torch/distributed/run.py:803] ***************************************** +Hyperparameters: + adam_eps: 1e-08 + adam_wd: 0.02 + artifact_dir: + attn_clip_sigmas: 13.0 + attn_out_gate_enabled: False + attn_out_gate_src: proj + awq_lite_bits: 8 + awq_lite_enabled: True + awq_lite_group_size: 64 + awq_lite_group_top_k: 1 + beta1: 0.9 + beta2: 0.95 + caseops_enabled: True + compressor: brotli + datasets_dir: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved + distributed: True + ema_decay: 0.9965 + embed_bits: 6 + embed_clip_sigmas: 15.0 + embed_lr: 0.6 + embed_wd: 0.085 + enable_looping_at: 0.85 + eval_seq_len: 2560 + eval_stride: 128 + fused_ce_enabled: True + gate_window: 12 + gated_attn_enabled: False + gated_attn_init_std: 0.01 + gated_attn_quant_gate: False + global_ttt_batch_seqs: 32 + global_ttt_chunk_tokens: 32768 + global_ttt_epochs: 1 + global_ttt_grad_clip: 1.0 + global_ttt_lr: 0.001 + global_ttt_momentum: 0.9 + global_ttt_respect_doc_boundaries: True + global_ttt_warmup_chunks: 0 + global_ttt_warmup_start_lr: 0.0 + gptq_calibration_batches: 16 + gptq_reserve_seconds: 0.5 + grad_accum_steps: 1 + grad_clip_norm: 0.3 + is_main_process: True + iterations: 20000 + ln_scale: True + local_rank: 0 + logfile: logs/cond_ppm_seed42.txt + logit_softcap: 30.0 + loop_end: 5 + loop_start: 3 + lqer_asym_enabled: True + lqer_asym_group: 64 + lqer_enabled: True + lqer_factor_bits: 4 + lqer_rank: 4 + lqer_top_k: 2 + matrix_bits: 6 + matrix_clip_sigmas: 12.85 + matrix_lr: 0.026 + max_wallclock_seconds: 600.0 + min_lr: 0.1 + mlp_clip_sigmas: 11.5 + mlp_mult: 4.0 + model_dim: 512 + model_path: final_model.pt + multi_exit_aux_weight: 0.1 + multi_exit_enabled: False + multi_exit_layers: 4,6,8 + multi_exit_mix_lr: 0.05 + multi_exit_mix_steps: 80 + muon_backend_steps: 5 + muon_momentum: 0.97 + muon_momentum_warmup_start: 0.92 + muon_momentum_warmup_steps: 1500 + muon_row_normalize: True + muon_wd: 0.095 + num_heads: 8 + num_kv_heads: 4 + num_layers: 11 + num_loops: 2 + parallel_final_lane: mean + parallel_start_layer: 5 + phased_ttt_num_phases: 1 + phased_ttt_prefix_docs: 2000 + ppm_byte_conditional_alpha: 25.0 + ppm_byte_conditional_beta: 0.9999 + ppm_byte_conditional_enabled: True + ppm_conf_threshold: 0.9 + ppm_enabled: False + ppm_gate_mode: binary + ppm_lambda_hi: 0.9 + ppm_lambda_lo: 0.05 + ppm_mix_level: byte + ppm_order: 5 + ppm_sigmoid_alpha: 15.0 + ppm_sigmoid_beta: 0.8 + ppm_subset_tokens: 5000000 + ppm_token_conf_aggregate: mean + prequant_ttt_batch_seqs: 32 + prequant_ttt_beta1: 0.9 + prequant_ttt_beta2: 0.999 + prequant_ttt_chunk_tokens: 32768 + prequant_ttt_compile: True + prequant_ttt_enabled: False + prequant_ttt_epochs: 21 + prequant_ttt_fedavg_weights: True + prequant_ttt_grad_clip: 1.0 + prequant_ttt_lr: 0.0005 + prequant_ttt_lr_final: 5e-05 + prequant_ttt_optimizer: adamw + prequant_ttt_weight_decay: 0.0 + qk_gain_init: 5.0 + quantized_model_path: final_model.int6.ptz + rank: 0 + rope_base: 10000.0 + rope_dims: 16 + rope_train_seq_len: 2048 + rope_yarn: False + run_id: cond_ppm_seed42 + scalar_lr: 0.02 + seed: 42 + skip_gates_enabled: True + sliding_window_batch_seqs: 16 + sliding_window_enabled: True + smear_gate_enabled: True + sparse_attn_gate_enabled: True + sparse_attn_gate_init_std: 0.0 + sparse_attn_gate_scale: 1.0 + stoch_depth_max: 0.02 + stoch_depth_schedule: linear + temp_cal_enabled: False + temp_cal_lr: 0.1 + temp_cal_steps: 50 + tie_embeddings: True + tied_embed_init_std: 0.005 + tied_embed_lr: 0.03 + tokenizer_path: ./tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model + train_batch_tokens: 786432 + train_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_train_*.bin + train_log_every: 500 + train_seq_len: 2048 + ttt_batch_size: 64 + ttt_beta1: 0.0 + ttt_beta2: 0.999 + ttt_chunk_size: 48 + ttt_enabled: False + ttt_eval_batches: + ttt_eval_seq_len: 2048 + ttt_grad_steps: 1 + ttt_k_lora: True + ttt_lora_lr: 0.0001 + ttt_lora_rank: 96 + ttt_mlp_lora: True + ttt_o_lora: True + ttt_optimizer: adam + ttt_weight_decay: 1.0 + val_batch_tokens: 524288 + val_bytes_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_val_bytes_*.bin + val_doc_fraction: 1.0 + val_files: ./data/datasets/fineweb10B_sp8192_caseops/datasets/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved/fineweb_val_*.bin + val_loss_every: 4000 + vocab_size: 8192 + warmdown_frac: 0.85 + warmup_steps: 20 + world_size: 8 + xsa_last_n: 11 +train_shards: 191 +val_tokens: 47851520 +model_params:35945671 +gptq:reserving 0s, effective=599500ms +warmup_cu_buckets:64,128,192,256 iters_each:3 +warmup_step: 1/20 +warmup_step: 2/20 +warmup_step: 3/20 +warmup_step: 4/20 +warmup_step: 5/20 +warmup_step: 6/20 +warmup_step: 10/20 +warmup_step: 20/20 +loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +loop_warmup_step: 1/20 +loop_warmup_step: 2/20 +loop_warmup_step: 3/20 +loop_warmup_step: 4/20 +loop_warmup_step: 5/20 +loop_warmup_step: 6/20 +loop_warmup_step: 10/20 +loop_warmup_step: 20/20 +0/20000 val_loss: 9.0076 val_bpb: 4.1158 +1/20000 train_loss: 9.0087 train_time: 0.0m tok/s: 11525486 +2/20000 train_loss: 12.8897 train_time: 0.0m tok/s: 11043353 +3/20000 train_loss: 10.2849 train_time: 0.0m tok/s: 9920688 +4/20000 train_loss: 8.7218 train_time: 0.0m tok/s: 9458182 +5/20000 train_loss: 7.9252 train_time: 0.0m tok/s: 9179229 +500/20000 train_loss: 2.7556 train_time: 0.8m tok/s: 8154771 +1000/20000 train_loss: 2.8086 train_time: 1.6m tok/s: 8128276 +1500/20000 train_loss: 2.8166 train_time: 2.4m tok/s: 8123940 +2000/20000 train_loss: 2.5104 train_time: 3.2m tok/s: 8125663 +2500/20000 train_loss: 2.5193 train_time: 4.0m tok/s: 8128319 +3000/20000 train_loss: 2.6998 train_time: 4.8m tok/s: 8129834 +3500/20000 train_loss: 2.6979 train_time: 5.6m tok/s: 8130058 +4000/20000 train_loss: 2.6810 train_time: 6.4m tok/s: 8132000 +4000/20000 val_loss: 2.5108 val_bpb: 1.1472 +4500/20000 train_loss: 2.4820 train_time: 7.3m tok/s: 8135030 +5000/20000 train_loss: 2.4552 train_time: 8.1m tok/s: 8136708 +layer_loop:enabled step:5273 frac:0.850 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10] +5500/20000 train_loss: 2.2723 train_time: 9.0m tok/s: 8033562 +5982/20000 val_loss: 2.3874 val_bpb: 1.0908 +stopping_early: wallclock_cap train_time: 599652ms step: 5982/20000 +peak memory allocated: 37566 MiB reserved: 42790 MiB +ema:applying EMA weights +diagnostic pre-quantization post-ema val_loss:2.36343036 val_bpb:1.07989878 eval_time:10852ms +Serialized model: 135417533 bytes +Code size (uncompressed): 247422 bytes +Code size (compressed): 42815 bytes +GPTQ:collecting Hessians from calibration data... +GPTQ:collected 67 Hessians + 67 act_stats in 4.1s +Quantized weights: + gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight + gptq (int6)+awqgrpint8+lqer_asym: tok_emb.weight + gptq (int6)+lqer_asym: blocks.mlp.fc.weight + passthrough (float16): blocks.attn.attn_gate_w, blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, parallel_post_lambdas, parallel_resid_lambdas, skip_gates, skip_weights, smear_gate.weight, smear_lambda +Serialized model quantized+brotli: 15758573 bytes +Total submission size quantized+brotli: 15801388 bytes +diagnostic quantized val_loss:2.39123900 val_bpb:1.09260510 eval_time:12055ms +cond_ppm tokens=47851520 bytes=164594398 cond_mix_bpb=0.979491 alpha=25.0 beta=0.9999 jit_seconds=114.6 +quantized_cond_ppm val_loss:2.33531315 val_bpb:0.97949078 +quantized_sliding_window val_loss:2.40982833 val_bpb:1.10119889 eval_time:206743ms