Skip to content

methodology.md

Latest commit

 

History

History
114 lines (87 loc) · 7.37 KB

methodology.md

File metadata and controls

114 lines (87 loc) · 7.37 KB
Raw

Methodology

A technical companion to the README. It documents the full pipeline for both tracks and the reasoning behind each choice.

1. Task and data

A 33-class subset of Something-Something-V2 (class 27 empty, 32 populated): 44,993 train / 6,745 val / 6,912 test clips; each clip is four 224x224 JPEG frames from the first 40% of the source video. Metric: top-1 accuracy. Frames are read directly from frames.zip (the NFS home has a ~300k inode quota that a flat extraction would exceed); labels come from the Kaggle CSVs, not the zip folder names (whatnext.data.zip_dataset).

2. The diagnostic that set the direction

whatnext.analysis.error_decomposition splits a model's validation errors into four buckets. On the best from-scratch encoder (EfficientFormer-L, val 0.4411), 92.1% of the 3,770 errors are fine visual confusions, 3.9% temporal inversions, 4.0% pretending-vs-real, and 0.0% left/right mirror. Conclusion: the limiting factor is representation quality, not the temporal head or the arrow of time. Everything in Track A follows from this.

3. Common training recipe (Track A)

AdamW, base LR 1.5e-4, weight decay 0.05, cosine schedule with warmup, label smoothing 0.15, bf16/fp16. EMA of weights (decay 0.9999) every step, SWA from epoch 40. Augmentation is clip-coherent (one set of random parameters per clip, applied to all four frames): horizontal flip (p 0.5) with the 18<->19 label swap, ColorJitter (p 0.5, strength 0.15), Gaussian blur (p 0.3). No RandomResizedCrop (cropping erases near-edge motion cues on four frames), no MixUp (whatnext.data.clip_transforms).

4. Track A encoders

  • From scratch (whatnext.models.encoders_2d, encoders_3d). Pure conv (ResNet, EfficientNet/V2), 3D conv (R(2+1)D-18, MC3, S3D, lifted from their native 16 frames to 4 by trilinear interpolation), and conv-attention hybrids (EfficientFormer-L/V2, MaxViT, CoAtNet, NextViT). The hybrids converge (~0.44 val) where a pure ViT collapses to the majority class (~6%): their convolutional stages wire in locality and translation invariance that 52k clips cannot teach a plain transformer.
  • Self-supervised on challenge data (the lever).
    • m4 (whatnext.models.mae2d_vitb): 2D MAE on ViT-B/16 (embed 768, depth 12, heads 12; decoder 384x4; patch 16; mask 0.75; normalised-pixel loss). Finetune wraps the last 6 ViT blocks with a zero-initialised temporal ("SpaceTime") attention so step 0 is an exact identity over the pretrained spatial weights.
    • m5 (whatnext.models.videomae_vits): VideoMAE tubelet MAE on ViT-S/16 (Conv3d tubelet 2x16x16 -> 392 tubes/clip; decoder 256x4; tube masking 0.75). m5b is the ViT-B variant seeded from our own iBOT checkpoint (converted via whatnext.models.convert_videomae_ckpt).

5. Track A temporal heads (whatnext.models.mae2d_heads, encoders_2d)

Mean pool, temporal Conv1d, (Bi)LSTM, attention transformer ([CLS] + sinusoidal positions + 2 self-attention layers), Perceiver (16 learned queries, cross-attention then a self-attention tower, mean over queries), an attentive probe (1 query), and a hybrid (Perceiver concatenated with BiLSTM). Finetune uses layer-wise LR decay (0.65 for ViT-B, 0.70 for ViT-S) and progressive drop-path.

6. Auxiliary self-supervision at finetune

  • Pair-Direction Head (weight 0.3): a small head reads ordered frame pairs and predicts temporal order, pushing the encoder to use the arrow of time.
  • Multi-Clip Consistency (weight 0.2): symmetric KL between two augmented views of the same clip.

7. SSL cascade (whatnext.ssl.pretrain_videomae)

A finished cosine cycle is essentially zero LR, so naively continuing does nothing. Each generation instead reloads the encoder but resets the optimiser, scheduler and decoder, opening a fresh cosine cycle. Reconstruction loss falls monotonically (0.30 -> 0.16 across ssl500 iBOT -> ssl1000) and validation accuracy keeps climbing through ssl800, which is why the most actionable improvement is simply more (better-scheduled) SSL. In code this is the --init-encoder (new generation) versus --resume (continue the same run) split.

8. Pseudo-label distillation (whatnext.ensemble.pseudo_label)

The closed-world rule forbids the test labels but not the test images. The best aggregator argmaxes a hard label for every test clip (no confidence threshold); the next model trains on the plain union of real and pseudo samples (ordinary cross-entropy, same augmentation, plus the direction aux). Three iterations; the BiLSTM head gains the most (0.5500 -> 0.6061 at ssl1000). Consumed by whatnext.finetune.track_a_videomae --pseudo-test-csv.

9. Per-class log-space bagged stacker (whatnext.ensemble.stacker)

Each member contributes cached softmax matrices P (val) and Q (test). The stacker learns per-class weights parameterised by a softmax over members, combined in log-space:

W[i,c] = exp(R[i,c]) / sum_j exp(R[j,c])
logit[n,c] = sum_i W[i,c] * log P[i,n,c] + b[c]

R and b are fit by AdamW (1500 steps) with a Tikhonov pull toward a flat prior (lambda 1e-4), bagged over 15 folds. The generalisation estimate uses nested cross-validation: each fold fits the weights on a train split, early-stops on a probe split, and reports top-1 on a separate held-out split (never used for fitting or selection), which removes the optimistic bias of reporting the best-step score on the selection set. The final submission stacks 95 checkpoints; on the clean members shipped here the nested-CV held-out top-1 is 0.6163 (Kaggle 0.6070). Baselines that this design beat (per-member temperature scaling, an MLP meta-stacker, a linear meta-stacker) live in whatnext.ensemble.calibration.

10. Sidecar: leakage-free validation re-injection (whatnext.ensemble.sidecar)

Once the weights are fixed, the 6,745 validation clips are still useful as training data. For each architecture, a sidecar model trained on train-only provides the out-of-sample validation probabilities that calibrate the stacker weights; a twin trained on train+val (~52k clips) provides the test probabilities. The fixed weights are applied to the twin. No weight is ever fit on a model that has seen the validation set, so this adds 15% more training data to the submission models with no leakage.

11. Track B: foundation models (whatnext.finetune.track_b_*)

Full finetune (no frozen layers), the SSv2 174-class head replaced by a fresh 33-class linear. Split learning rate (encoder ~1e-5, head ~30x higher), AdamW weight decay 0.10, grad-accumulation x8 at batch 1, label smoothing 0.10, bf16, the same clip-coherent 18<->19 augmentation, no RandomResizedCrop, no LoRA. The four frames are lifted to the model's native length (InternVideo2: 4 -> 8 by repeat; V-JEPA: 4 -> 16 by trilinear, at 256 px). The decisive observation: one epoch on the full corpus beats several epochs on a subset, where the collapsing training loss signals memorisation. Best single model: InternVideo2-1B at 0.7214 val; the stacker concentrates 99.4% of its weight on the two SSv2-self-supervised members for the 0.7517 submission.

12. Inference (whatnext.inference)

Per-member softmax probabilities are cached for validation and test with test-time augmentation: horizontal flip (with the 18<->19 probability swap) and 5-crop, averaged. These caches are the stacker's inputs. The submission writer emits the Kaggle CSV (video_name,predicted_class) with the one known fallback row.