This project was a wide, systematic search: roughly 150 experiments across more than 60 families. The repository ships the clean, canonical implementation of each idea that mattered; this document records the breadth of what was tried, what worked, and what did not, so the result is legible as a research process and not just a final number. Honest negative results are included on purpose.
- Pure convolutional (ResNet-18, EfficientNet-B0, EfficientNetV2-S/M): solid but plateau early.
- 3D convolutional (R(2+1)D-18, MC3-18, S3D): adapted from their native 16 frames to 4 by trilinear interpolation; competitive but heavy for 52k clips.
- Conv-attention hybrids (EfficientFormer-L/V2, MaxViT, CoAtNet, NextViT-S): the best from-scratch family, topping out near val 0.44. Their early convolutional stages wire in locality and translation invariance.
- Pure Vision Transformer from scratch: a clear negative result. With no convolutional prior and only ~52k clips, it never learns locality and collapses to ~6% (the majority-class baseline). This is what motivated either convolutional hybrids or self-supervised pretraining.
- An optical-flow channel (Farneback) was scoped but not pursued once SSL proved far more effective.
A full sweep on top of a fixed encoder: mean pool, temporal Conv1d, LSTM, BiLSTM, attention transformer ([CLS] + sinusoidal positions), Perceiver (16 latent queries), an attentive probe (1 query), and a hybrid (Perceiver + BiLSTM). No single head dominates everywhere, which is exactly why the final ensemble keeps all four head types per backbone, and why the per-class stacker pays off.
Driven by the error diagnosis (92% of errors are fine visual confusions), the effort moved from heads to representations:
- MAE-2D on ViT-B/16 (
m4), pretrained on the challenge frames; at finetune, the last six blocks gain a zero-initialised temporal attention so the pretrained spatial weights are untouched at step 0. - VideoMAE tubelet MAE on ViT-S/16 (
m5), and a ViT-B variant (m5b) seeded from our own iBOT initialisation. - The SSL cascade: generations ssl500 (iBOT) -> ssl600 -> ssl700 -> ssl800 -> ssl1000, each reloading the encoder but resetting optimiser/scheduler/decoder. Reconstruction loss fell monotonically and downstream accuracy rose through ssl800. The regression at ssl1000 looks like an optimisation instability, not a ceiling: pseudo-label distillation recovered and exceeded it (val 0.6061). More SSL epochs is the single most promising untried lever.
- Pair-Direction Head (order prediction, weight 0.3) and Multi-Clip Consistency (symmetric KL between two augmentations, weight 0.2). Together they lifted the MAE ViT-B single model to val 0.5122.
Three iterations (v577 -> v592 -> v597) of: aggregate -> hard-label all test clips -> retrain on real + pseudo. The BiLSTM head benefited most (0.5500 -> 0.6061). The first solo member above 0.60 came from this loop.
- Per-class log-space bagged stacker (
ensemble.stacker): the canonical, leakage-free aggregator, with nested cross-validation for an honest estimate. This is the one that produced 0.6070. - Alternatives that it beat (
ensemble.calibration): per-member temperature scaling, an MLP meta-stacker over concatenated logits, a linear/logistic meta-stacker. They have far more free parameters and no convex/product-of-experts inductive bias, and underperformed in the small-validation regime. - Member-set engineering: deduplication (best head per backbone), top-K sweeps, two-stage family grouping. Forcing diversity beyond a point hurt.
- Sidecar / twin scheme (
ensemble.sidecar): re-injects the 6,745 validation clips into the submission models without leaking them into weight estimation. - The final pool is 95 distinct checkpoints, all closed-world legal (from-scratch + our own MAE/VideoMAE).
A systematic sweep over four pretraining regimes (finetune.track_b_*):
- ImageNet (ConvNeXt-T/S, EfficientNetV2-S + LSTM): ~0.49, the video signal is missing.
- Kinetics-400 (MViTv2-S, TimeSformer, UniFormer-S, Swin3D): 0.51 to 0.54, wrong class distribution.
- SSv2 supervised (VideoMAE-B/L): 0.57 to 0.59.
- SSv2 self-supervised then finetuned (V-JEPA-2-L, InternVideo2-1B): 0.68 to 0.72; InternVideo2-1B at 0.7214 is the best single model.
Key findings:
- Domain dominates architecture: Kinetics -> SSv2 is worth ~10 points, more than any architecture change.
- One epoch on the full corpus beats several epochs on a subset: on a subset the training loss collapses (memorisation) and the score drops. Split learning rate (encoder ~1e-5, fresh head ~30x higher), no frozen layers, no LoRA.
- Negative results: a frozen-backbone linear probe on V-JEPA-2 stalls at ~0.37 (vs ~0.70 full finetune); non-action extractors (DINOv2 + MLP 0.39, CoTracker3 + MLP 0.15) are far off; forcing diversity in the final stacker (weight floors, top-K constraints, an MLP aggregator) always degraded the Kaggle score, so it concentrates 99.4% of weight on the two SSv2-self-supervised members. A RIFE frame-interpolation idea was prototyped but not adopted.
- Clip-coherent augmentation with the 18<->19 label-aware flip (drives mirror confusions to 0%).
- Self-supervision on the challenge data only (Track A).
- Pseudo-label distillation on test images (Track A).
- Leakage-free validation re-injection (sidecar).
- Test-time augmentation (horizontal flip with class swap, 5-crop) cached once and reused by the stacker.
Push the SSL cascade much further (ssl1500 to ssl2000, better scheduled), which the trend suggests would lift validation past 0.65; try soft pseudo-labels (full distributions) to keep teacher uncertainty; and run a fourth distillation round seeded from the 0.6070 submission.