Skip to content

Add VLA v2 execution notebook and v2_experiments benchmark harness#13

Merged
Vishal-sys-code merged 1 commit into
mainfrom
codex/implement-improvements-for-v2-in-notebook
May 20, 2026
Merged

Add VLA v2 execution notebook and v2_experiments benchmark harness#13
Vishal-sys-code merged 1 commit into
mainfrom
codex/implement-improvements-for-v2-in-notebook

Conversation

@Vishal-sys-code

Copy link
Copy Markdown
Owner

Motivation

  • Provide a reproducible execution harness for VLA v2 that targets the current fast stack (VLASequential, VLAParallel, and VLATriton when available) and saves paper-ready artifacts.
  • Consolidate benchmarking, small-model ablations, and a tiny LM proxy evaluation into a single script to match VLAv3-era references and make comparisons consistent.
  • Ensure results are portable and discoverable by emitting CSV/JSON/PNG/PDF artifacts plus a manifest describing the run.

Description

  • Add notebooks/08_VLA_v2_Execution_Notebook.ipynb which demonstrates creating a V2Config, running run_full_v2_suite, and inspecting generated artifacts.
  • Add scripts/v2_experiments.py that defines V2Config, a GatedNormalizedSM ablation module, helpers (set_seed, make_vla_backend), benchmarks (benchmark_streaming, run_mqar_capacity_proxy), evaluation (run_tiny_lm_eval, run_sm_ablation), plotting (save_plot), and the orchestration function run_full_v2_suite.
  • run_full_v2_suite conditionally includes the triton backend if HAS_TRITON is true and writes mqar_capacity_multiseed.csv, streaming_benchmarks.csv, lm_eval_tiny.json, sm_ablation_metrics.json, plot files (.png/.pdf), and manifest.json to the configured out_dir.

Testing

  • No CI tests were added as part of this change.
  • Performed local automated smoke runs by importing scripts.v2_experiments, instantiating V2Config, and executing run_sm_ablation and run_tiny_lm_eval which completed successfully on the test environment.
  • Plot generation is guarded by a matplotlib import check so plotting is skipped cleanly when the plotting backend is not available.

Codex Task

@vercel

vercel Bot commented May 20, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
variational-linear-attention Ready Ready Preview, Comment May 20, 2026 6:34am

@Vishal-sys-code Vishal-sys-code merged commit b803e47 into main May 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant