VLAFactory

A unified, minimal framework that ties together the three stages of improving Vision-Language-Action (VLA) models:

┌─────────────┐   ┌──────────────┐   ┌────────────────────┐
│  DataForge  │──▶│ VLM Training │──▶│   VLA Training     │
│  (Stage 1)  │   │  (Stage 2)   │   │    (Stage 3)       │
│  recipes +  │   │ LLaMA-Factory│   │ StarVLA / AutoVLA  │
│  operators  │   │              │   │ / SimLingo         │
└─────────────┘   └──────────────┘   └────────────────────┘
      │                  │                      │
      ▼                  ▼                      ▼
ShareGPT JSON      HF checkpoint           VLA checkpoint

VLAFactory wraps LLaMA-Factory (Stage 2) and several VLA codebases (StarVLA, AutoVLA, SimLingo) behind a thin adapter layer. Its job is to glue them together with a small, opinionated layer that lets you iterate on the data mix as quickly as you can iterate on the training config. It does not vendor or pin those training frameworks — it shells out to them (see Third-Party Strategy).

Why

Most teams that work on VLAs end up with two scattered halves:

1. Data scripts — proximity-based selection, VLM relabeling, scene-graph extraction, mixing, deduplication, format conversion. Each lives in a separate repo with a separate config system.
2. Training codebases — LLaMA-Factory for VLM fine-tuning, plus one or two VLA repos for the action head, each with their own configs, datasets, and checkpoint formats.

VLAFactory unifies the data side with a single typed operator abstraction and YAML recipes, uses ShareGPT JSON as a shared currency between data and VLM training, and keeps the VLA training repos behind a thin VLABackend interface. No external artifact store, no DAG engine — just typed operators writing Apache Arrow shards into a working directory, and a fingerprint-based runner that skips stages whose outputs already exist.

Design Principles

Read these before refactoring — they are decisions, not preferences.

1. A tiny operator interface. Every DataForge operator subclasses one of two thin bases — ArrowOperator (process a batch as an Arrow table) or DictOperator (process rows as dicts, good for async I/O) — and implements one process_batch method. There is no per-category taxonomy (no Selector, Transform, Relabeler, …); whether an operator reads a JSONL, a parquet, or images is its own detail.
2. Filesystem as the source of truth. Caching is "does the stage's Arrow shard exist and does its fingerprint still match?". Multi-run execution falls out for free: run the recipe again and unchanged stages are skipped. No artifact manifest, no provenance tracker.
3. Wrapping existing team code is a small class. Take a script you already have (an extract_features.py, a train_estimator.py, a relabeler) and expose it as an operator by writing one class with a process_batch method that imports your code — plus one line to register it.
4. VLAFactory does NOT own environments. LLaMA-Factory, StarVLA, AutoVLA, and SimLingo each have their own install docs — we follow them verbatim. When the CLI invokes a stage, it assumes the right env is active, or it shells out via conda run -n <env> ....
5. ShareGPT JSON is the dataset currency. It's what LLaMA-Factory natively accepts, so Stage 1 → Stage 2 is a bridge function, not a converter.

Repository Layout

VLAFactory/
├── configs/                            # Example YAML configs
│   ├── data/                           # DataForge recipes (+ .yaml.j2 templates)
│   ├── vlm/                            # VLM training configs
│   ├── vla/                            # VLA training configs
│   └── pipelines/                      # End-to-end pipeline configs
│
├── vlafactory/                         # The Python package
│   ├── dataforge/                      # ===== STAGE 1 =====
│   │   ├── operator.py                 # ArrowOperator / DictOperator base classes
│   │   ├── registry.py                 # @register_operator decorator
│   │   ├── schema.py                   # Typed Schema / Field
│   │   ├── pipeline/                   # loader (YAML/Jinja2 → Pipeline), model, compiler
│   │   ├── execution/engine.py         # ExecutionEngine — fingerprint caching, --incre
│   │   ├── io/arrow_store.py           # Arrow IPC shard read/write + meta.json
│   │   ├── resources/                  # file_store, scene_models, vlm_client, clip_model, …
│   │   ├── modules/                    # scene (RAM++/GroundingDINO/SAM/Depth), semantic, vlm
│   │   └── operators/                  # All operators, flat (auto-imported):
│   │       ├── sources.py              # arrow_source, parquet_source, jsonl_source
│   │       ├── laion_source.py         # laion_source
│   │       ├── filters.py              # expression_filter, dedup, subsample, row/null_filter, hash_subset
│   │       ├── transforms.py           # select_columns, rename_columns, json_expand, schema_assert
│   │       ├── mixers.py               # concat_tables, join_columns, shuffle
│   │       ├── sinks.py                # jsonl_sink, parquet_sink, sharegpt_sink, parquet_merge
│   │       ├── download.py             # download
│   │       ├── clip_embed.py           # clip_embed              (GPU)
│   │       ├── classifier.py           # classifier_train, classifier_score
│   │       ├── scene_extract.py        # scene_extract           (GPU: detection/seg/depth)
│   │       ├── semantic_extract.py     # semantic_extract
│   │       ├── vlm_annotate.py         # scene/semantic/two_call/robotic_annotate (VLM)
│   │       └── vlm_relabeler.py        # vlm_relabeler
│   │
│   ├── vlm_training/                   # ===== STAGE 2 =====
│   │   ├── llamafactory_adapter.py     # Generate LLaMA-Factory YAML, invoke CLI
│   │   └── dataset_bridge.py           # Register ShareGPT JSON in LLaMA-Factory
│   │
│   ├── vla_training/                   # ===== STAGE 3 =====
│   │   ├── base.py                     # VLABackend interface + backend registry
│   │   ├── checkpoint_bridge.py        # LoRA merging + format normalization
│   │   ├── starvla.py                  # Thin adapter → third_party/StarVLA/
│   │   ├── rlinf_starvla.py            # Thin adapter → RLinf (Hydra GRPO)
│   │   ├── autovla/                    # Vendored (static upstream)
│   │   └── simlingo/                   # Thin shell-out adapter → external SimLingo checkout
│   │
│   ├── eval/                           # Per-stage evaluators (data / vlm / driving / vla / vlm-bench)
│   └── cli.py                          # `vlafactory data|vlm|vla|pipeline|eval|resource|setup-models`
│
├── third_party/                        # Integrations (see Third-Party Strategy)
│   └── VLMEvalKit/                     # Registered git submodule (the eval fork)
│
├── docs/
│   ├── architecture-design.md          # Full design doc
│   ├── running_the_pipeline.md         # Operator's guide: whole pipeline + each part
│   ├── EMNLP_DEMO_RUNBOOK.md           # Copy-pasteable CPU tour + GPU pipeline reference
│   ├── simlingo_pipeline.md            # SimLingo training + eval
│   └── vlmeval_benchmark.md            # VLMEvalKit benchmark sweep
│
├── examples/                           # candidates.jsonl, eval samples, colab, toy recipes
└── tests/

Installation

VLAFactory itself is a tiny pure-Python package (no torch in the core):

git clone https://github.com/cxcscmu/VLAFactory.git
cd VLAFactory
pip install -e .        # core: pyarrow / numpy / scikit-learn / rich / pyyaml / datasets

This gives you the vlafactory CLI and lets you write / run recipes. The heavier operators and evaluators pull their deps from optional extras — install them opportunistically:

Extra	Install	Needed for
scene	pip install -e ".[scene]"	scene operators (scene_extract, scene_annotate) + setup-models weights
clip	pip install -e ".[clip]"	clip_embed operator / clip_model resource / real eval data embeddings
viz	pip install -e ".[viz]"	UMAP projection for eval data --viz (falls back to t-SNE/PCA)
vlmbench	pip install -e ".[vlmbench]"	eval vlm-bench aggregate (pandas)
merge	pip install -e ".[merge]"	Stage-2 → Stage-3 LoRA checkpoint merge (peft/accelerate)
dev	pip install -e ".[dev]"	pytest + ruff

The only registered git submodule is the eval fork; fetch it when you need eval vlm-bench:

git submodule update --init third_party/VLMEvalKit

Stage 2 and Stage 3 environments

VLAFactory does not install or pin environments for LLaMA-Factory or any of the VLA backends. Add them as your own checkouts under third_party/ and follow their install docs:

# Stage 2 — LLaMA-Factory (user-added checkout; only its data/ dir is vendored in-repo)
git clone https://github.com/<your-org>/LLaMA-Factory.git third_party/LLaMA-Factory
# then follow third_party/LLaMA-Factory/README.md to create its env

# Stage 3 — StarVLA (user-added checkout)
git clone https://github.com/<your-org>/StarVLA.git third_party/StarVLA
# then follow third_party/StarVLA/README.md to create its env

# SimLingo is an external checkout you point at, not something you add here:
git clone https://github.com/cxcscmu/simlingo.git   # → set extra.simlingo_dir in the VLA config

When you run a stage, either activate the right env first or set conda_env: <env_name> in the stage config — the CLI shells out via conda run -n <env_name> ....

Quickstart

1. Write a DataForge recipe

A recipe is a pipeline header plus a list of stages; each stage names an operator, wires its inputs to an upstream stage's .output, and passes operator config. This is a lightly trimmed cut of the shipped configs/data/emnlp_demo.yaml — CPU-only, and runnable as-is:

pipeline:
  name: emnlp_demo
  workdir: /tmp/vlafactory_demo          # node-local: fast small-shard writes

stages:
  - name: load                            # raw candidate driving captions
    op: jsonl_source
    config: { source: ./examples/emnlp_demo/candidates.jsonl }

  - name: quality_filter                  # keep scenes with ≥ 2 grounded objects
    op: expression_filter
    inputs: { data: $load.output }
    config: { expression: "n_objects >= 2" }

  - name: dedup                           # drop duplicate captions
    op: dedup
    inputs: { data: $quality_filter.output }
    config: { key: caption }

  - name: sample                          # deterministic training budget
    op: subsample
    inputs: { data: $dedup.output }
    config: { n: 6, seed: 42 }

  - name: sharegpt                        # LLaMA-Factory-ready ShareGPT JSON
    op: sharegpt_sink
    inputs: { data: $sample.output }
    config: { dest: /tmp/vlafactory_demo/selected.jsonl,
              image_column: filename, response_column: caption }

See vlafactory data list-operators for the full catalog (32 operators), and docs/architecture-design.md for the operator model. .yaml.j2 + a vars file gives you templated/sharded recipes (TOML is still parsed for backward compatibility).

2. Run it

vlafactory data run    configs/data/emnlp_demo.yaml        # runs: 12 → 10 → 9 → 6 rows
vlafactory data run    configs/data/emnlp_demo.yaml        # again: every stage cached & skipped
vlafactory data run    configs/data/emnlp_demo.yaml --only load,quality_filter
vlafactory data run    configs/data/emnlp_demo.yaml --rerun dedup    # rerun a stage + downstream
vlafactory data run    configs/data/emnlp_demo.yaml --force          # ignore caches
vlafactory data status configs/data/emnlp_demo.yaml        # per-stage cache status
vlafactory data inspect /tmp/vlafactory_demo/selected.jsonl

Prefer to watch it run? scripts/emnlp_demo_play.sh is an interactive tour of the whole system (explain → real command → output). docs/EMNLP_DEMO_RUNBOOK.md is the copy-pasteable version, and examples/colab/ has it as a notebook.

3. Fine-tune a VLM

vlafactory vlm train configs/vlm/example.yaml

This registers the Stage-1 ShareGPT JSON in LLaMA-Factory's dataset_info.json, writes a LLaMA-Factory training YAML, and shells out to llamafactory-cli train ....

4. Train a VLA on the fine-tuned VLM

vlafactory vla list-backends                          # autovla  rlinf_starvla  simlingo  starvla
vlafactory vla train configs/vla/starvla_example.yaml

5. Or run the whole thing end-to-end

vlafactory pipeline run configs/pipelines/robot_e2e.yaml
vlafactory pipeline run configs/pipelines/robot_e2e.yaml --from stage3

For the autonomous-driving stack (DataForge → VLM mid-train → SimLingo), use configs/pipelines/driving_e2e.yaml (nuScenes) or driving_e2e_carla.yaml (CARLA).

6. Evaluate any stage

vlafactory eval data    <embeddings>                  # Stage 1: dataset diversity/coverage
vlafactory eval vlm     <generated_predictions.jsonl> # Stage 2: driving exact-match
vlafactory eval vlm-bench run …                       # Stage 2: general VLM ability (VLMEvalKit fork)
vlafactory eval driving <nuscenes_eval.txt>           # Stage 3: SimLingo open-loop L2@3s / CARLA ADE
vlafactory eval vla     <rollouts.json>               # Stage 3: closed-loop rollout success

Operator's guide (whole pipeline + each part): docs/running_the_pipeline.md. SimLingo training + eval: see docs/simlingo_pipeline.md. VLMEvalKit benchmark sweep: see docs/vlmeval_benchmark.md.

Multi-Run Execution (Without the Ceremony)

Your data pipelines rarely run in one shot — CPU pre-processing on one machine, GPU feature extraction on a cluster, a scoring step that needs a different env, and so on. VLAFactory handles this by treating the filesystem as the source of truth:

• The ExecutionEngine walks stages in declaration order.
• Each stage's meta.json stores a fingerprint = hash of (its config
  • referenced resource configs + the upstream fingerprint).
• If the stage's Arrow shard exists and the fingerprint matches → skip. Otherwise → run.

Use --only to run a subset, --rerun X to invalidate a stage and its downstream, --force to re-run everything, and --incre for resumable delta runs (operators with an incremental_key get only new rows, merged into the existing shard — used heavily for sharded scene extraction). No DAG engine, no manifest: when you want to know what's done, run vlafactory data status (or ls the workdir).

Extending VLAFactory

Add a new operator

1. Create vlafactory/dataforge/operators/my_op.py.
2. Subclass ArrowOperator (implement process_batch_arrow(table) -> table) or DictOperator (implement process_batch_dict(batch) -> batch); declare input_schema() / output_schema(), and resource_specs() / parallelism() as needed. Either process_batch may be async def.
3. Decorate with @register_operator("my_op").
4. Add from . import my_op to operators/__init__.py (registration is import-driven — forgetting this is the #1 cause of Unknown operator).
5. Reference my_op in a recipe.

Add a new VLA backend

1. Create a module under vlafactory/vla_training/.
2. Subclass VLABackend, implement train() and (optionally) supports_vlm().
3. Decorate with @register_vla_backend("my_backend").
4. Import it from vla_training/__init__.py.
5. Reference backend: my_backend in a VLA config.

Third-Party Strategy

Integration modes are deliberately mixed — don't "normalize" one to another.

Repo	Integration mode	Why
VLMEvalKit	git submodule → fork (the only one)	Eval-only; inference + scoring stay upstream's
LLaMA-Factory	user-added checkout under third_party/ (its data/ dir vendored)	Pure consumer; you supply/pin the env
StarVLA	user-added checkout under third_party/	Active upstream; adapter stays thin for rebases
RLinf	user-added checkout under third_party/	Hydra GRPO backend; you supply the env
AutoVLA	vendored (vlafactory/vla_training/autovla/)	Static upstream; treat as first-class code
SimLingo	external checkout, shell-out	Thin Hydra adapter; clone github.com/cxcscmu/simlingo → set extra.simlingo_dir

License

Apache-2.0. See LICENSE.