Demo: Using T to build a data science pipeline

This repository serves as a demonstration of using the new T programming lanaguage created by Bruno Rodrigues to orchestrate a data simulation pipeline. The original version of the pipeline was built with the {targets} R package and featured in the R/Pharma 2023 workshop Leveraging {targets} and {crew} to simulate clinical trials. You can view the workshop example {targets} pipeline at the wlandau/rpharma2023-pipeline repository.

Note: The T lanaguage has been designed to be compatible with AI-assisted development. To put this claim to the test, I used Claude Opus 4.7 through Open Router and opencode inside Visual Studio Code to assist wtih the conversion process. You can view the entire transcript of my prompts in the agent_session_notes/session-full-t-pipeline-dev.md document in this repository. From my initial prompt to completion, the total cost was approximately $7.00 (still less than the typical cost of a take-out meal!)

Development Setup

Prerequisites

• Install the Nix package manager following the appropriate vignette from the {rix} documentation site:
  • Linux and Windows setup guide
  • MacOS setup guide

Getting Started

1. Clone this repository to your computer and enter the reproducible environment:

nix develop

2. Run the analysis:

t run src/pipeline.t

This will generate a Quarto report based on the template report.qmd in this repository.

Project Structure

• src/pipeline.t — Main T pipeline (port of the legacy targets DAG)
• report.qmd — Quarto report consuming the results node
• targets_legacy_code/ — Original R targets project (copied from aforementinoed workshop repository, and not version-controlled here)
• data/ — Input data files (read-only)
• outputs/ — Generated artifacts (e.g. results.csv written via pipeline_copy)
• pipeline-output/ — Local mirror of built Nix artifacts (created by pipeline_copy())
• tests/ — Test files

Pipeline overview

Legacy target (_targets.R)	T node (src/pipeline.t)
tar_map_rep(simulations)	sim_strong_700, sim_strong_800, sim_null_700, sim_null_800 (R nodes)
tar_target(results)	simulations (bind_rows) + results (group/summarize)
tar_quarto(report)	report (Quarto node rendering report.qmd)

Scale is controlled by the seq_len(...) literals inside each sim_* node (default: 100 sims/scenario). Bump them to 10 000 to match the legacy 25 batches \u00d7 400 reps configuration.

After a successful build, the rendered HTML is at:

pipeline-output/report/artifact/report.html

Open it in a browser to view the results table.

Migrating a targets pipeline to T

This project was ported from a targets pipeline (see wlandau/rpharma2023-pipeline). The notes below capture the steps and pitfalls encountered during the migration, so future ports go faster.

1. Inventory the legacy pipeline

Before writing any T code, map out the legacy DAG on paper:

• Targets (tar_target, tar_map_rep, tar_quarto, …): each becomes one or more T nodes.
• R functions in R/: these get inlined into rn(...) node bodies, or — for larger projects — moved to src/R/*.R and pulled in with rn(script = "...").
• Scenarios / branching: tar_map_rep patterns translate to explicit per-scenario rn() nodes, one per branch. T pipelines are more verbose here but the DAG stays auditable.
• Quarto reports: tar_quarto(report) becomes node(script = "report.qmd", runtime = Quarto).

2. Configure tproject.toml

Declare every R package the legacy pipeline library()-s into [r-dependencies]:

[r-dependencies]
packages = ["dplyr", "tibble", "arrow", "jsonlite"]

• arrow is required for any R node that exchanges DataFrames with T via serializer = ^arrow.
• jsonlite is pulled in by T's R bridge for cross-language metadata.
• For Quarto reports, add quarto and which to [additional-tools].

After editing tproject.toml, run t update and re-enter nix develop so the new packages are on PATH and the Quarto tlang filter is symlinked into _extensions/.

3. Write the pipeline incrementally

The fastest path is one node at a time. Build often, fix the first error, repeat. The list below documents the syntactic gotchas we hit — all of them produced cryptic error messages that take a moment to unpack.

Polyglot code lives inside <{ ... }>, not { ... }

-- Wrong: T tries to parse R as T expressions
sim_node = rn(command = { lm(y ~ x, data) })

-- Right: <{ ... }> is a RawCode block treated as opaque text
sim_node = rn(command = <{ lm(y ~ x, data) }>)

Format selectors are symbols (^name), not strings

-- Wrong: TypeError ("String literals are not allowed for `serializer`")
rn(command = <{ ... }>, serializer = "arrow")

-- Right
rn(command = <{ ... }>, serializer = ^arrow)

The published docs sometimes show "arrow"; the T parser in version 0.52.0 rejects strings. The error message tells you exactly what symbol to use.

deserializer is a per-dependency dict for multi-input nodes

-- Wrong: T complains a single strategy is ambiguous
simulations = node(
  deserializer = ^arrow,
  command      = bind_rows(sim_a, sim_b, sim_c, sim_d)
)

-- Right
simulations = node(
  deserializer = [
    sim_a: ^arrow, sim_b: ^arrow, sim_c: ^arrow, sim_d: ^arrow
  ],
  command = bind_rows(sim_a, sim_b, sim_c, sim_d)
)

Local R variable names must not shadow pipeline node names

T's lexical analyzer scans <{ ... }> blocks for identifiers that match other pipeline nodes and wires them as dependencies. If your R code has results <- do.call(...) and your pipeline also has a node called results, T thinks the R node depends on results — creating a phantom cycle. Rename the local R variable (out, df_local, …).

Comments (-- or #) and string literals inside RawCode are stripped by the analyzer, so node names mentioned in comments are safe.

mutate, summarize, group_by, arrange require $col syntax

-- Wrong: T expects $col on the LHS of mutate/summarize too
|> mutate(reject = $p_value < 0.05)
|> summarize(success = mean($reject))

-- Right
|> mutate($reject = to_integer($p_value < 0.05))
|> summarize($success = mean($reject, na_rm = true))

Also worth knowing: arrange() takes one column and an optional direction. arrange($a, $b) is interpreted as "sort by $a with direction $b" and fails. Use arrange($a, "desc"), not arrange($a, $b).

4. Add the Quarto report

Quarto report wiring is mostly implicit, with two non-obvious details.

The path-substitution rule is global, not chunk-scoped

T does a sed substitution of read_node("name") over the entire .qmd source, including prose. Don't mention read_node("results") in markdown paragraphs — it will be rewritten to a Nix-store path in the rendered output. Use a different phrasing in the prose ("the results node", with backticks but without parentheses).

read_node() inside {t} chunks returns a path string, not a value

After substitution, the chunk literally contains a string like "/nix/store/.../results/artifact". To get the DataFrame back, wrap in deserialize(...):

results = deserialize(read_node("results"))
results

For R chunks, the T R helper package provides a real read_node() function that handles deserialization implicitly — but pure {t} chunks need the explicit deserialize call.

5. Debug with the artifact, not the log

Two debugging fallbacks that proved more reliable than read_log():

1. Inspect _pipeline/pipeline.nix — T-level errors that occur during Nix derivation generation get baked into this file as literal error strings. rg Error _pipeline/pipeline.nix surfaces them before Nix even tries to build.

2. Read the error artifact directly — when a node fails inside its sandbox, the error is serialized to /nix/store/<hash>-pipeline_output/<node>/artifact. The file is mostly binary, but the human-readable error message appears as plain text within it:

   cat /nix/store/<hash>-pipeline_output/<failed_node>/artifact
   # or
   strings /nix/store/<hash>-pipeline_output/<failed_node>/artifact

   The pipeline_output path is printed by t run on a failed build.

6. Materialize and inspect

build_pipeline(p)   -- builds all nodes into the Nix store
pipeline_copy()     -- mirrors artifacts into ./pipeline-output/

Without pipeline_copy(), every artifact (including the rendered HTML report) lives only in /nix/store/... and is read-only. pipeline_copy() makes them browsable from the project root.

Dependencies

Dependencies are managed declaratively via tproject.toml.

To add a new dependency:

1. Add it to the [dependencies] section of tproject.toml:

   [dependencies]
   my-pkg = { git = "https://github.com/user/my-pkg", tag = "v0.1.0" }

2. Run nix develop — the package is automatically fetched
3. Commit tproject.toml

No imperative install commands — flake.nix reads tproject.toml directly.

Editor Support

This project includes support for the T Language Server (LSP).

1. Configure your editor following the Editor Support Guide.
2. Always launch your editor from within the nix develop environment (or use direnv).

Once active, you'll get autocompletion for T functions, variables, and DataFrame columns (via $).

License

MIT