SPECIFICATION.md

# Specification

- [Specification](#specification)
  - [Introduction](#introduction)
    - [Composition by Precommands](#composition-by-precommands)
    - [Composition by Subshells](#composition-by-subshells)
    - [Composition by Rules](#composition-by-rules)
    - [Composition by Distribution](#composition-by-distribution)
  - [Terminology](#terminology)
    - [keys, values and tasks](#keys-values-and-tasks)
    - [strictly relative path](#strictly-relative-path)
  - [Project Structure](#project-structure)
    - [Workspace script](#workspace-script)
  - [Versioning](#versioning)
    - [Immutability of Versions](#immutability-of-versions)
    - [Increased Patch Numbers](#increased-patch-numbers)
    - [Increased Minor Numbers](#increased-minor-numbers)
    - [Increased Major Numbers](#increased-major-numbers)
    - [Nonstandard Semantic Versions](#nonstandard-semantic-versions)
  - [Assets](#assets)
    - [Asset Identity](#asset-identity)
    - [Web assets](#web-assets)
    - [Cell assets](#cell-assets)
    - [Workspace assets](#workspace-assets)
    - [Self assets](#self-assets)
    - [Local assets](#local-assets)
    - [Zip Archive Reproducibility](#zip-archive-reproducibility)
    - [Loading Assets](#loading-assets)
  - [Bundles](#bundles)
    - [Saving Bundles](#saving-bundles)
  - [Forms](#forms)
    - [Form Variables](#form-variables)
      - [Variable Availability](#variable-availability)
      - [${SLOT.request}](#slotrequest)
      - [${SLOT.SlotName}](#slotslotname)
      - [${SLOTABS.SlotName}](#slotabsslotname)
      - [${SLOTNAME.SlotName}](#slotnameslotname)
      - [Object Slot ABI](#object-slot-abi)
      - [${/} directory separator](#-directory-separator)
      - [${.exe.execution}](#exeexecution)
      - [${.exe.target}](#exetarget)
      - [${HOME}](#home)
      - [${CACHE}](#cache)
      - [${DATA}](#data)
      - [${CONFIG}](#config)
      - [${STATE}](#state)
      - [${RUNTIME}](#runtime)
      - [${HOMEABS}, ${CACHEABS}, ${DATAABS}, ${CONFIGABS}, ${STATEABS}, ${RUNTIMEABS}](#homeabs-cacheabs-dataabs-configabs-stateabs-runtimeabs)
    - [Execution Constraints](#execution-constraints)
      - [OSFamily](#osfamily)
    - [Precommands](#precommands)
    - [Environment Modifications](#environment-modifications)
      - [+NAME=VALUE](#namevalue)
      - [-NAME](#-name)
      - [\<NAME=VALUE](#namevalue-1)
    - [Form Processing](#form-processing)
      - [Processing Order](#processing-order)
      - [Execution Step Cacheing](#execution-step-cacheing)
      - [Windows command-line quoting](#windows-command-line-quoting)
      - [Windows `--cmd.exe` special form](#windows---cmdexe-special-form)
  - [Objects](#objects)
    - [Saving and Loading Objects](#saving-and-loading-objects)
    - [Object Slots](#object-slots)
  - [Values](#values)
    - [Value Shell Language (VSL)](#value-shell-language-vsl)
    - [VSL Lexical Rules](#vsl-lexical-rules)
      - [Types of Words](#types-of-words)
    - [Variables available in VSL](#variables-available-in-vsl)
    - [remote UI\_MODULE@VERSION COMMAND](#remote-ui_moduleversion-command)
    - [get-object MODULE@VERSION -s REQUEST\_SLOT (-f FILE | -d DIR/)](#get-object-moduleversion--s-request_slot--f-file---d-dir)
    - [run-object MODULE@VERSION -s REQUEST\_SLOT (-c COMMAND | -m MEMBER)](#run-object-moduleversion--s-request_slot--c-command---m-member)
    - [run-function MODULE@VERSION (-f FILE | -d DIR/) -- CLI\_FORM\_DOC](#run-function-moduleversion--f-file---d-dir----cli_form_doc)
    - [enter-object MODULE@VERSION -s REQUEST\_SLOT -- CLI\_FORM\_DOC](#enter-object-moduleversion--s-request_slot----cli_form_doc)
    - [merge-object MODULE@VERSION -s REQUEST\_SLOT (-f FILE | -d DIR/)](#merge-object-moduleversion--s-request_slot--f-file---d-dir)
    - [get-asset MODULE@VERSION FILE\_PATH (-f FILE | -d DIR/)](#get-asset-moduleversion-file_path--f-file---d-dir)
    - [run-asset MODULE@VERSION FILE\_PATH (-c COMMAND | -m MEMBER)](#run-asset-moduleversion-file_path--c-command---m-member)
    - [get-bundle MODULE@VERSION (-f FILE | -d DIR/)](#get-bundle-moduleversion--f-file---d-dir)
    - [Options: -f FILE and -d DIR and -x GLOB and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob)
    - [Option: \[-n STRIP\]](#option--n-strip)
    - [Option: \[-m MEMBER\]](#option--m-member)
  - [Subshells](#subshells)
    - [subshell options](#subshell-options)
    - [subshell: get-object MODULE@VERSION -s REQUEST\_SLOT](#subshell-get-object-moduleversion--s-request_slot)
    - [subshell: run-object MODULE@VERSION -s REQUEST\_SLOT (-c COMMAND | -m MEMBER)](#subshell-run-object-moduleversion--s-request_slot--c-command---m-member)
    - [subshell: run-function MODULE@VERSION -- CLI\_FORM\_DOC](#subshell-run-function-moduleversion----cli_form_doc)
    - [subshell: get-asset MODULE@VERSION FILE\_PATH](#subshell-get-asset-moduleversion-file_path)
    - [subshell: run-asset MODULE@VERSION FILE\_PATH (-c COMMAND | -m MEMBER)](#subshell-run-asset-moduleversion-file_path--c-command---m-member)
    - [Anonymous Files: `-f :` or `-f BASENAME`](#anonymous-files--f--or--f-basename)
    - [Anonymous Directories: `-d :`](#anonymous-directories--d-)
    - [Object ID with Build Metadata](#object-id-with-build-metadata)
    - [JSON Files](#json-files)
    - [JSON Canonicalization](#json-canonicalization)
  - [Distributions](#distributions)
    - [Distributions are sealed](#distributions-are-sealed)
      - [Adding to the sealed set](#adding-to-the-sealed-set)
    - [Distributed Value Stores](#distributed-value-stores)
    - [OpenBSD signify keys](#openbsd-signify-keys)
      - [Distribution versioning](#distribution-versioning)
    - [GitHub SLSA Level 2](#github-slsa-level-2)
    - [GitHub SLSA Level 3](#github-slsa-level-3)
    - [Distribution Scripts](#distribution-scripts)
  - [Scripts](#scripts)
    - [Script Introduction](#script-introduction)
    - [Script Phases](#script-phases)
    - [Lua Specification](#lua-specification)
    - [Lua Global Variables](#lua-global-variables)
      - [Lua Global Variable - arg](#lua-global-variable---arg)
      - [Lua Global Variable - loadstring](#lua-global-variable---loadstring)
      - [Lua Global Variable - next](#lua-global-variable---next)
      - [Lua Global Variable - tostring](#lua-global-variable---tostring)
      - [Lua Global Variable - print](#lua-global-variable---print)
      - [Lua Global Variable - printf](#lua-global-variable---printf)
      - [Lua Global Variable - tonumber](#lua-global-variable---tonumber)
      - [Lua Global Variable - type](#lua-global-variable---type)
      - [Lua Global Variable - assert](#lua-global-variable---assert)
      - [Lua Global Variable - error](#lua-global-variable---error)
    - [Lua build library](#lua-build-library)
      - [build.newrules](#buildnewrules)
    - [Lua jsondk library](#lua-jsondk-library)
      - [jsondk.encode](#jsondkencode)
      - [jsondk.decode](#jsondkdecode)
      - [jsondk.null](#jsondknull)
    - [Lua math library](#lua-math-library)
      - [math.abs](#mathabs)
      - [math.acos](#mathacos)
      - [math.asin](#mathasin)
      - [math.atan](#mathatan)
      - [math.ceil](#mathceil)
      - [math.cos](#mathcos)
      - [math.deg](#mathdeg)
      - [math.exp](#mathexp)
      - [math.floor](#mathfloor)
      - [math.fmod](#mathfmod)
      - [math.huge](#mathhuge)
      - [math.log](#mathlog)
      - [math.max](#mathmax)
      - [math.maxinteger](#mathmaxinteger)
      - [math.min](#mathmin)
      - [math.mininteger](#mathmininteger)
      - [math.modf](#mathmodf)
      - [math.pi](#mathpi)
      - [math.rad](#mathrad)
      - [math.sin](#mathsin)
      - [math.sqrt](#mathsqrt)
      - [math.tan](#mathtan)
      - [math.tointeger](#mathtointeger)
      - [math.type](#mathtype)
      - [math.ult](#mathult)
    - [Lua package library](#lua-package-library)
      - [require](#require)
      - [package.registrykey](#packageregistrykey)
    - [Lua request.rule library](#lua-requestrule-library)
      - [request.rule.generatesymbol](#requestrulegeneratesymbol)
    - [Lua request.execution library](#lua-requestexecution-library)
      - [request.execution.OSFamily](#requestexecutionosfamily)
      - [request.execution.ABIv3](#requestexecutionabiv3)
      - [request.execution.OSv3](#requestexecutionosv3)
    - [Lua request.io library](#lua-requestio-library)
      - [request.io.open](#requestioopen)
      - [request.io.read](#requestioread)
      - [request.io.write](#requestiowrite)
      - [request.io.list](#requestiolist)
      - [request.io.isfile](#requestioisfile)
      - [request.io.isdir](#requestioisdir)
      - [request.io.realpath](#requestiorealpath)
      - [request.io.toasset](#requestiotoasset)
      - [request.io.flush](#requestioflush)
      - [request.io.close](#requestioclose)
    - [Lua request.submit library](#lua-requestsubmit-library)
      - [request.submit.outputid](#requestsubmitoutputid)
      - [request.submit.outputmodule](#requestsubmitoutputmodule)
      - [request.submit.outputversion](#requestsubmitoutputversion)
    - [Lua request.ui library](#lua-requestui-library)
      - [request.ui.glob](#requestuiglob)
      - [request.ui.spawn](#requestuispawn)
      - [request.ui.capture](#requestuicapture)
      - [request.ui.checksum](#requestuichecksum)
      - [request.ui.readfile](#requestuireadfile)
      - [request.ui.writefile](#requestuiwritefile)
      - [request.ui.signify](#requestuisignify)
      - [request.ui.sleep](#requestuisleep)
      - [request.ui.buildpubkey](#requestuibuildpubkey)
    - [Lua string library](#lua-string-library)
      - [string.byte](#stringbyte)
      - [string.find](#stringfind)
      - [string.format](#stringformat)
      - [string.len](#stringlen)
      - [string.lower](#stringlower)
      - [string.rep](#stringrep)
      - [string.sub](#stringsub)
      - [string.upper](#stringupper)
    - [Lua stringdk library](#lua-stringdk-library)
      - [stringdk.quote\_posix\_shell](#stringdkquote_posix_shell)
      - [stringdk.quote\_value\_shell\_literal](#stringdkquote_value_shell_literal)
      - [stringdk.quote\_value\_shell\_evalable](#stringdkquote_value_shell_evalable)
      - [stringdk.quote\_windows\_batch](#stringdkquote_windows_batch)
      - [stringdk.sanitizesubpath](#stringdksanitizesubpath)
    - [Lua table library](#lua-table-library)
      - [table.concat](#tableconcat)
      - [table.getn](#tablegetn)
      - [table.insert](#tableinsert)
      - [table.move](#tablemove)
      - [table.pack](#tablepack)
      - [table.remove](#tableremove)
      - [table.unpack](#tableunpack)
    - [Lua unified library](#lua-unified-library)
      - [unified.existingoutput](#unifiedexistingoutput)
      - [unified.sections](#unifiedsections)
      - [unified.scriptmodver](#unifiedscriptmodver)
      - [unified.asset](#unifiedasset)
      - [unified.envmod](#unifiedenvmod)
      - [unified.output](#unifiedoutput)
      - [unified.assign](#unifiedassign)
      - [unified package target](#unified-package-target)
    - [Lua workspace globals](#lua-workspace-globals)
      - [import](#import)
        - [import type=github-l2](#import-typegithub-l2)
    - [Custom Lua Modules](#custom-lua-modules)
    - [Introduction to Custom Lua Rules](#introduction-to-custom-lua-rules)
    - [Function Rules](#function-rules)
    - [Function Rule Command - `declareoutput`](#function-rule-command---declareoutput)
    - [Function Rule Command - `submit`](#function-rule-command---submit)
    - [UI Rules](#ui-rules)
    - [UI Rule Command - `submit`](#ui-rule-command---submit)
    - [UI Rule Command - `ui`](#ui-rule-command---ui)
    - [Rule Argument - `request`](#rule-argument---request)
    - [Rule Argument - `continue_`](#rule-argument---continue_)
    - [Rule Request Documents](#rule-request-documents)
    - [Embedded File Scripts](#embedded-file-scripts)
      - [Behavior of Embedded Lua](#behavior-of-embedded-lua)
      - [Embedded Language Codes](#embedded-language-codes)
      - [Recognizing Embedded Lua](#recognizing-embedded-lua)
    - [Writing Lua Rules](#writing-lua-rules)
      - [Rule Requirements](#rule-requirements)
    - [Error Handling in Rules](#error-handling-in-rules)
    - [Form Document](#form-document)
      - [Form Command Line](#form-command-line)
      - [Option Groups](#option-groups)
  - [Data Flow](#data-flow)
    - [Task Model](#task-model)
    - [Trace Store](#trace-store)
    - [Value Store](#value-store)
      - [Value Id Formulas](#value-id-formulas)
      - [Object Ids Hide Build Non-Determinism](#object-ids-hide-build-non-determinism)
      - [v - parsed values.json AST](#v---parsed-valuesjson-ast)
      - [i - index file](#i---index-file)
      - [BLD - Build Metadata](#bld---build-metadata)
      - [V256 - SHA256 of Values File](#v256---sha256-of-values-file)
      - [P256 - SHA256 of Asset](#p256---sha256-of-asset)
      - [Z256 - SHA256 of Zip Archive File](#z256---sha256-of-zip-archive-file)
      - [CT - Compatibility Tag](#ct---compatibility-tag)
      - [VCI - Values Canonical ID](#vci---values-canonical-id)
      - [VCK - Values Checksum](#vck---values-checksum)
      - [FRM - Form](#frm---form)
      - [ACI - Asset Canonical Id](#aci---asset-canonical-id)
      - [BCI - Bundle Canonical Id](#bci---bundle-canonical-id)
  - [Evaluation](#evaluation)

## Introduction

This specification documents interoperable systems that use composition to build multi-language, repeatable software in a loose federation of packages and implementations.

These systems are parameterizable, incremental and remote cacheable build systems that have the following requirements:

- Interoperable systems: There is an open-source `dksrc/dk0` *reference implementation* build system. The `dk` build system does not yet conform to this specification (as of 2025-12-19) but will change. Both are OCaml-based, and the hope is that other programming languages can have their own implementations.
- Composition: This specification has four (4) different ways to take functionality provided by others and use them in your own builds (ie. composition).
- Build multi-language, repeatable software: The domain is software that must be built with multiple programming languages. The goal is to have at least one implementation capable of opt-in, bit-by-bit reproducibility.
- Loose federation of packages and implementations: There is a default, optional central registry of vendors who build packages themselves, using any specification-conforming implementation of the build system they desire. The registry is of vendors and their signing keys, not of the packages themselves.

Most of the meat is in the "composition" requirement, so this specification will start by explaining the four (4) compositions.

### Composition by Precommands

At the lowest level the build system operates on a JSON data model.
Build configuration can be represented directly in JSON files or JSON with comments ("JSONC") files.

Multiple build configurations can be composed through **precommands** that are linked by identifiers:

```json
// filename: producer.values.json
{
  "$schema": "https://diskuv.com/dk/schema/dk-value-1.0.json",
  "schema_version": { "major": 1, "minor": 0 },
  "bundles": [
    {
      "id": "OurExample_Composition.SomeFiles@1.2.3",
      "listing": {
        "origins": [ {
            "name": "github-release",
            "mirrors": [
              "https://github.com/diskuv/dk/releases/download/2.4.202508011516-signed"
            ] } ]
      },
      "assets": [
        {
          "origin": "github-release",
          "path": "dk-darwin_arm64",
          "size": 8810960,
          "checksum": {
            "sha256": "aedc1831f3dc4af8c3fd9eefcf4fd2edf9b0f47e3534e382e20368ff15857393"
          }
        },
        {
          "origin": "github-release",
          "path": "dk-windows_x86_64.exe",
          "size": 8732160,
          "checksum": {
            "sha256": "68514ecd6d4ba6508acab15745473f2a00a51e09a78e1d72fa284d68704093d7"
          } } ] } ]
}
```

```json
// filename: consumer.values.json
{
  "$schema": "https://diskuv.com/dk/schema/dk-value-1.0.json",
  "schema_version": { "major": 1, "minor": 0 },
  "forms": [
    {
      "id": "OurExample_Composition.UseFiles@4.5.6",
      "precommands": {
        "private": [
          "get-asset OurExample_Composition.SomeFiles@1.2.3 -p dk-darwin_arm64 -f ${SLOT.Release.Darwin_arm64}/dk",
          "get-asset OurExample_Composition.SomeFiles@1.2.3 -p dk-windows_x86_64 -f ${SLOT.Release.Windows_x86_64}/dk.exe"
        ]
      },
      "outputs": {
        "assets": [
          {
            "slots": ["Release.Darwin_arm64"],
            "paths": ["dk"]
          },
          {
            "slots": ["Release.Windows_x86_64"],
            "paths": ["dk.exe"]
          } ] } } ]
}
```

As a user, if you were to run:

```sh
get-object OurExample_Composition.UseFiles@4.5.6 -s Release.Darwin_arm64 -d target/
```

the macOS/Silicon executable `dk` would appear in the `target/` directory because
`OurExample_Composition.UseFiles@4.5.6` uses `OurExample_Composition.SomeFiles@1.2.3`.

In general:

- You submit *forms* that produce *objects* created from *assets*.
- The **assets** are input materials. These are files and folders that may be remote: source code, data files, audio, image and video files.
- A **form** is a document with fields and a submit button. *Tip for engineers*: A form does not need to be entered on a graphical user interface. If you are comfortable with the DOS or Unix terminal, the document is a command line in your terminal. That is, you type the name of an executable followed by options like `--username` as the fields, and then you press ENTER to submit the form. The `dk` scripting system (doc: <https://github.com/diskuv/dk>) is a simple way to make standalone executables/forms.
- An **object** is a folder that the form produces.

The relevant sections of the specification are:

- [Assets](#assets)
- [Forms](#forms)
- [Objects](#objects)
- [Values](#values)

### Composition by Subshells

The [precommands we saw](#composition-by-precommands) were *value* shell commands:

```sh
get-asset OurExample_Composition.SomeFiles@1.2.3 -p dk-darwin_arm64 -f ${SLOT.Release.Darwin_arm64}/dk
```

These value shell commands can spawn other value shell commands by using the syntax `$(subcommand to spawn ...)`.

Consider the following snippet from JSON build configuration that fetches PowerShell using a tool provided by `.NET`:

```json
{
  "$schema": "https://diskuv.com/dk/schema/dk-value-1.0.json",
  "schema_version": { "major": 1, "minor": 0 },
  "forms": [
    {
      "id": "CommonsBase_Shell.Pwsh@7.5.4",
      "function": {
        "execution": [
          {
            "name": "OSFamily",
            "value": "$(get-asset CommonsBase_Shell.Pwsh.Lookup@1.0.0 -p osfamily -m ./${SLOTNAME.request})"
          }
        ],
        "envmods": [
          "+DOTNET_ROOT=$(get-object CommonsBase_Dotnet.SDK@10.0.100-rc.2.25502.107 -s ${SLOTNAME.Release.execution_abi} -d :)"
        ],
        "commands": [
          "$(get-object CommonsBase_Dotnet.SDK@10.0.100-rc.2.25502.107 -s ${SLOTNAME.Release.execution_abi} -d :)/dotnet${.exe.execution}",
          "tool",
          "install",
          "PowerShell",
          "--arch", "$(get-asset CommonsBase_Dotnet.Lookup@1.0.0 -p arch -m ./${SLOTNAME.request})",
          "--tool-path", "${SLOT.request}",
          "--version", "7.5.4",
          "--configfile", "$(get-asset CommonsBase_Shell.Pwsh.Bundles@7.5.4 -p NuGet.Config -f :file:NuGet.Config)"
        ]
      },
      "outputs": {
        "assets": [
          {
            "slots": [
              "Release.Windows_x86_64",
              "Release.Darwin_arm64"
            ],
            "paths": [
              ".store/powershell/7.5.4/powershell/7.5.4/.nupkg.metadata",
              // ...
              ".store/powershell/7.5.4/project.assets.json",
              "pwsh.exe"
            ]
          }
        ]
      }
    }
  ]
}
```

The `.NET` tool (`dotnet.exe`) has many requirements that are satisfied using subshells:

- *execution*: The build system must only run the Windows `dotnet.exe` on Windows, macOS `dotnet`, etc.
- *envmods*: The subshell will do the install of the .NET runtime system, and set `DOTNET_ROOT` to the installation directory.
- *args*: We want to run the dotnet.exe tool. A subshell gets us the path to `dotnet.exe`. And it helps set command line flags .NET needs.

The relevant sections of the specification are:

- [Assets](#assets)
- [Forms](#forms)
- [Objects](#objects)
- [Values](#values)
- [Subshells](#subshells)

### Composition by Rules

*Values* can be dynamically generated by a rule function.

*Parameterizable* in the [Introduction](#introduction) means the use of [partial application](https://en.wikipedia.org/wiki/Partial_application) to define new builds, but that doesn't roll off the tongue. [rule functions](#function-rules) are curried functions that take a single [request document](#rule-request-documents) parameter.

Consider the `OurTest_Exec.PostObject.TestRequest.EchoRequest@1.0.0` rule:

```lua
local M = { id = "OurTest_Exec.PostObject.TestRequest@1.0.0" }
local json = require("jsondk")
rules = build.newrules(M)

function rules.EchoRequest(command, request)
    local path = "a/path"
    if command == "declareoutput" then
        return {
            declareoutput = {
                return_asset = {
                    id = "OurTest_Exec." .. request.rule.generatesymbol() .. "@1.0.0",
                    path = path
                }
            }
        }
    elseif command == "submit" then
        local file = request.io.open("some/asset/file", "w")

        request.io.write(file, "This line is from the example. There is more:\n")
        request.io.write(file, jsondk.encode(request.user, { indent = 1 }))

        local origin, asset = request.io.toasset(file, {
            path = path, origin_name = "example-origin"
        })
        return {
            submit = {
                values = {
                    schema_version = { major = 1, minor = 0 },
                    bundles = {
                        {
                            id = request.submit.outputid,
                            listing = {
                                origins = { origin }
                            },
                            assets = { asset }
                        }
                    }
                }
            }
        }
    end
end

return M
```

When the function rule is run with the value shell:

```sh
run-function OurTest_Exec.PostObject.TestRequest.EchoRequest@1.0.0 -f out-file pet[0][species]=Dahut pet[0][name]=Hypatia "pet[1][species]=Felis Stultus" pet[1][name]=Billie
```

the build system will create a task for one [asset](#assets), fetch that asset and save it to the output file `out-file`:

```text
This line is from the example. There is more:
{
  "pet": [
    { "name": "Hypatia", "species": "Dahut" },
    { "name": "Billie", "species": "Felis Stultus" }
  ]
}
```

The relevant sections of the specification are:

- [Assets](#assets)
- [Forms](#forms)
- [Objects](#objects)
- [Values](#values)
- [Scripts](#scripts)

### Composition by Distribution

The build system allows remote importing of both the JSON build configuration *and* the build artifacts.

Anyone who wants to distribute a package can programmatically create a JSON build config with

- *assets* containing the build artifacts and JSON build configuration
- a *distribution* saying where the build artifacts were built

For example:

```json
{
  "$schema": "https://diskuv.com/dk/schema/dk-value-1.0.json",
  "schema_version": { "major": 1, "minor": 0 },
  "bundles": [
    {
      "assets": [
        // ... the build artifacts and JSON files
      ],
      "id": "CommonsBase_Std.Distribution@2.4.202510100005",
      "listing": {
        "origins": [
          {
            "name": "CommonsBase_Std",
            "mirrors": [
              "https://github.com/diskuv/dk/releases/download/2.4.202510100005"
            ]
          }
        ]
      }
    }
  ],
  "distributions": [
    {
      "build": {
        // details about the build
      },
      "id": "CommonsBase_Std@2.4.202510100005",
      "license": { "spdx": "Apache-2.0" },
      "producer": {
        "github_slsa_v1_l2": { "repository": "diskuv/dk" },
        "openbsd_signify": {
          "public_key": "untrusted comment: CommonsBase_Std-2.4\nRWSosfKCnNCBOIYVnoJMLwGKyImGd6YMrWSjj929hv087OMmR4pvf0pe\n"
        }
      }
    }
  ]
}
```

A full-example is <https://github.com/dkpkg/CommonsBase_LLVM/blob/5b2e2cdd080eefca919cdcb436e88bd67d5b6b57/etc/dk/i/CommonsBase_Std.2.6.20260718222208.values.json>.

The relevant sections of the specification are:

- [Assets](#assets)
- [Forms](#forms)
- [Objects](#objects)
- [Values](#values)
- [Scripts](#scripts)
- [Distributions](#distributions)

## Terminology

### keys, values and tasks

A value is a bundle, an asset, a form, an object or an script. A value is immutable (it should not change).
The [Values](#values) section has more detail.

A key is an identifier in the format `MODULE@VERSION` used to:

- locate an up-to-date value from a [Value Store](#value-store)
- if there is no up-to-date value, locate a recipe (called a "task") from a [Task Model](#task-model) that can produce that value

The simple `make` build system has the following:

- key: a filename
- value: the contents of a file
- task: a Makefile recipe

The build system in this specification is a generalization of those keys.

### strictly relative path

- Not an absolute path
- After the path is normalized, there are no path segments that start with `..` will raise an error.
- After the path is normalized, there are no path segments that contain a forward or backward slash. For example, even though Unix filenames can contain backslashes, but they cannot be strictly relative paths.

## Project Structure

The build system borrows terminology from the [Buck2 build system](https://buck2.build/docs/concepts/key_concepts/).

A **project**:

- has a **project directory**.
- is a container for cells (more on this shortly)

A **cell** is:

- a subdivision of the project source code
  - there is at least one subdivision called `root`
  - build system implementations define the cells using conventions, project configuration files, and/or command-line options.

Cells exist to make efficient builds and to locating source code even when source code is vendored.

- In small projects the `root` may be the only cell; all project source code belongs to `root`.
- In large projects (ex. monorepos), the project tree can be broken into smaller cells. Only parts of the build that depend on smaller cells will be rebuilt when a single project source file changes.

However, since cells must be given on the command line or some other user-specific configuration file, a **distributed package** for use by other packages should **never use cells**. Instead [workspace scripts](#workspace-script) have workspace assets that can be used.

### Workspace script

When a build system command that takes a [unified script](UNIFIED_SCRIPTS.md) (for example `dk0 test <script>` or `dk0 distribute <script>`) runs, the *workspace script* is the first unified script, searched in the order below, that contains a `## workspace` section:

1. The user `<script>` itself.
2. A file named `dk.u` in the directory containing `<script>`, if it exists.
3. A file named `dk.u` in the first ancestor directory of `<script>` that contains one.

Having a workspace makes available:

- [importing third party distributions](#import)
- [workspace-scoped assets](#unifiedasset). The workspace script's [unified.asset](#unifiedasset) declarations in non-workspace sections are available to the user `<script>`.

## Versioning

The build systems use semantic versioning defined by [semver 2.0](https://semver.org/).

### Immutability of Versions

Once a value is distributed with a version, the value should not change. Violations mean that
consumers will have incorrect and outdated values in their caches.

Comment or formatting edits to a values.json or a values.lua that leave behavior
identical may safely be edited on the same distributed version.

Sealing is append-only and gated at the distribution level; see
[Increased Minor Numbers](#increased-minor-numbers).

### Increased Patch Numbers

Example: 0.3.0 to 0.3.1

A behavior-preserving repair should increase the patch number.
Existing consumers can keep their existing versions and opt-in to the repair
by importing a release that carries the new version.

Even though [distributions are sealed](#distributions-are-sealed), an increased patch
number does not require a new [signify key](#openbsd-signify-keys).

### Increased Minor Numbers

Example: 0.3.0 to 0.4.0

A new capability should increase the minor number:

- Adding a new rule to a scriptmodule (values.lua), or a new scriptmodule, is a new capability.
- Adding a new module to a values.jsonc, or a new values.jsonc, is a new capability.

A distribution may add to its sealed set, either a brand-new module or a
strictly higher `MAJOR.MINOR` version of an already-sealed module, only when
the distribution's own `MAJOR.MINOR` is strictly higher than the latest
sealed release. A release within an already-sealed `MAJOR.MINOR` line can
neither add modules nor add module `MAJOR.MINOR` versions.

Minor numbers require a new [signify key](#openbsd-signify-keys) when the new minor number has
not already been [sealed by the previous distribution](#distributions-are-sealed).
In other words, a small number of minor number increases will eventually
lead to the producer having to create a new set of sealed signify keys;
that requires the producer to establish that they have possession of the
secret key.

That is a security property: new capabilities
increase the security surface, and versioning requires the producer
to eventually authenticate their ownership of the changes.

Consumers opt in by referencing the new version; existing references
keep their existing behavior.

### Increased Major Numbers

Example: 0.3.0 to 1.0.0

A backwards-incompatible change should increase the major number.

Major numbers, like minor numbers, require a new [signify key](#openbsd-signify-keys) when the new major number has
not already been [sealed by the previous distribution](#distributions-are-sealed).

Consumers opt in by referencing the new version; existing references
keep their existing behavior.

### Nonstandard Semantic Versions

Often values do not have a natural semantic version.

An example would be a privacy policy contained in a bundle module `YourOrg_Std.StringsForWebSiteAndPrograms.PrivacyPolicy`.
`YourOrg_Std.StringsForWebSiteAndPrograms.PrivacyPolicy@1.0.20250904` could be used for the 2025-09-04 privacy policy.

## Assets

Assets are remote or local files that are inputs to a build. All assets have SHA-1 or SHA-256 checksums.

Assets are accessed with the [get-asset](#get-asset-moduleversion-file_path--f-file---d-dir) command described in a later section of the document.

### Asset Identity

Assets are located with one or more URL or file path mirrors:

| Scheme     | What                                   |
| ---------- | -------------------------------------- |
| `https://` | [HTTP over TLS web asset](#web-assets) |
| `http://`  | [HTTP web asset](#web-assets)          |
| `cell://`  | [Cell asset](#cell-assets)             |
| `file://`  | [Local asset](#local-assets)           |
| *none*     | [Local asset](#local-assets)           |

Assets are also created on-demand:

| Location | What                                 |
| -------- | ------------------------------------ |
| `dk.u`   | [Workspace asset](#workspace-assets) |

Each asset is uniquely identified by its SHA256 checksums.
That unique identification means that mirrors for assets can be added
at any time (including in the future) without triggering rebuilds.

### Web assets

Examples:

```text
https://example.com/asset1.zip
http://example.com/asset2/
```

### Cell assets

Examples:

```text
cell://root/etc/table/dotnet
```

Cells are local directories within the [project structure](#project-structure). The `root` cell is predefined to be the working directory that the build system was called from.

For example, `cell://root/etc/table/dotnet` is the local directory `etc/table/dotnet` within the cell `root`.

### Workspace assets

Assets are created by `dk.u` workspace script's [unified.asset](#unifiedasset) declarations. The module-version in the non-workspace section title becomes the base of the asset.

So:

```text
### CommonsBase_GNU.Make.Apparatus@4.4.1

  % unified.asset { name="BuildW32Bat", file="assets/p/patched-make-build_w32.bat" }
  \dk.asset(byteSize: "11131", checksum: (sha256: "3f9825744b486bc5413806cccbae36a4d354e944765fba68c26d37884369a06d"))\;
```

creates an asset `CommonsBase_GNU.Make.Apparatus.BuildW32Bat@4.4.1` with path `-p assets/p/patched-make-build_w32.bat`.

### Self assets

Examples:

```text
selfasset://test-origin
```

provides access to [custom Lua rule assets](#function-rules) described in later sections like in the following example:

```lua
local M = { id = "OurTest_Exec.PostObject.TestRequest@1.0.0" }
rules = build.newrules(M)
function rules.EchoRequest(command, request)
    local path = "a/path"
    if command == "submit" then
        local file = request.io.open("some/asset/file", "w")
        request.io.write(file, "This is my asset!\n")
        local origin, asset = request.io.toasset(file, {
            path = path, origin_name = "test-origin"
        })
        return {
            submit = {
                values = {
                    schema_version = { major = 1, minor = 0 },
                    bundles = {
                        {
                            id = request.submit.outputid,
                            listing = {
                                origins = { origin }
                            },
                            assets = { asset }
                        }
                    }
                }
            }
        }
    end
end

return M
```

[request.io.toasset](#requestiotoasset) creates the `selfasset` URLs on-demand.

### Local assets

Examples:

```text
file://C:/source
C:\source
file:///usr/src/main.tar.gz
/usr/src/main.tar.gz
a/b/c
```

A local asset may be either:

- a file
- a directory

A relative file or directory is relative to the [workspace directory](#workspace-script).

A local directory path is always zipped into a zip archive file, where the [Zip Archive Reproducibility (next section)](#zip-archive-reproducibility) standards will be followed.

### Zip Archive Reproducibility

 For reproducibility, the generated zip archive file will:

- have the zip last modification time to the earliest datetime (Jan 1, 1980 00:00:00)
- have each zip entry with its modification time to the earliest datetime (Jan 1, 1980 00:00:00)
- have each zip file entry set its external file attribute to a Unix "regular file" with `rwxr-x---` permissions (octal `0100750`). Every entry is marked with these same permissions because, when a zip is created on a Windows host for a Unix target, the creator cannot reliably tell which files are executable; a uniform attribute keeps the archive bytes reproducible regardless of host. This stored attribute does **not** by itself make files executable on extraction: an extracting command (`get-object`, `run-object`, `get-asset`, ...) writes non-executable files by default and makes executable only the entries matched by its `-e GLOB` options (see [Options: -f FILE and -d DIR and -x GLOB and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob)).
- compress every entry with the DEFLATE method (zip [compression method](https://www.iana.org/assignments/media-types/application/zip) `8`) at compression level 5, even for tiny file entries (nothing is stored uncompressed)
- always accompany a zip directory entry with a zip64 directory entry
- produce the DEFLATE bit stream with the reference encoder: the vendored `miniz` at the commit pinned in `MlFront_ZipFile`'s notes. RFC 1951 admits many valid DEFLATE encodings of the same data, so byte-for-byte reproducibility across implementations depends on using that same encoder. dk0 and dk1 use the native `miniz`; a JavaScript implementation (dkjs) uses the same `miniz` compiled to WebAssembly. The archive bytes, and therefore the [Z256](#z256---sha256-of-zip-archive-file) of the archive, are then identical across implementations.

The multi-disk-volume "index" archive format, which lets a single entry be fetched from a large remote archive without downloading the whole archive, is specified in `MlFront_ZipFile`'s notes, and the same reference `miniz` produces it on every implementation.

### Loading Assets

The asset key of [get-asset](#get-asset-moduleversion-file_path--f-file---d-dir):

```sh
get-asset MODULE@VERSION -p PATH
```

has the components:

1. MODULE: Module ID
2. VERSION: Module Version
3. PATH: Asset path

The first two (2) components uniquely identify a [bundle](#bundles) while
the third component uniquely identifies the asset within a bundle like:

```json
"bundles": [
  // This is bundle identified by id = MODULE@VERSION
  {
    "id": "DkDistribution_Std.Bundle@2.4.202508011516-signed",
    "listing": {
      "origins": [
        {
          "name": "github-release",
          "mirrors": [
            "https://github.com/diskuv/dk/releases/download/2.4.202508011516-signed"
          ]
        }
      ]
    },
    "assets": [
      // This is the asset identified by path = PATH
      {
        "origin": "github-release",
        "path": "SHA256.sig",
        "size": 151,
        "checksum": {
          "sha256": "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
        }
      }
    ]
  }
]
```

## Bundles

Bundles are a named collection of assets.

Bundles are accessed with the [get-bundle](#get-bundle-moduleversion--f-file---d-dir) command described in a later section of the document.

### Saving Bundles

When a value shell command reads an bundle and saves it to a file (ex.
[get-bundle -f FILE](#get-bundle-moduleversion--f-file---d-dir)),
the members of the bundle are zipped and the zip archive bytes are copied directly to the file.
The standards of [Zip Archive Reproducibility](#zip-archive-reproducibility) will be followed.

When a value shell command reads an bundle and saves it to a directory (ex.
[get-bundle -d DIR](#get-bundle-moduleversion--f-file---d-dir)),
the members of the bundle are copied into the directory tree.

That sounds inefficient, but the build system is allowed to optimize a set of value shell commands.
For example, if one shell command saves output into a directory,
and a second shell command reads data from created by the first shell command,
the build system can give the second shell command a symlink to the first directory
**without** using a zip archive as an intermediate artifact.

## Forms

A form has a [request slot](#slotrequest) that must always be specified by the user as a parameter.

### Form Variables

#### Variable Availability

Some variables are available in the Value Shell Language (VSL); see [Variables available in VSL](#variables-available-in-vsl)

All variables are available in `.forms.function.args` and `.forms.function.envmods`.

#### ${SLOT.request}

The output directory for the *request slot*. The `-s REQUEST_SLOT` option (ex. `get-object MODULE@VERSION -s REQUEST_SLOT`) is the request slot.

If the command has no request slot (ex. `get-bundle MODULE@VERSION`) and you use `${SLOT.request}`, an error is reported.

#### ${SLOT.SlotName}

The output directory for the form function for the slot named `SlotName`.

`SlotName` is a period-separated list of "*parts*":

- lowercase context variables, and/or
- capitalized namespace terms

The namespace terms and context variables can be combined in any order.

See [${SLOTNAME.SlotName}](#slotnameslotname) for a description of parts and the context variables.

Output directories for the build system in install mode are the end-user installation directories, while for other modes the output directory may be a sandbox temporary directory.

Expressions are only evaluated if *all* the output types the expression uses are valid for the build system. For example, an expression that uses the output directory `${SLOT.Release.Darwin_arm64}` will be skipped by the build system in install mode if the end-user machine's ABI is not `darwin_arm64`.

More generally:

| Type                           | Expression Evaluated? | Immediate Thunk Controller |
| ------------------------------ | --------------------- | -------------------------- |
| `${SLOT.Release.Agnostic}`     | Always                | A sandbox directory        |
| `${SLOT.Release.Darwin_arm64}` | Always                | A sandbox directory        |

| Type                           | Expression Evaluated?              | Install Thunk Controller |
| ------------------------------ | ---------------------------------- | ------------------------ |
| `${SLOT.Release.Agnostic}`     | Always                             | The install directory    |
| `${SLOT.Release.Darwin_arm64}` | Only if the end-user machine's ABI | The install directory    |
|                                | is `darwin_arm64`                  |                          |

#### ${SLOTABS.SlotName}

The absolute path of the output directory for the form function for the slot named `SlotName`.

CAUTION: ⚠️ Using this variable can lead to [non-relocatable packages](https://docs.conda.io/projects/conda-build/en/stable/resources/make-relocatable.html).

Prefer [`${SLOT.SlotName}`](#slotslotname) unless an absolute path is required.

#### ${SLOTNAME.SlotName}

The name of the slot `SlotName` after expanding any context variables.

`SlotName` is a period-separated list of "*parts*":

- lowercase context variables, and/or
- capitalized namespace terms

The namespace terms and context variables can be combined in any order.

For example, `${SLOTNAME.Release.execution_abi}` has two parts:

1. `Release` is a namespace term.
2. `execution_abi` is a context variable. Context variables are expanded according to the table below.

| Context Variable | Example Value    | Description                                                                    |
| ---------------- | ---------------- | ------------------------------------------------------------------------------ |
| execution_abi    | Windows_x86_64   | The ABI for the [execution platform](https://bazel.build/extending/platforms). |
| target_abi       | Windows_x86      | The ABI of the generated executables and libraries.                            |
|                  |                  | Defaults to the execution ABI.                                                 |
| request          | Release.Agnostic | The *request slot* from the `-s REQUEST_SLOT` command line option              |
|                  |                  | (ex. `get-object MODULE@VERSION -s Release.Agnostic`)                          |

If the command has no request slot (ex. `get-bundle MODULE@VERSION`) and you use the `request` context variable, an error is reported.

In the example we have been using, `${SLOTNAME.Release.execution_abi}` will resolve to `Release.Windows_x86_64` if the build is executing on a Windows 64-bit [execution platform](https://bazel.build/extending/platforms).

The list of `execution_abi` values is updated periodically from the `t_abi` enumeration (sum type) values in the [dkml-c-probe](https://github.com/diskuv/dkml-c-probe) project.
At the time of writing, the list is:

- `Android_arm32v7a`
- `Android_arm64v8a`
- `Android_x86`
- `Android_x86_64`
- `Darwin_arm64`
- `Darwin_x86_64`
- `DragonFly_x86_64`
- `FreeBSD_x86_64`
- `Linux_arm32v6`
- `Linux_arm32v7`
- `Linux_arm64`
- `Linux_arm64_musl`
- `Linux_x86`
- `Linux_x86_64`
- `Linux_x86_64_musl`
- `Linux_x86_musl`
- `NetBSD_x86_64`
- `OpenBSD_x86_64`
- `Windows_arm32`
- `Windows_arm64`
- `Windows_x86`
- `Windows_x86_64`

A musl (Alpine) host self-identifies its execution ABI as `Linux_x86_64_musl`. When a build resolves an object at a slot containing the execution ABI (for example `-s Release.execution_abi`) and the object does not publish a `Linux_x86_64_musl` slot, the resolver retries at the bare `Release.Linux_x86_64` slot and notes the substitution on stderr: the bare Linux execution tools are statically linked, so a musl host runs them natively. The fallback applies only to the execution ABI (a `target_abi` request never degrades), and a miss at both slots reports the original `Linux_x86_64_musl` slot. Import and `get-bundle` host-slot requests use the bare `Release.Linux_x86_64` slot directly.

#### Object Slot ABI

An object slot's ABI conventionally falls into one of two categories:

- A build-time tool whose output does not depend on the target uses the execution ABI slot (`Release.execution_abi`), the ABI of the [execution platform](https://bazel.build/extending/platforms) it runs on.
- A target artifact, like a compiled executable, a shared library, or a whole runtime tree that ships to the target, uses the target ABI slot (`Release.target_abi`).

An interpreted runtime that performs native compilation during the build is both a build-time tool and a source of target artifacts. Conventionally, that interpreted runtime would use a target ABI slot like `Release.target_abi` and run with an emulator like Rosetta on macOS, QEMU on Linux, or WOW64 on Windows. The interpreter runs on the execution host and its native compilation produces artifacts for the target. For example, a cross-build of a Python distribution on a `Darwin_arm64` host for a `Darwin_x86_64` target fetches the `Darwin_x86_64` CPython runtime and runs its `python3` under Rosetta to install wheels. A wheel with a C extension, like one built through setuptools, compiles the extension for the target ABI `Darwin_x86_64`.

#### ${/} directory separator

The directory separator. Except for one edge case (below), it is always `/` even on Windows. That is, form commands can assume the `/` separator, which can simplify function code when the function interacts with MSYS2.

There is a special edge case for the build system in install mode: the build system in install mode will set the directory separator to `\` on Windows and `/` on Unix.
This allows installation to canonicalized UNC paths for Windows like the remote file `\\Server2\Share\Test\Foo.txt` or [long-path capable](https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry) `\\?\C:\Test\Foo.txt`.

#### ${.exe.execution}

The executable suffix for the [execution platform](https://bazel.build/extending/platforms).

On a Windows execution platform it is `.exe`; otherwise it is empty.

#### ${.exe.target}

The executable suffix for the [target platform](https://bazel.build/extending/platforms).

When the build system is running in "install" mode, the executable suffix will be:

- `.exe` on Windows
- `` on Unix

When the build system is running normally, the executable suffix will be `.exe` even on Unix. This behavior:

- reduces the need for seperate `.precommands` for Windows and Unix, and separate `.function.args`
- is a performance and space optimization since a common executable suffix increases the chances that non-ABI specific artifacts share the same hash across Windows and Unix.

#### ${HOME}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the home directory to be the OS-specific home directory for the install end-user.

#### ${CACHE}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the cache directory to be the OS-specific cache directory (ex. `Temporary Internet Files` on Windows, the XDG-compliant cache directory in Unix).

#### ${DATA}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the data directory to be the OS-specific data directory (ex. `LocalAppData` on Windows, the XDG-compliant data directory in Unix).

#### ${CONFIG}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the config directory to be the OS-specific config directory (ex. `LocalAppData` on Windows, the XDG-compliant config directory in Unix).

#### ${STATE}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the state directory to be the OS-specific data directory (ex. `LocalAppData` on Windows, the XDG-compliant state directory in Unix).

#### ${RUNTIME}

A temporary directory for the form function.

There is a special edge case for the build system in install mode: the build system in install mode will set the runtime directory to be the OS-specific data directory (ex. `LocalAppData` on Windows, the XDG-compliant runtime directory in Unix).

#### ${HOMEABS}, ${CACHEABS}, ${DATAABS}, ${CONFIGABS}, ${STATEABS}, ${RUNTIMEABS}

The absolute path to [${HOME}](#home), [${CACHE}](#cache), [${DATA}](#data), [${CONFIG}](#config), [${STATE}](#state) or [${RUNTIME}](#runtime).

### Execution Constraints

Form functions may have execution constraints like the following that restricts the function to only run on a Windows execution platform:

```json
        "execution": [
          {
            "name": "OSFamily",
            "value": "windows"
          }
        ]
```

These names and values follow the *Platform Lexicon* defined by the Bazel build tool: <https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/platform.md/>.

> Tip: be careful! If you specify the `ISA` property pair, it may be ignored today but recognized in a future version.

**If** there is a need to constrain the execution, it is conventional to use a lookup table to map slots to execution values.
The use of `get-asset` and the `$(...)` subshell will be explained in later sections. But for now, here is the convention
that will restrict the execution of each of your slots to a specific OSFamily:

```json
  "forms": [
    {
      // ...
      "function": {
        "execution": [
          {
            "name": "OSFamily",
            "value": "$(get-asset SomeWhere_Std.Lookup@1.0.0 -p osfamily -m ./${SLOTNAME.request})"
          }
        ],
        // ...
      },
      // ...
    }
  ],
  "bundles": [
    {
      "id": "SomeWhere_Std.Lookup@1.0.0",
      "listing": {
        "origins": [
          {
            "name": "table-pwsh",
            "mirrors": ["some-project-path/table"]
          }
        ]
      },
      "assets": [
        {
          "path": "osfamily",
          "origin": "table-pwsh",
          "size": 13980,
          "checksum": {
            "sha256": "5b59b7adbd5d4ccf24c51ac26e144754f0089f8c94de3b44a8bcf60ea81fb029"
          }
        }
      ]
    }
  ]
```

and then define in your project a folder `some-project-path/table` with files having the names of your slots:

- `Release.Windows_x86_64` - the contents should be the OSFamily value `windows`
- `Release.Darwin_arm64` - the contents should be `macos`
- etc.

Finally, use the `--autofix` flag to set the `size` and `checksum` fields automatically.

This is slightly complex on purpose ... the use of execution constraints is discouraged. Instead, try to make your package cross-platform.

Tip: In the future, remote execution platforms will be supported using most of the same mechanisms as [distributions](#distributions).

#### OSFamily

One of the following: `windows`, `macos`, `linux` (includes Android), `netbsd`, `freebsd` (includes DragonFly BSD), and `openbsd`.

### Precommands

The `precommands` are a **set** of commands run *before* an form's `function`. It is not a sequence of commands since you
cannot make assumptions about the order of the precommands until the `sequential` flag is enabled.

The following optimizations are allowed:

- Precommands may be run in parallel unless the `"precommands": { "sequential": true, ... }` flag is enabled.

Precommands *will* be skipped if the requested slot does not match the precommand output slot.
For example, let's say you issue the command `get-object THE_ID -s Release.Agnostic`.
Let's also say `THE_ID` object has two precommands:

1. `get-asset ... -f ${SLOT.Release.Agnostic}`
2. `get-object ... -d ${SLOT.Something.Else}`

Then the second precommand may be skipped because the requested slot `Release.Agnostic` does not match `Something.Else`.

### Environment Modifications

The names in the environment follow the [POSIX specification](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html), except on Windows the environment names are folded to be case-insensitive.
The rules are:

- The character set for the environment names is the [Portable Character Set](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html#tagtcjh_3) except `=` (`<U003D>`) and `NUL`.
- The environment name can't start with a digit
- MlFront restricts more strictly than POSIX for whitespace: the only whitespace accepted are spaces (`<U0020>`). Linefeeds and other non-space whitespace are not accepted.
- MlFront restricts more strictly than POSIX for control characters: the `BEL` (`<U0007>`) is not accepted.

The values in the environment are encoded as UTF-8 and may contain [variables](#form-variables).

Because the first character of an environment modification unambiguously determines the type of modification, like `+` in `+NAME=VALUE` or `-` in `-NAME`, no escaping of environment names or values is required.

#### +NAME=VALUE

Add or set the environment variable with name `NAME` to `VALUE`.

The `VALUE` may contain [variables](#form-variables).

You may use `NAME=` to set the environment variable to empty. It is preferable to use [-NAME](#-name) since programs have inconsistent behavior when environment variables are empty; some treat empty as an unset environment variable, while others treat empty as empty.

#### -NAME

Remove the environment variable with name `NAME`.

#### <NAME=VALUE

Prepends `VALUE` and a path separator to the environment variable with name `NAME`.
However, the path seperator (`;` or `:` on Windows or Unix, respectively) is not added if the environment variable is empty.

The `VALUE` may contain [variables](#form-variables).

For example, `<PATH=C:\Windows\system32` prepends `C:\Windows\system32;` to the PATH on Windows and prepends `C:\Windows\system32:` to the PATH on Unix.
In this example, the Unix prepending does not make sense, which is why the best practice is to use [variables](#form-variables) for the `VALUE`
like `<PATH=${CACHE}${/}bin` so the modification is portable across operating systems.

### Form Processing

#### Processing Order

The order of processing is as follows:

1. The form's subshells in the function `args` and `envmods` (if any) are executed, in parallel if supported by the build system.
2. The form's precommands are executed, in parallel if supported by the build system.
3. If there is a breakpoint from the `enter-object` command, a system shell (PowerShell, bash, etc.) is invoked.
4. Each form function command line (`commands`) is executed *unless* the requested slot does not match the command output slot.
   For example, let's say you issue the command `get-object THE_ID -s Release.Agnostic`.
   Let's also say `commands` has two commands:

   1. "get-asset", ..., "-f", "${SLOT.Release.Agnostic}"
   2. "get-object", ..., "-d", "${SLOT.Something.Else}"

   Then the second command may be skipped because the requested slot `Release.Agnostic` does not match `Something.Else`.

5. The form's output files are verified to exist.
6. The [`${SLOT.slotname}`](#slotslotname) that are part of the form's arguments and precommands are made available to other forms.

#### Execution Step Cacheing

Each precommand and each function command is a [build task](#task-model). A task is skipped (that is, its previous result is reused) when both of the following are unchanged from the previous run:

1. Task key: the command's argument values
2. Dependencies: the `$(...)` subshells. If any subshell dependency produces a different value (because the asset or object content changed), the dependencies differ and the step re-executes.

> [!TIP]
> Avoid `precommands` that place values in intermediate directories rather than `${SLOT.*}` directories. Instead use subshells. If you use intermediate directories the intermediate files won't be updated or invalidated.
>
> ```json
> // WRONG: precommand + cp = cached
> "precommands": { "private": ["get-asset MOD@VER -p path/to/script.py -f script.py"] },
> "commands": [
>   ["coreutils", "cp", "script.py", "${SLOT.request}/script.py"],
>   ["python3", "script.py"]
> ]
> 
> // CORRECT: subshell creates dependency, re-runs when asset is updated or invalidated
> "commands": [
>   ["python3", "$(get-asset MOD@VER -p path/to/script.py -f script.py)"]
> ]
> ```
>
> Alternatively, each build system implementation has a way to update workspace asset checksums and invalidate values.

#### Windows command-line quoting

On Windows, `function.commands` are rendered into one
`lpCommandLine` string parameter to the Windows API function call
[CreateProcessW](https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw).
The string is assembled by `cmd.exe` (Windows Batch) quoting of each
`function.commands` command (ie. commands with spaces will be surrounded with double quotes, etc.),
then concatenating them all with spaces.

#### Windows `--cmd.exe` special form

When a form needs `cmd.exe /c`, prefer the special `function.commands` form
whose exact shape is `["--cmd.exe", "/c", "<VSL-quoted string>"]`. Spaces added for clarity:

```json
{
  "function": {
    "commands": [
      "--cmd.exe",
      "/c",
      "\"  echo hi > `\"${SLOT.Release.Agnostic}\\some-file.txt`\"   \""
    ]
  }
}
```

Stripped of the [VSL quoting](#vsl-lexical-rules) the unquoted argument to `/c` is:

```text
   echo hi > "${SLOT.Release.Agnostic}\some-file.txt"   
```

That expands to being the `lpCommandLine` parameter to the Windows API function call [CreateProcessW](https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw):

```text
cmd.exe /c "   echo hi > "...\somewhere\Release.Agnostic\some-file.txt"   "
```

## Objects

An object is a BLOB, which is a sequence of bytes. The object may be categorized by how the object comes to exist:

- a "generated" object created by a [form](#forms)
- anything else is an "input" object. For example, a file in your project may be an "input" object.

But to re-iterate: There is no concept of an object being a "file" or a "directory".
The object is just a sequence of bytes.

In both cases the build system treats the objects as immutable,
and the objects may be cached and/or persisted to disk whenever necessary.

When a value shell command is being run (described in the upcoming [Value Shell Language](#value-shell-language-vsl) section),
an object is made available on disk. At this time an object is "realized" into either a file or a directory.
That is the subject of the next [Saving and Loading Objects](#saving-and-loading-objects) section.

> Design Note: Why blur the distinction between files and directories?
> These objects are meant to be *cloud-friendly* so they need to
> have a canonical representation on cloud value stores like AWS S3. We don't need strict typing everywhere!
> And using a compressed archive means accessing the multiple
> outputs of a form function is quite straightforward; in contrast, other build systems expose the user to added complexity
> (confer: [make: Handling Tools that Produce Many Outputs](https://www.gnu.org/software/automake/manual/html_node/Multiple-Outputs.html)).

### Saving and Loading Objects

When a value shell command reads an immutable object and saves it to a file (ex.
[get-object -f FILE](#get-object-moduleversion--s-request_slot--f-file---d-dir)),
the bytes of the immutable object are copied directly to the file.

When a value shell command reads an immutable object and saves it to a directory (ex.
[get-object -d DIR](#get-object-moduleversion--s-request_slot--f-file---d-dir)),
the bytes of the immutable object are:

- *when the bytes have a zip file header* uncompressed and unzipped into the directory
- *when the bytes do not have a zip file header* copied into the directory in a file named `OBJECT`

When a value shell command saves a file as an immutable object, the file's bytes are saved as-is.

When a value shell command saves a directory as an immutable object, the directory is zipped and the zip archive bytes are saved.
The standards of [Zip Archive Reproducibility](#zip-archive-reproducibility) will be followed.

That sounds inefficient, but the build system is allowed to optimize a set of value shell commands.
For example, if one shell command saves output into a directory,
and a second shell command reads data from created by the first shell command,
the build system can give the second shell command a symlink to the first directory
**without** using a zip archive as an intermediate artifact.

### Object Slots

Each object has one or more slots. Each slot is a container for the object's files.

There are no built-in slots. However, `Release.Agnostic` is the conventional slot for files that are ABI-agnostic.

The names of the slots are period-separated "MlFront standard namespace terms". Each of these terms:

- are drawn from the character set `'A' .. 'Z' | 'a' .. 'z' | '0' .. '9' | '_'`
- must start with a capital letter
- must not contain a double underscore (`__`)
- must not be a MlFront library identifier (ie. a double camel cased string followed by an underscore and another camel cased string, like `XyzAbc_Def`)

## Values

### Value Shell Language (VSL)

**All encodings of VSL are UTF-8 unless explicitly noted as different.**

There is a POSIX shell / PowerShell styled language to query for objects and assets.
For example, the "command":

```sh
get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe
```

will get the object with the id `OurStd_Std.Build.Clang@1.0.0` and place it in the `clang.exe` file.

There are two ways to run these shell commands:

1. Directly from the command line. For example, `${dk_build_system} get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe` where `${dk_build_system}` could be the `dk0` reference implementation.
2. Embedded as "precommands" in a values file. For example,

   ```json
   { // ...
    "precommands": {
      "private": [
        "get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe"
      ]
    },
    // ...
   }
   ```

All commands have a output path (ex. `-f echo.exe`). Most command have two forms:

- `-f FILE` (ex. `-f echo.exe`)
- `-d DIR` (ex. `-d target`)

but some commands may only have the `-d DIR` directory output.

The best practice for relative paths is to use the forward slash (`/`) as directory separator in the output paths for readability (no escaping in JSON) and portability (backslashes don't work on Unix).
However, when the path is an absolute directory, use the native format, including UNC paths on Windows (ex. `\\Server2\Share\Test\Foo.txt`).

For security, the commands may be evaluated in a sandbox or a chroot environment. Do not use `..` path segments or they may fail to resolve in sandboxes.

### VSL Lexical Rules

A value shell command is a **command line** that is split into **words**.

In a JSONC (JSON with comments) values file, each precommand is a command line:

```json
"precommands": {
  "private": [
    // command line 1
    "get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe",
    "...",
    // command line N
    "get-object OurStd_Std.Build.GCC@1.0.0 -s Release.Agnostic -f gcc.c"
  ]
}
```

The `get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe` command line has the words `get-object`, `OurStd_Std.Build.Clang@1.0.0`, `-s`, `Release.Agnostic`, `-f`, and `clang.exe`.

Words are directly used in other places in the values file:

```json
"function": {
    "commands": [
      // word 1
      "sh",
      // word 2
      "-c",
      // word 3
      "\"find . > ${SLOT.Release.Agnostic}/some-file\""
    ],
    "envmods": [
      // word 4: ${CACHE}/dkcoder
      "+DKCODER_CTX_CACHE_DIR=${CACHE}/dkcoder"
    ]
}
```

so the individual words can be assembled into a command line.

Each function argument in `args` and each environment modification value in `envmods` must be one value shell word.
In word 3 we used double quotes to squash several words into one shell word. In the next section the different ways to form words are explained.

#### Types of Words

The `get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -f clang.exe` from the previous section had the words:

- `get-object`
- `OurStd_Std.Build.Clang@1.0.0`
- `-s`
- `Release.Agnostic`
- `-f`
- `clang.exe`

These are examples of **bare words**; that is, words without any surrounding quotes.

Each bare word is one or more of the following *components without any interleaving spaces*:

- literals like `clang.exe`
- variables like `${SLOT.Release.Agnostic}` that get expanded to the **string value** of the named variable from the next section [Variables available in VSL](#variables-available-in-vsl)
- subshells like `$(get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -d :)` that get expanded to a **temporary file or directory** output of the subshell expression

An example of a single bare word that has all three types of components is:

```sh
$(get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -d :)/clang${.exe}
```

which is compromised of a subshell which could expand to `some/dir/clang-1.0.0/bin`, the literal `/clang` and the variable `${.exe}` which could expand to `.exe`. That is, the bare word could expand to `some/dir/clang-1.0.0/bin/clang.exe`. No quotations were required.

The lexical rules allow for a subshell to be nested in another subshell, although nested subshells should not be required until form parameters are added. For completeness, the bare word `$(get-asset MyAssets_Std.Bundle@1.0.0 -p $(get-object OurStd_Std.Build.Clang@1.0.0 -s Release.Agnostic -d :) -d :)` is valid, although nonsensical.

VSL words can also have quotes surrounding them so that spaces can be handled. The three types of words are:

1. Bare words (we've already covered these) with characters that are not whitespace, single quotes (`'`) or double quotes (`"`).
2. **Single-quoted** words like `'C:\My Documents'`
3. **Double-quoted** words like `"${CONFIG}\Floor Plans\Master Bedroom.rvt"`

Backticks (`` ` ``) are ordinary literal characters in bare words. They do **not** escape the following character unless they are inside a double-quoted word.

Single-quoted words (`'...'`) evaluate *literally* to the text inside the single quotes, including any whitespace and newlines.

Double-quoted words (`"..."`) squash many *bare* words into a single word by:

- evaluating each "inner bare word" (each bare word inside the double-quotes) using the rules above. However, the inner bare words can contain single-quote (`'`) characters which are treated like any ordinary character.
- keeping the whitespace *between* each inner bare words

The inner bare words and the whitespace between the inner words are concatenated into a single word.

Within double-quotes:

- single-quote (`'`) characters are ordinary literal characters
- literal double-quote (`"`) characters must be escaped using the `` ` `` (backtick, aka. grave accent) as the escape character
- literal backtick (`` ` ``) characters must be escaped using the `` ` `` (backtick, aka. grave accent) as the escape character

> *Historical reasoning: Backticks were chosen for compatibility with Windows paths and familiarity with PowerShell; carets `^` were rejected since in Windows Batch the caret has complex rules.*

See [ESCAPING.md](ESCAPING.md) for worked examples of simple and complex VSL escaping.

### Variables available in VSL

- `${SLOT.slotname}`
- `${/}`
- `${SRC}`
- `${HOME}`
- `${DATA}`
- `${CONFIG}`
- `${STATE}`
- `${RUNTIME}`

### remote UI_MODULE@VERSION COMMAND

`remote UI_MODULE@VERSION [REMOTE_OPTION=VALUE...] [REPOSITORY] COMMAND...`

Run a local value shell command, or a top-level `test`, `lua`, or `run`
command, on a remote execution engine.

If `UI_MODULE@VERSION` is not fully qualified, short forms `<NamespaceTerm>*@<Ver>` are expanded to `CommonsBase_Remote.<NamespaceTerm>*@<Ver>` like:

```text
remote GitHub@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help
```

into:

```text
remote CommonsBase_Remote.GitHub@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help
```

and short forms `<Vendor><Qualifier>.<NamespaceTerm>*@<Ver>` are expanded to `<Vendor><Qualifier>_Remote.<NamespaceTerm>*@<Ver>` like:

```text
remote BuildBuddy.Cloud@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help
```

into:

```text
remote BuildBuddy_Remote.Cloud@0.1.0 run-object MODULE@VERSION -s REQUEST_SLOT -m ./tool -- --help
```

The `REPOSITORY` argument is optional when the remote rule can infer or has
configured a repository. GitHub repositories are written in a form accepted by
the remote rule, such as `github.com/OWNER/REPO`.

Leading `REMOTE_OPTION=VALUE` words are passed to the remote rule before the
optional repository and inner command are parsed.

> [!TIP]
> The CommonsBase_Remote.GitHub rule recognizes the following options (with
> defaults):
>
> - `workspace=dk.u`
> - `sessions=4`
> - `retention=8`
> - `create_repo=false`
> - `dry_run=false`
> - `execution_abi=Windows_x86_64`
> - `target_abi=` (defaults to the execution ABI)
> - `linux_image=` (defaults to the newest active dated `quay.io/pypa/manylinux_2_28_x86_64` tag, resolved at submit)
>
> The GitHub remote rule uses a workspace script, defaulting to `dk.u`, and a
> maximum session count, defaulting to 4. If `create_repo=true`, the rule
> creates a **private** GitHub repository when the requested repository
> does not already exist. It stores the age recipient and its OpenBSD
> signify public signature in GitHub Actions repository variables, stores the
> age secret key in a GitHub secret, and transfers staged local assets through
> GitHub prereleases. Each polling iteration for workflow runs or prereleases is
> reported as progress. If a `git fetch` followed by `git rebase` updates a
> session branch, the session is treated as contended; the implementation waits
> 30 seconds and tries another session.
>
> The `execution_abi=` option selects the GitHub-hosted execution platform:
> `Windows_x86_64` (the default; `runs-on: windows-latest`), `Darwin_arm64`
> (`macos-latest`) or `Linux_x86_64` (`ubuntu-latest` in a
> `quay.io/pypa/manylinux_2_28_x86_64` container, glibc 2.28). The
> `target_abi=` option selects the ABI of the generated executables and
> libraries and defaults to the execution ABI. When only `target_abi=` is
> given, the execution ABI is derived from it: `Windows_x86_64`, `Windows_x86`
> and `Windows_arm64` build on `Windows_x86_64`; `Darwin_arm64` and
> `Darwin_x86_64` build on `Darwin_arm64`; `Linux_x86_64`, `Linux_x86` and
> `Linux_x86_64_musl` build on `Linux_x86_64`. When the target differs from
> the execution ABI, `--target-abi TARGET_ABI` is prepended to the audited
> argument vector so the remote `dk0` builds for the requested target. A
> `target_abi=` request is exact and never [degrades to the bare Linux
> slot](#slotnameslotname).
>
> The runner labels and container images match the dk package distribute
> workflows generated by [`prepare-version --ci github`](#distributions), so
> remote results are ABI-compatible with distributed dk packages. The
> `Linux_x86_64_musl` target builds on the same glibc manylinux container with
> the `Linux_x86_64` execution ABI; the target slot's musl cross toolchain
> comes from the package's own values. The Linux container image is resolved
> at submit time to the newest active dated `manylinux_2_28_x86_64` tag on
> quay.io, falling back to the maintained `latest` tag when the tag listing is
> unreachable; `linux_image=IMAGE` selects a specific image.

The words after the remote options and optional repository are parsed as the
inner command. For value shell commands, the inner command uses the same VSL
variable and subshell expansion as a direct local invocation. In particular,
`run-object` and `run-asset` command-line arguments may contain variables and
subshells such as `$(get-object ...)`. The audited command stored in the
session branch is a single-line POSIX shell command, so the GitHub workflow
(etc.) can pass the exact argument vector to the build tool.

Before submitting work, an implementation prints the resolved inner
command in value shell syntax, using two spaces between top-level terms. This
is intentionally different from shell syntax and is shown so the user can see
exactly what command is being sent to the remote execution engine.

Remote value shell commands must return results in the same observable shape
as local commands. For example, `run-object` and `run-asset` still produce the
same captured stdout/stderr value shape, and the local implementation imports
the returned valuestore/tracestore data before presenting output. Remote
`test`, `lua`, and `run` return the remote exit status and display the remote
stdout, stderr, and log output.

> [!NOTE]
> For remote execution, CommonsBase_Remote.GitHub creates exactly one
> [object value](#objects). That object contains a Cap'n Proto metadata
> file plus payload files: stdout/stderr text files or a zip holding directory
> contents. The single object value and its single leaf trace (that is, no
> dependency trace) are then exported and re-imported with the same
> orchestration as `distribute` and `import-github-l2`.
>
> To avoid a single leaf trace from interfering with conventional traces,
> a synthetic key is used. Only the `get-asset` and `get-bundle` commands,
> the commands that naturally have no dependencies, do not use synthetic keys.
>
> Local `-f FILE` and `-d DIR` extraction happens after import.

### get-object MODULE@VERSION -s REQUEST_SLOT (-f FILE | -d DIR/)

Get the contents of the slot `REQUEST_SLOT` for the object uniquely identified by `MODULE@VERSION`.

| Option      | Description                                                                   |
| ----------- | ----------------------------------------------------------------------------- |
| `-f FILE`   | Place object in `FILE`                                                        |
| `-d DIR/`   | The object must be a zip archive, and its contents are extracted into `DIR/`. |
| `-n STRIP`  | See [Option: [-n STRIP]](#option--n-strip)                                    |
| `-m MEMBER` | See [Option: [-m MEMBER](#option--m-member)]                                  |
| `-x GLOB`   | Exclude globbed files from `-d DIR/`. May be repeated.                        |
| `-e GLOB`   | Make globbed files executable in `-d DIR/` and `-f FILE`. May be repeated.    |

See [Options: -f FILE, -d DIR, -x GLOB, and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob) for output behavior.

The object `ID` implicitly or explicitly contains build metadata; see [ID with Build Metadata](#object-id-with-build-metadata).

### run-object MODULE@VERSION -s REQUEST_SLOT (-c COMMAND | -m MEMBER)

Run a command from the object slot `REQUEST_SLOT` for the object uniquely identified by `MODULE@VERSION`.

| Option         | Description                                                                                                  |
| -------------- | ------------------------------------------------------------------------------------------------------------ |
| `-c COMMAND`   | Run the strictly relative `COMMAND` inside the anonymous directory tree produced by `get-object -d :`.       |
| `-m MEMBER`    | Extract only `MEMBER`, normalize it as a file path, and use that normalized path as the command to run.      |
| `-n STRIP`     | See [Option: [-n STRIP]](#option--n-strip). Only valid with `-c COMMAND`.                                    |
| `-x GLOB`      | Exclude globbed files from the anonymous directory. May be repeated.                                         |
| `-e GLOB`      | Make globbed files executable in the anonymous directory. May be repeated.                                   |
| `-- [ARGS...]` | Interpret `ARGS` as a value shell command line with variables and subshells, like `function.commands` words. |

The command is executed with the same current working directory model as [run-function](#run-function-moduleversion--f-file---d-dir----cli_form_doc), but the executable path is always resolved relative to the anonymous command directory. `COMMAND` and normalized `MEMBER` must be [strictly relative file paths](#strictly-relative-path).

The standard output and standard error streams are captured in separate files but stored as a single `x<ID>` value in the valuestore.

The output and error stream files are capped at 16777211 bytes, which is the
lower of the approximate Cap'n Proto `Text` payload limit of 512 MB and the
maximum string length on 32-bit platforms.

> [!TIP]
> For larger content, redirect the command's standard output or standard error to form rule output files using shell redirection or a log capture executable.

### run-function MODULE@VERSION (-f FILE | -d DIR/) -- CLI_FORM_DOC

Run the [Lua function rule](#introduction-to-custom-lua-rules) uniquely identified by `MODULE@VERSION` using the JSON constructed from `CLI_FORM_DOC`.

| Option      | Description                                                                   |
| ----------- | ----------------------------------------------------------------------------- |
| `-f FILE`   | Place object in `FILE`                                                        |
| `-d DIR/`   | The object must be a zip archive, and its contents are extracted into `DIR/`. |
| `-n STRIP`  | See [Option: [-n STRIP]](#option--n-strip)                                    |
| `-m MEMBER` | See [Option: [-m MEMBER](#option--m-member)]                                  |
| `-x GLOB`   | Exclude globbed files from `-d DIR/`. May be repeated.                        |
| `-e GLOB`   | Make globbed files executable in `-d DIR/` and `-f FILE`. May be repeated.    |

If no `-f` or `-d` option is given, the object is dumped to the standard output (ie. your console).

See [Options: -f FILE, -d DIR, -x GLOB, and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob) for output behavior.

See [Form Document](#form-document) for the `CLI_FORM_DOC` form parameters. If there are none, the `-- CLI_FORM_DOC` can be left out.

The object `ID` implicitly or explicitly contains build metadata; see [ID with Build Metadata](#object-id-with-build-metadata).

### enter-object MODULE@VERSION -s REQUEST_SLOT -- CLI_FORM_DOC

Enter a shell like PowerShell or `/bin/bash` that has the contents of the slot `REQUEST_SLOT` for the object uniquely identified by identifier `ID`.

The shell is meant only for debugging problems, and may not appear if the object `ID` has been successfully built.

See [Form Document](#form-document) for form parameters. If there are none, the `-- CLI_FORM_DOC` can be left out.

The object `MODULE@VERSION` implicitly or explicitly contains build metadata; see [ID with Build Metadata](#object-id-with-build-metadata).

### merge-object MODULE@VERSION -s REQUEST_SLOT (-f FILE | -d DIR/)

Build the object uniquely identified by `MODULE@VERSION` and merge the contents of the slot `REQUEST_SLOT` into the requested output.

| Option      | Description                                                                                             |
| ----------- | ------------------------------------------------------------------------------------------------------- |
| `-f FILE`   | Place merged object in `FILE`                                                                           |
| `-d DIR/`   | Merge contents of the zip archive into the existing directory `DIR/`. The object must be a zip archive. |
| `-n STRIP`  | See [Option: [-n STRIP]](#option--n-strip)                                                              |
| `-m MEMBER` | See [Option: [-m MEMBER](#option--m-member)]                                                            |
| `-x GLOB`   | Exclude globbed files from `-d DIR/`. May be repeated.                                                  |
| `-e GLOB`   | Make globbed files executable in `-d DIR/` and `-f FILE`. May be repeated.                              |

**More than one `merge-object` can use the same output directory `DIR`**.

See [Options: -f FILE, -d DIR, -x GLOB, and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob) for output behavior.

The object `MODULE@VERSION` implicitly or explicitly contains build metadata; see [ID with Build Metadata](#object-id-with-build-metadata).

### get-asset MODULE@VERSION FILE_PATH (-f FILE | -d DIR/)

Get the contents of the asset at `FILE_PATH` for the bundle `MODULE@VERSION`.

| Option      | Description                                                                  |
| ----------- | ---------------------------------------------------------------------------- |
| `-f FILE`   | Place asset in `FILE`                                                        |
| `-d DIR/`   | The asset must be a zip archive, and its contents are extracted into `DIR/`. |
| `-n STRIP`  | See [Option: [-n STRIP]](#option--n-strip)                                   |
| `-m MEMBER` | See [Option: [-m MEMBER](#option--m-member)]                                 |
| `-x GLOB`   | Exclude globbed files from `-d DIR/`. May be repeated.                       |
| `-e GLOB`   | Make globbed files executable in `-d DIR/` and `-f FILE`. May be repeated.   |

See [Options: -f FILE, -d DIR, -x GLOB, and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob) for output behavior.

### run-asset MODULE@VERSION FILE_PATH (-c COMMAND | -m MEMBER)

Run a command from the asset `FILE_PATH` in the bundle `MODULE@VERSION`.

| Option         | Description                                                                                                  |
| -------------- | ------------------------------------------------------------------------------------------------------------ |
| `-c COMMAND`   | Run the strictly relative `COMMAND` inside the anonymous directory tree produced by `get-asset -d :`.        |
| `-m MEMBER`    | Extract only `MEMBER`, normalize it as a file path, and use that normalized path as the command to run.      |
| `-n STRIP`     | See [Option: [-n STRIP]](#option--n-strip). Only valid with `-c COMMAND`.                                    |
| `-x GLOB`      | Exclude globbed files from the anonymous directory. May be repeated.                                         |
| `-e GLOB`      | Make globbed files executable in the anonymous directory. May be repeated.                                   |
| `-- [ARGS...]` | Interpret `ARGS` as a value shell command line with variables and subshells, like `function.commands` words. |

The command is executed with the same current working directory model as [run-function](#run-function-moduleversion--f-file---d-dir----cli_form_doc), but the executable path is always resolved relative to the anonymous command directory. `COMMAND` and normalized `MEMBER` must be [strictly relative file paths](#strictly-relative-path).

The standard output and standard error streams are captured in separate files but stored as a single `x<ID>` value in the valuestore.

The output and error stream files are capped at 16777211 bytes, which is the
lower of the approximate Cap'n Proto `Text` payload limit of 512 MB and the
maximum string length on 32-bit platforms.

> [!TIP]
> For larger content, redirect the command's standard output or standard error to form rule output files using shell redirection or a log capture executable.

### get-bundle MODULE@VERSION (-f FILE | -d DIR/)

Get the archive file for the bundle `MODULE@VERSION`.

| Option     | Description                                                                         |
| ---------- | ----------------------------------------------------------------------------------- |
| `-f FILE`  | Place bundle in `FILE`. `FILE` will be a zip archive with **all** the bundle files. |
| `-d DIR/`  | **All** the bundle files will be extracted into `DIR/`.                             |
| `-n STRIP` | See [Option: [-n STRIP]](#option--n-strip)                                          |
| `-x GLOB`  | Exclude globbed files from `-d DIR/`. May be repeated.                              |
| `-e GLOB`  | Make globbed files executable in `-d DIR/` and `-f FILE`. May be repeated.          |

See [Options: -f FILE, -d DIR, -x GLOB, and -e GLOB](#options--f-file-and--d-dir-and--x-glob-and--e-glob) for output behavior.

*What about the `-m MEMBER` option?*

Use [get-asset](#get-asset-moduleversion-file_path--f-file---d-dir) to get a specific asset.
Having a `-m MEMBER` option would be equivalent but redundant and slightly confusing since
not much about assets implies they are stored as archives.

### Options: -f FILE and -d DIR and -x GLOB and -e GLOB

Each output file (a single file if `-f FILE` or zero or more if `-d FILE`) has a *normalized entry* name.

Any `-x GLOB` or `-e GLOB` option operates on a glob matched against the normalized entry name:

- For `-f FILE` the normalized entry name is the basename of `FILE`.
- For `-d DIR` the normalized entry name is zip archive entry name, with any redundant `./` and `somedir/../` path segments removed, and leading segments removed if `-n STRIP` was used.

For example, the options `-e '**/*.exe' -e 'bin/*' -x 'doc/**'` will cause any `.exe` output file to be made *executable*,
and any files in the output `bin/` directory to be made executable, and to not output any entries (including deeply nested entries)
to the `doc/` directory.

When an output file is made executable:

- On Unix the file's executable bit will be set (ie. `chmod a+x OUTPUTFILE`)
- On macOS, if the output file is a Mach-O file that needs a code signature *and* the output file is **not* within an Apple app or bundle, then the file will be code signed with the machine signature (ie. `/usr/bin/codesign --sign - --force OUTPUTFILE`). Any output file within a `*.app/Contents/**` file tree or `*.bundle/Contents/**` file tree (case-insensitive) is considered to be within an Apple app or bundle. For example, if the output file were `t/x/y/SomeApp.app/Contents/bin/somefile` it would not be codesigned because `SomeApp` is presumed to be signed already within an existing or future `x/y/SomeApp.app/Contents/_CodeSignature/`.

No command may write to the same output file. Specifically:

- It is an error to have more than one `get-object` or `get-bundle` or `get-asset` or `resume-object` or `merge-object` use the same `FILE`.
- It is an error to have more than one `get-object` or `get-bundle` or `get-asset` or `resume-object` use the same `DIR` or otherwise overlap the same `DIR`. Overlapping means one command can't write to the subdirectory of another command's `DIR`.

Use [`merge-object`](#merge-
--s-request_slot--f-file---d-dir----cli_form_doc) when you want to write into the same directory.
Even so, no `merge-object` may extract the same file in the same output directory.

### Option: [-n STRIP]

`-n STRIP` defaults to zero.

`STRIP` is how many levels of the zip archive to strip. Many zip archives place all content under a versioned root directory:

```text
llvmorg-19.1.3-win32/
  LICENSE.txt
  README.txt
  src/
```

To leave the directory structure as-is, set `STRIP` to `0`. To strip away the top level `llvmorg-19.1.3-win32` directory, set `STRIP` to `1`.

### Option: [-m MEMBER]

Gets the zip file member from the object or asset, which must be a zip archive.

## Subshells

### subshell options

A subshell may have options for the build system that precede the subshell command.

Each option must:

- precede the subshell command (ie. precede `get-object`, etc.)
- start with a dash (`-`)
- be a single valueshell [word](#value-shell-language-vsl)

For example, `$(--path=absnative -q '-x=still one word' get-object ...)` has three options:

- `--path=absnative`
- `-q`
- `-x=still one word`

Any options that are not recognized by the build system must be ignored.

The recognized options are:

| Option         | Description                                                        |
| -------------- | ------------------------------------------------------------------ |
| `--path=STYLE` | Sets the subshell to return a path in one of the styles:           |
|                | - `rel`: default. relative path, Unix style                        |
|                | - `absnative`: absolute Windows path on Windows, Unix path on Unix |
|                | - `absunix`: absolute Unix-style path (ex. `/a/b`, `C:/a/b`) if    |
|                | possible. Unchanged if not (ex. Windows UNC paths).                |

### subshell: get-object MODULE@VERSION -s REQUEST_SLOT

Get the contents of the slot `REQUEST_SLOT` for the object uniquely identified by `MODULE@VERSION`.

| Option                  | Description                                                                                                                 |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `-f :` or `-f BASENAME` | Place object in an [anonymous file](#anonymous-files--f--or--f-basename) and return its filepath                            |
| `-d :`                  | The object must be a zip archive, and its contents are extracted into an [anonymous directory](#anonymous-directories--d-). |
| `-n STRIP`              | See [Option: [-n STRIP]](#option--n-strip)                                                                                  |
| `-m MEMBER`             | See [Option: [-m MEMBER](#option--m-member)]                                                                                |
| `-x GLOB`               | Exclude globbed files from `-d DIR/`. May be repeated.                                                                      |
| `-e GLOB`               | Make globbed files executable in `-d DIR/` and `-f ...`. May be repeated.                                                   |

If none of the `-f :`, `-f BASENAME`, or `-d :` options are specified, the contents are captured and returned with the following restrictions:

- the content may not exceed 1024 bytes
- no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
- the byte 0 (ASCII NUL) may not be in the content as a security measure

### subshell: run-object MODULE@VERSION -s REQUEST_SLOT (-c COMMAND | -m MEMBER)

Run the executable selected by `-c COMMAND` or `-m MEMBER` from the object slot
`REQUEST_SLOT` in `MODULE@VERSION`.

The `-f :`, `-f BASENAME`, and `-d :` options are accepted.

The `-x GLOB`, `-e GLOB` and `-n STRIP` options keep the
same meanings they have for the top-level `run-object` command when selecting
the command tree.

The returned content has the same restrictions as the other content-returning
subshells:

- the content may not exceed 1024 bytes
- no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
- the byte 0 (ASCII NUL) may not be in the content as a security measure

### subshell: run-function MODULE@VERSION -- CLI_FORM_DOC

Submit the JSON constructed from `CLI_FORM_DOC` to the rule uniquely identified by `MODULE@VERSION`.

| Option                  | Description                                                                                                                 |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `-f :` or `-f BASENAME` | Place object in an [anonymous file](#anonymous-files--f--or--f-basename) and return its filepath                            |
| `-d :`                  | The object must be a zip archive, and its contents are extracted into an [anonymous directory](#anonymous-directories--d-). |
| `-n STRIP`              | See [Option: [-n STRIP]](#option--n-strip)                                                                                  |
| `-m MEMBER`             | See [Option: [-m MEMBER](#option--m-member)]                                                                                |
| `-x GLOB`               | Exclude globbed files from `-d DIR/`. May be repeated.                                                                      |
| `-e GLOB`               | Make globbed files executable in `-d DIR/` and `-f ...`. May be repeated.                                                   |

See [Form Document](#form-document) for the `CLI_FORM_DOC` form parameters. If there are none, the `-- CLI_FORM_DOC` can be left out.

If none of the `-f :`, `-f BASENAME`, or `-d :` options are specified, the contents are captured and returned with the following restrictions:

- the content may not exceed 1024 bytes
- no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
- the byte 0 (ASCII NUL) may not be in the content as a security measure

### subshell: get-asset MODULE@VERSION FILE_PATH

Get the contents of the asset at `FILE_PATH` for the bundle `MODULE@VERSION`.

| Option                  | Description                                                                                                                |
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `-f :` or `-f BASENAME` | Place asset in an [anonymous file](#anonymous-files--f--or--f-basename) and return its filepath                            |
| `-d :`                  | The asset must be a zip archive, and its contents are extracted into an [anonymous directory](#anonymous-directories--d-). |
| `-n STRIP`              | See [Option: [-n STRIP]](#option--n-strip)                                                                                 |
| `-m MEMBER`             | See [Option: [-m MEMBER](#option--m-member)]                                                                               |
| `-x GLOB`               | Exclude globbed files from `-d DIR/`. May be repeated.                                                                     |
| `-e GLOB`               | Make globbed files executable in `-d DIR/` and `-f ...`. May be repeated.                                                  |

If none of the `-f :`, `-f BASENAME`, or `-d :` options are specified, the contents are captured and returned with the following restrictions:

- the content may not exceed 1024 bytes
- no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
- the byte 0 (ASCII NUL) may not be in the content as a security measure

### subshell: run-asset MODULE@VERSION FILE_PATH (-c COMMAND | -m MEMBER)

Run the executable selected by `-c COMMAND` or `-m MEMBER` from the asset at
`FILE_PATH` in `MODULE@VERSION`.

The `-f :`, `-f BASENAME`, and `-d :` options are accepted.

The `-x GLOB`, `-e GLOB` and `-n STRIP` options keep the
same meanings they have for the top-level `run-asset` command when selecting
the command tree.

The returned content has the same restrictions as the other content-returning
subshells:

- the content may not exceed 1024 bytes
- no translation is performed on the bytes (UTF-16 is not translated to UTF-8, etc.)
- the byte 0 (ASCII NUL) may not be in the content as a security measure

### Anonymous Files: `-f :` or `-f BASENAME`

Place the object or asset in an anonymous regular file and return the file path.

The file will be named `BASENAME` and placed in a directory with no other files.

If the simplified `-f :` expression is used, the file will be named `a.dat`.

The file will be made executable if one of the `-e GLOB` patterns match. The executable will:

- on Unix execution machines its executable bits are set (bitwise OR `0o111`)
- on macOS execution machines it will be codesigned if it is a Mach-O file that must be signed

Conventionally if the file is to be made executable, the `BASENAME` includes an `.exe` extension so it can run on Windows.
However the `.exe` extension is not required if the file will not run on Windows.

### Anonymous Directories: `-d :`

Place the object or asset in an anonymous directory and return the directory path.

### Object ID with Build Metadata

These rules apply to the `*-object` commands **only**:

- `get-object MODULE@VERSION ...`
- `merge-object MODULE@VERSION ...`
- `run-function MODULE@VERSION ...`
- `enter-object MODULE@VERSION ...`

The purpose of these rules is to ensure that unique builds can be uniquely and deterministically identified.

Versions can have explicit build metadata.
For example, the VERSION `1.0.0+bn-20250801235901.commit-054d5983` has the two dot-separated build metadata fields: `bn-20250801235901` and `commit-054d5983`.

If the version `VERSION` has explicit build metadata in the format `bn-*`, then the object is **locked** to that specific build number.
In the above example the object is locked to build number `20250801235901` because that is the build metadata with format `bn-*`.

When the version `VERSION` has no explicit build metadata, or the version `VERSION`'s build metadata does not include a `bn-*` field, then the first matching rule of the following rules determines what the build metadata will be:

1. If a lockfile has a build metadata reference (ex. `1.0.0` = `bn-20250801235901+commit-054d5983`), the build metadata is used.
2. The constructive trace store list of traces `key(i), dependencies(i), result(i)` is scanned. If there is a trace `i` where the version of `key(i)` matches the `VERSION` and where `result(i)` is an object value, then the build metadata of the *latest* such `key(i)` will be used.
3. The build metadata will be constructed from the `-t TIMESTAMP` command line option, with the `bn-YYYYMMDDhhmmss` format.
4. The build metadata will be `bn-20250101000000`.

Important: the system clock is never consulted.

In CI, the best practice is to use the `-t TIMESTAMP` option, and base it on either:

- the source control commit timestamp.
  - Pro: Very easy to go back to the source code.
  - Con: If you need to force a new build from existing source code, you must create an empty commit-
- a monotonically increasing build number (ex. `GITHUB_RUN_NUMBER` if you use GitHub Actions).
  - Pro: No empty commits.
  - Con: Depending on your CI provider, it may be hard to go from a build number back to the source code.

Here are some examples for using the source control commit timestamp:

```yaml
# file: .github/workflows/example-build.yml

# CI System: GitHub Actions
# Variable Name: github.event.head_commit-timestamp
- name: Build project
  env:
    # Any dk build system is fine. dk0 is the reference implementation.
    dk_build_system: dk0
  run: ${dk_build_system} -t "${{ github.event.head_commit-timestamp }}" ...
```

```yaml
# file: .gitlab-ci.yml

# CI System: GitLab CI
# Docs: https://docs.gitlab.com/ci/variables/predefined_variables/#predefined-variables
# Variable Name: CI_COMMIT_TIMESTAMP
# Variable Example: 2022-01-31T16:47:55-08:00
variables:
  # Any dk build system is fine. dk0 is the reference implementation.
  dk_build_system: dk0
job:
  script:
    - ${dk_build_system} -t "$CI_COMMIT_TIMESTAMP" ...
```

Here are some example of using a monotonically increasing build number:

```yaml
# GitHub Actions: GITHUB_RUN_NUMBER https://docs.github.com/en/actions/reference/workflows-and-actions/variables
# GitLab CI: CI_PIPELINE_IID https://docs.gitlab.com/ci/variables/predefined_variables/
# Azure Pipelines: Build.BuildId https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml

# FILLMEIN ... wait for `-n RUN_NUMBER` option to complement `-t TIMESTAMP`
# FILLMEIN ... `-n` includes leading zeroes so lexographic comparisons work
```

### JSON Files

The schema is at [etc/jsonschema/mlfront-value.json](../etc/jsonschema/mlfront-value.json).

On Windows the JSON files are expected to be terminated with LF not CRLF line endings.

The build system is resilient to CRLF line endings:

- The [values canonical id](#vci---values-canonical-id) normalizes the JSON with all carriage returns removed before calculating the canonical id
- The [values checksums](#vck---values-checksum) normalizes the JSON with all carriage returns removed before conversion to an AST

### JSON Canonicalization

The form is reconstructed as JSON exactly with the following keys (and only the following keys) in the exact order:

1. `assets.files.checksum.sha1`
2. `assets.files.checksum.sha256`
3. `assets.files.path`
4. `assets.files.size`
5. `assets.id`
6. `forms.function.args`
7. `forms.function.envmods`
8. `forms.id`
9. `forms.outputs.files.paths`
10. `forms.outputs.files.slots`
11. `forms.precommands.private`
12. `forms.precommands.public`
13. `schema_version.major`
14. `schema_version.minor`

with:

- all whitespace between JSON tokens removed from the canonicalized JSON
- all non-existent array values replaced with empty arrays, and all non-existent boolean balues replaced with `false`
- all `assets.files` sorted by the ascending lexographical UTF-8 byte encoding of `assets.files.path`
- `assets.files.checksum.sha1` removed if `assets.files.checksum.sha256` is present

The above canonicalization should conform to [RFC 8785]; if there are any ambiguities [RFC 8785] must be followed.

Of particular note is that the `assets.listing.origins` and `assets.files.origin`
fields are not present in the canonicalization since
the bundle path, checksum and size uniquely identifiy an asset. In other words,
if you have a locally cached file with the same checksum as a remote bundle,
you can substitute the locally cached file without changing identifiers
in the value store.

[RFC 8785]: https://www.rfc-editor.org/rfc/rfc8785

For example, the form:

```json
{
  "schema_version":{"major":1,"minor":0},
  "forms": [
    {
      "id": "FooBar_Baz@0.1.0",
      "precommands": {
        "private": [
          "private1"
        ],
        "public": [
          "public1"
        ]
      },
      "function": {
        "commands": [
          "arg1"
        ],
        "envmods": [
          "-PATH"
        ]
      },
      "outputs": {
        "assets": [
          {
            "paths": [
              "outpath1"
            ],
            "slots": [
              "output1"
            ]
          }
        ]
      }
    }
  ],
  "bundles": [
    {
      "id": "DkDistribution_Std.Bundle@2.4.202508011516-signed",
      "listing": {
        "origins": [
          {
            "name": "github-release",
            "mirrors": [
              "https://github.com/diskuv/dk/releases/download/2.4.202508011516-signed"
            ]
          }
        ]
      },
      "assets": [
        {
          "origin": "github-release",
          "path": "SHA256.sig",
          "size": 151,
          "checksum": {
            "sha256": "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
          }
        },
        {
          "origin": "github-release",
          "path": "SHA256",
          "size": 559,
          "checksum": {
            "sha256": "4bd73809eda4fb2bf7459d2e58d202282627bac816f59a848fc24b5ad6a7159e"
          }
        }
      ]
    }
  ]
}
```

is canonicalized to:

```json
{"bundles":[{"assets":[{"checksum":{"sha256":"4bd73809eda4fb2bf7459d2e58d202282627bac816f59a848fc24b5ad6a7159e"},"path":"SHA256"},{"checksum":{"sha256":"0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"},"path":"SHA256.sig"}],"id":"DkDistribution_Std.Bundle@2.4.202508011516-signed"}],"forms":[{"function":{"commands":["arg1"],"envmods":["-PATH"]},"id":"FooBar_Baz@0.1.0","outputs":{"assets":[{"paths":["outpath1"],"slots":["output1"]}]},"precommands":{"private":["private1"],"public":["public1"]}}],"schema_version":{"major":1,"minor":0}}
```

## Distributions

A **distribution** is a build that generates [values](#values). In the build system, metadata about distributions can be collected in the values.jsonc files along with *attestations*:

- An *attestation* is a cryptographically verifiable statement (ex. "the build produced bundle A at time T") that is signed by a human (ex. you) or a machine (ex. GitHub Actions).

To increase supply chain security guarantees, the build system will reject assets and objects that are produced by humans or machines without attestations that you have explicitly trusted.

The following sources of attestation are recognized:

- A human can sign a build by using an [OpenBSD signify key](https://www.openbsd.org/papers/bsdcan-signify.html).
- GitHub Actions can sign a build using one of two [SLSA security levels](https://slsa.dev/spec/v1.0/levels):
  - Level 2: <https://docs.github.com/en/actions/how-tos/secure-your-work/use-artifact-attestations/use-artifact-attestations#generating-artifact-attestations-for-your-builds>
  - Level 3: <https://docs.github.com/en/actions/how-tos/secure-your-work/use-artifact-attestations/increase-security-rating>

Unfortunately, GitLab does not yet provide a [minimal level of attestation](https://gitlab.com/groups/gitlab-org/-/epics/15859#note_2540189548).

The build system maintains a *trust store*, with the dk OpenBSD signify key for the `CommonsBase_Std` packages as the only trusted entity by default.

### Distributions are sealed

A published distribution is **sealed**: the exact set of value versions it
contains (its bundles, forms and objects, each at a fixed version) is
determined when the distribution is built and is signed by the distribution's
attestation.

When a consumer imports a distribution, only the value
versions that distribution sealed are resolvable. Requesting a module the
imported distribution did not publish must fail. Requesting different versions
of a module than the distribution sealed must also fail.

Sealing module versions provides the following security feature:

- The signify signature and
  SLSA provenance attest the precise values the distribution was built with.
  The attestation guarantees a consumer receives exactly the version
  that were attested.
- Because a consumer
  can only build and use the value versions the trusted signer explicitly
  attested, a compromised mirror, a tampered cache, or a later malicious build
  cannot introduce an unpublished or substituted module into the consumer's build
  graph.

A package that already has published releases cannot have a
module added to an existing release: the module must be introduced by
building and publishing a new distribution version.

#### Adding to the sealed set

A new distribution version whose `MAJOR.MINOR` is strictly higher than the
latest sealed release may **add** to the sealed set:

- a brand-new module that no sealed release published, or
- a strictly higher `MAJOR.MINOR` module version of an already-sealed module.

Already-sealed (module, version) pairs are immutable for every version bump.
Patch-only releases (ie. the same `MAJOR.MINOR`) must not change the
sealed set.

### Distributed Value Stores

A distribution includes a `.zip` file of some or all of the value store. Entries in the zipfile must be `./{value_id}`;
for example, `./vnmfdhn7lw4wepx2qiunrmgm4o5lx4wwsf2yfj7xyxggkg5kdsltq`.

The following values must be present:

- the "j" values file for any values.json with a form or bundle having a library identical to the distribution library `id`
- the "w" parsed values ast for any values.json with a form or bundle having a library identical to the distribution library `id`
- the "o" object file for any object having a library identical to the distribution library `id`

The following values will be ignored if present:

- the "c" constants

A unoptimal implementation can simply zip up the value store directory, but done frequently that leads to wasted bandwidth, storage and lengthier builds.

An optimal implementation only includes the necessary values.

### OpenBSD signify keys

[Securing OpenBSD From Us To You]: https://www.openbsd.org/papers/bsdcan-signify.html

The security policy is described in [Securing OpenBSD From Us To You].
Signify builds keys are automatically created specific to the user (you) on a machine. The private keys should not be shared except among a small set of trusted co-signers (ex. your manager and a peer).

Any OpenBSD signify key and any GitHub repository encountered by the build system is automatically denied.
However, you are prompted if you want to accept or deny the key or repository, with the default as deny.

When accepting an OpenBSD signify public key embedded in a values.json build file, the

- accompanying continuations that were signed with the private key are accepted. These continuations are public keys for the next version, and imported once and never overwritten.
- accompanying builds that were signed with the private key are accepted
- *all* builds signed with the private key that with packages that share the major + minor parts of the accompanying build are accepted

Once a major and minor component of a package has been signed, no lower versioned distribution key can claim:

- that major and minor component
- any later versions to that major and minor component

That may be difficult to understand without an example. We'll use the Key Rotation example from [Securing OpenBSD From Us To You], where OpenBSD is about to release OpenBSD version 5.6. They:

- have an existing public/private key pair for version 5.6
- have generated public/private key pairs for version 5.7 *before* 5.6 is released
- have generated public/private key pairs for version 5.8 *before* 5.6 is released

In the build system the version 5.6 would be a distribution; let's say `OpenBSD_Std@5.6.0`:

```json
{
  "distributions": {
    "id": "OpenBSD_Std@5.6.0",
    "producer": {
      "openbsd_signify": {
        // key from https://ftp.eu.openbsd.org/pub/OpenBSD/signify/openbsd-56-base.pub
        "public_key": "untrusted comment: openbsd 5.6 base public key\nRWR0EANmo9nqhpPbPUZDIBcRtrVcRwQxZ8UKGWY8Ui4RHi229KFL84wV"
      }
    },
    "continuations": {
      "attestation": {
        "openbsd_signify": {
          // this is a (fake) signature of SHA256(plaintext), where
          // plaintext is `{"5.7":"...","5.8":"..."}` (see below)
          "signature": "untrusted comment: signature from signify secret key\nRWTAeKJJ1MTF3UpxzBCu6NaM6HPJNTj5CZ+M5XNJKNeEHBLQSsstzHGbSo8rPYNgw3Z98pN7WKiIwBIyRrKuIdKBRA6qlaci6wI="
        }
      },
      "continuations": {
        // https://ftp.eu.openbsd.org/pub/OpenBSD/signify/openbsd-57-base.pub
        "5.7": "untrusted comment: openbsd 5.7 base public key\nRWSvUZXnw9gUb70PdeSNnpSmodCyIPJEGN1wWr+6Time1eP7KiWJ5eAM",
        // https://ftp.eu.openbsd.org/pub/OpenBSD/signify/openbsd-58-base.pub
        "5.8": "untrusted comment: openbsd 5.8 base public key\nRWQNNZXtC/MqP3Eiu+6FBz/qrxiWQwDhd+9Yljzp62UP4KzFmmvzVk60"
      }
    }
  }
}
```

In the same distribution in values.jsonc we also must have at least one build:

```json
{
  "distributions": {
    "id": "OpenBSD_Std@5.6.0",
    "producer": { /* ... */ },
    "continuations": { /* ... */ },
    "build": {
      "attestation": {
        // this is a (fake) signature of SHA256(plaintext), where
        // plaintext is `{"modules":...,...}` (see below)
        "openbsd_signify": "untrusted comment: signed by key c078a249d4c4c5dd\nRWTAeKJJ1MTF3UpxzBCu6NaM6HPJNTj5CZ+M5XNJKNeEHBLQSsstzHGbSo8rPYNgw3Z98pN7WKiIwBIyRrKuIdKBRA6qlaci6wI="
      },
      "build_to_sign": {
        "modules": ["DkExe_Std.Form@1.0.202501010000"],
        /* other build fields described in the mlfront-values.json schema */
      }
    }
  }
}
```

The `OpenBSD_Std@5.6.0` public key has signed for `DkExe_Std.Form@1.0.202501010000`, but can also sign
for any `DkExe_Std.Form@1.0.*` (and no others!) since those all share the major and minor number of the `DkExe_Std.Form` in values.jsonc.

The `OpenBSD_Std@5.7.0` public key can sign for either `DkExe_Std.Form@1.0.*` or a later version (ex. `DkExe_Std.Form@1.1.*`), but
cannot sign for both. That is, once a `OpenBSD_Std@5.7.0` public key is seen to accompany `DkExe_Std.Form@1.1.*`, the association `OpenBSD_Std@5.7.0 may-sign DkExe_Std.Form@1.1.*` is stored in the trust store, and it can no longer
sign any other major + minor version.

Most important, the `OpenBSD_Std@5.7.0` public key **cannot** sign for an earlier version (ex. `DkExe_Std.Form@0.7.*`) than `OpenBSD_Std@5.6.0`.

The net effect is a tendency of the trust store to increase versions over time, so that these OpenBSD signify keys have an implicit rotation policy as these versions increase. You choose how many keys you want to keep alive at any time (the current key plus the number of keys in your `continuations`), and do a rotation by doing a new build using a key from one of your continuations.

So, let's say you keep 2 continuations alive (plus the current key) like OpenBSD does. When you start using a key from one of your continuations (ex. `OpenBSD_Std@5.7.0`), you should will keep `N-1 = 1` keys in rotation (ex. "continuations" should contain `OpenBSD_Std@5.8.0` still) **and** create one new key (ex. "continuations" should have a new `OpenBSD_Std@5.9.0` key).

#### Distribution versioning

The first trusted release that claims a `MAJOR.MINOR` establishes its key, and that
association is imported once and never overwritten.

The implication is that distribution keys must be rotated for a new `MAJOR.MINOR`
distribution package version.

- A release within the current line (a new `MAJOR.MINOR.PATCH`, ex. the
  `MAJOR.MINOR.<UTC timestamp>` tags of a dk package) is signed by the line's
  existing secret key.
- A release on a line listed in the current line's `continuations` is signed
  by that pre-signed key. The consumer accepts it through the continuation
  chain with no new trust decision. The producer must possess the continuation's
  secret key to perform the release.
- A release on a line absent from every trusted release's `continuations` has
  no chain to it. Every consumer is prompted to accept its key as if the
  producer were new.

### GitHub SLSA Level 2

When accepting a GitHub repository (SLSA Level 2), the:

- accompanying builds attested by GitHub are accepted
- *all* builds attested by GitHub are accepted

The build system will download the GitHub CLI (using the default trusted `CommonsBase_Std` library) to do verification of the builds.

### GitHub SLSA Level 3

When accepting a known, vetted GitHub Actions script (SLSA Level 3), the:

- accompanying builds attested by GitHub are accepted
- *all* builds produced by the GitHub Actions scripts that are attested by GitHub are accepted

The build system will download the GitHub CLI (using the default trusted `CommonsBase_Std` library) to do verification of the builds.

### Distribution Scripts

🚧*missing docs*: make a section on cram tests independent of distributions

When writing distribution scripts:

1. Running one rule in a script module will bring in the entire script module.
   In particular, if a distribution script runs a function rule from a
   `*.values.lua` scriptmodule, the entire `*.values.lua` scriptmodule is
   distributed. Function rules should follow the `F_<Name>` naming convention to
   avoid colliding with UI rule names in the same scriptmodule.

   It is best to run every function rule for lightweight testing during distribution.
2. Use `${CONFIG}` for files in the cram test directory.
3. Use `${RUNTIME}/<unique path to cram test>` for -f and -d options since outputs are relative to the
   current dir like `run-function`, `dialog`, and `exec`. It is set unique to the cram
   test. The `<unique path to cram test>` is to avoid race conditions on Windows where
   Windows Defender (etc.) may not make a file visible or rewritable immediately after creation.
4. Declare each static rule input in the rule's `declareoutput` response.

## Scripts

### Script Introduction

The build system has first-class support for Lua as a scripting language.

Lua scripts are processed by the build system in a couple places:

- REGULAR SCRIPT: In `values.lua` or `*.values.lua` files in the same include directories (`-I`) as the [Values](#values) (`values.json[c]` and `*.values.json[c]`) files
- EMBEDDED SCRIPT: Embedded in comments at the top of **single-file** scripts.

For example, a regular script may be:

```lua
-- file: values.lua
SomeRule = require('SomeLibrary_Std.SomeRule')
SomeRule = SomeRule.at('1.0.0') -- this should be on the same line except bug with OCaml Lua parser
SomeRule:Executable {
  id='OurTest_Std.OurMain@2.3.4',
  files={
    glob={
      origin='someorigin',
      patterns={'**/*.ml', '**/*.mli'},
      exclude={'tests/**'}
    }
  }
}
```

while embedded in an OCaml single-file script the Lua script is inside the `!dk` comment:

```ocaml
#!/usr/bin/env
let () = print_endline "In the beginning ..."
let () = print_endline "We ran this inside our executable."
(*
SomeRule = require('SomeLibrary_Std.SomeRule')
SomeRule = SomeRule.at('1.0.0')
SomeRule:Executable {
  id=build.me.id,
  files=build.me.asset
}
!dk!p *)
```

All scripts in a running build share the same Lua state, and Lua is interpreted serially.
That means two things:

- All Lua scripts must be fast. The "continuation" mechanism, described in a later subsection, lets scripts give parallelizable work to the build engine through [subshells](#subshells).
- Lua scripts must be written to minimize use of global variables.

### Script Phases

A Lua script is scanned once but evaluated (ie. interpreted) twice.

The first evaluation does a quick scan in a very restrictive sandbox to find which dependencies the script needs and what modules and rules the script exports.
This first evaluation happens in the the [VALUESCAN](#evaluation) phase documented later in the specification.

The second evaluation runs the script conventionally.
This second evaluation happens in the the [VALUELOAD](#evaluation) phase documented later in the specification.
All Lua functions behave as documented later in this specification.

Care is needed so that the script completes without errors in the sandbox of the first *VALUESCAN* evaluation. The *VALUESCAN* sandbox does the following:

1. `require('Mod.X_Y_Z')` (preferred) or `require(dependency).at(version)` (deprecated)
   will capture the name and version of the dependency, but not load the dependency.
2. `assert(...)` and `error(...)` continue to do Lua conventional error checking
3. `build.is_building` will return a false-y value (ie. `nil`, or `false` if the Lua implementation version is modern)
4. All other built-in functions (ex. `print()`, `table.unpack`) are defined to return a sensible Lua value but do nothing.
5. The fallback for reading an unknown key from a table (ex. `print(a.b.some_unknown_field)`) is to return `nil` (conventionally it would error).
6. The fallback for writing an unknown key to a table (ex. `a.b.some_unknown_field = 1`) is to return `nil` (conventionally it would error).
7. The fallback for unknown functions (ex. `some_unknown_function()`) is a function that returns `nil` (conventionally it would error).

### Lua Specification

The overall design goal is to maintain conventional Lua behavior as much as possible. The end user, to the extent possible, should be able to use their favorite Lua IDEs to edit their Lua build scripts.

---

The build system uses Lua 2.5 for its syntax (no `for` loops) and its data model (no metatables), but uses functions available from Lua 5.1+ (ex. `require`).

> Historical note: Lua 2.5 was published in 1996 and lacks several features of modern-day Lua: `for` loops, metaprogramming for metatables, and coroutines. However, rules are mostly configuration, and a full programming language makes hermetic, bounded-time builds difficult or impossible. So even if a future specification uses a later Lua version, several features will be disabled.

---

To support Lua IDEs:

- Lua 5.1+: The Lua convention is one module exported by script. So unlike `value.json[c]`, a `[*.]values.lua` script only has one module. To export functions and rules from the module, the module returns a Lua table per the Lua 5.2+ convention (and compatible with Lua 5.1).

---

Lua names (aka identifiers), for maximum portability, use the Lua 2.5 lexical conventions:

> Identifiers can be any string of letters, digits, and underscores, not beginning with a digit.

and the Lua 5.4 reserved words:

- and
- break
- do
- else
- elseif
- end
- false
- for
- function
- goto
- if
- in
- local
- nil
- not
- or
- repeat
- return
- then
- true
- until
- while

### Lua Global Variables

#### Lua Global Variable - arg

`arg`

A table defined only when Lua is run [embedded](#embedded-file-scripts)
*or* a file is run directly as a Lua script.

Before running any code, lua collects all command-line arguments in a global table called
`arg`. The script name goes to index 0, the first argument after the script name goes to index 1, and so on.
Any arguments before the script name (that is, the interpreter name plus its options) go to negative indices.
For example, in the call:

```sh
${dk_build_system} lua -la b.lua t1 t2
```

the table is like this:

```lua
arg = { [-2] = "lua", [-1] = "-la",
        [0] = "b.lua",
        [1] = "t1", [2] = "t2" }
```

If there is no script in the call, the interpreter name goes to index 0, followed by the other arguments. For instance, the call

```sh
${dk_build_system} lua -e "print(arg[1])"
```

will print `-e`. If there is a script, the script is called with arguments `arg[1], ···, arg[#arg]`.

#### Lua Global Variable - loadstring

`loadstring (string [, chunkname])`

Compiles the string.

If there are no errors, returns the compiled chunk as a function; otherwise, returns `nil` plus the error message. The environment of the returned function is the global environment.

```lua
chunk = assert(loadstring(s))
chunk()
```

When absent, `chunkname` defaults to the given string or an abbrevation of it.

Compatibility: Lua 5.1, 5.2, 5.3 but removed from 5.4.

#### Lua Global Variable - next

`next (table, index)`

This function allows a program to traverse all fields of a table. Its first argument is a table and its second argument is an index in this table. It returns the next index of the table and the value associated with the index. When called with nil as its second argument, the function returns the first index of the table (and its associated value). When called with the last index, or with nil in an empty table, it returns nil.
In Lua there is no declaration of fields; semantically, there is no difference between a field not present in a table or a field with value nil. Therefore, the function only considers fields with non nil values. The order in which the indices are enumerated is not specified, even for numeric indices. If the table is modified in any way during a traversal, the semantics of next is undefined.

#### Lua Global Variable - tostring

`tostring (e)`

This function receives an argument of any type and converts it to a string in a reasonable format.

Table contents are not converted. See [jsondk.encode](#jsondkencode) to show inside of a table.

#### Lua Global Variable - print

`print (e1, e2, ...)`

This function receives any number of arguments, and prints their values in a reasonable format. Each value is printed in a new line.
This function is not intended for formatted output, but as a quick way to show a value, for instance for error messages or debugging.

See [printf](#lua-global-variable---printf) for functions for formatted output.

See [jsondk.encode](#jsondkencode) to print tables.

#### Lua Global Variable - printf

`printf("format", ...)`

This function performs like its C counterpart, printing a formatted string.

It is equivalent to this Lua code:

```lua
print(string.format(format, unpack(arg))
```

without the newline inserted by `print`.

`format` is a formatting string containing C `printf()` style formatting codes.
It is followed by a list of arguments to be substituted into the format string.

> This function was borrowed from [Premake's printf](https://premake.github.io/docs/globals/printf/).

#### Lua Global Variable - tonumber

`tonumber (e)`

This function receives one argument, and tries to convert it to a number. If the argument is already a number or a string convertible to a number (see Section 4.2), then it returns that number; otherwise, it returns nil.

#### Lua Global Variable - type

`type (v)`

This function allows Lua to test the type of a value. It receives one argument, and returns its type, coded as a string. The possible results of this function are "nil" (a string, not the value nil), "number", "string", "table", "function" (returned both for C functions and Lua functions), and "userdata".

Lua 5.1+ compatibility: Unlike Lua 2.5, the `type` function does *not* return a "tag" as a second result.

#### Lua Global Variable - assert

`assert (v [, message])`

Raises an error if the value of its argument v is false (i.e., `nil` or in a future specification `false`); otherwise, returns all its arguments.
In case of error, `message` is the error object; when absent, it defaults to `assertion failed!`

Compatible with Lua 5.1.

#### Lua Global Variable - error

`error (message)`

This function issues an error message and terminates the last called function from the library.
It never returns.

Lua 5.1+ compatibility: The "level" argument in `error (message, [level])` is ignored.

### Lua build library

`build` is a Lua table with access to the running build.

#### build.newrules

```lua
local M = { id = '...' }
fnrules = build.newrules(M)
function fnrules.SomeRule(command,request)
  -- ...
end
return M

-- or if interactive user interface rules are needed ...

local M = { id = '...' }
fnrules, uirules = build.newrules(M)
function fnrules.SomeRule(command,request)
  -- ...
end
function uirules.SomeRuleThatCanTakeOverConsole(command,request)
  -- ...
end
return M
```

`build.newrules(M)` creates a `fnrules` and `uirules` field inside the module table `M`.

The `fnrules` and `uirules` fields will both be empty tables, and those empty tables are returned.

- `fnrules` are *function* rules that can be used in `values.json[c]` files or invoked by the end-user.
- `uirules` are *interactive* rules that can only be invoked by the end-user.

See [Custom Lua Rules](#introduction-to-custom-lua-rules) for a detailed explanation of the difference between `fnrules` and `uirules`.

### Lua jsondk library

The `jsondk` library is embedded into the build system (nothing needs to be downloaded) but it must be accessed through `jsondk = require('jsondk')`.
That keeps with the [design goal to maintain Lua conventions](#lua-specification).

There is a popular, unrelated Lua library [dkjson](https://dkolf.de/dkjson-lua/) for parsing JSON.
Since the chance for confusion is high, `jsondk` is broadly compatible with `dkjson`.

#### jsondk.encode

```lua
jsondk = require('jsondk')
tbl = {
  animals = { "dog", "cat", "aardvark" },
  instruments = { "violin", "trombone", "theremin" }
}
str = jsondk.encode (tbl, { indent = true })

-- or
jsondk = require('jsondk')
str = jsondk.encode (tbl)
```

Converts a Lua value to JSON:

- `nil` values are not printed
- `jsondk.null` values are encoded as JSON null

If `indent` is truthy then the JSON is pretty-printed.

#### jsondk.decode

```lua
jsondk = require('jsondk')
str = '{"animals":["dog","cat","aardvark"],"bugs":null}'
jsondk.decode (str)

-- or
jsondk = require('jsondk')
value, errmsg, errrendered, sb, sl, sc, eb, el, ec = jsondk.decode (str)
```

Converts JSON to a Lua value:

- Large numbers are converted to floating-point numbers with a possible loss of precision. If outside the floating-point range, an error is raised.
- JSON nulls are converted to `jsondk.null` Lua values

If the JSON could be converted, the result is the first return value.

Otherwise:

- `value` is `nil`
- `errmsg` is a brief error message
- `errrendered` is a prerendered error
- `sb`, `sl`, and `sc` are the starting byte offset (zero-based), line and column (1-based)
- `eb`, `el`, and `ec` are the ending byte offset (zero-based), line and column (1-based)

#### jsondk.null

```lua
jsondk = require('jsondk')
tbl = {
  animals = { "dog", "cat", "aardvark" },
  bugs = jsondk.null,
  trees = nil
}
str = jsondk.encode (tbl)

-- {
--   "animals":["dog","cat","aardvark"],
--   "bugs":null
-- }
```

The `jsondk.null` Lua value represents JSON null.

### Lua math library

This Lua 5.4 compatible library provides basic mathematical functions. It provides all its functions and constants inside the table `math`.
Functions with the annotation "integer/float" give integer results for integer arguments and float results for non-integer arguments.
The rounding functions `math.ceil`, `math.floor`, and `math.modf` return an integer when the result fits in the range of an integer, or a float otherwise.

#### math.abs

`math.abs (x)`

Returns the maximum value between x and -x. (integer/float)

#### math.acos

`math.acos (x)`

Returns the arc cosine of x (in radians).

#### math.asin

`math.asin (x)`

Returns the arc sine of x (in radians).

#### math.atan

`math.atan (y [, x])`

Returns the arc tangent of y/x (in radians), using the signs of both arguments to find the quadrant of the result. It also handles correctly the case of x being zero.

The default value for x is 1, so that the call math.atan(y) returns the arc tangent of y.

#### math.ceil

`math.ceil (x)`

Returns the smallest integral value greater than or equal to x.

#### math.cos

`math.cos (x)`

Returns the cosine of x (assumed to be in radians).

#### math.deg

`math.deg (x)`

Converts the angle x from radians to degrees.

#### math.exp

`math.exp (x)`

Returns the value `eˣ` (where `e` is the base of natural logarithms).

#### math.floor

`math.floor (x)`

Returns the largest integral value less than or equal to x.

#### math.fmod

`math.fmod (x, y)`

Returns the remainder of the division of `x` by `y` that rounds the quotient towards zero. (integer/float)

#### math.huge

`math.huge`

The float value `HUGE_VAL`, a value greater than any other numeric value.

#### math.log

`math.log (x [, base])`

Returns the logarithm of `x` in the given base. The default for base is `e` (so that the function returns the natural logarithm of `x`).

#### math.max

`math.max (x, ···)`

Returns the argument with the maximum value according to the Lua operator `<`.

#### math.maxinteger

`math.maxinteger`

An integer with the maximum value for an integer.

#### math.min

`math.min (x, ···)`

Returns the argument with the minimum value, according to the Lua operator <.

#### math.mininteger

`math.mininteger`

An integer with the minimum value for an integer.

#### math.modf

`math.modf (x)`

Returns the integral part of x and the fractional part of x. Its second result is always a float.

#### math.pi

`math.pi`

The value of π.

#### math.rad

`math.rad (x)`

Converts the angle x from degrees to radians.

#### math.sin

`math.sin (x)`

Returns the sine of x (assumed to be in radians).

#### math.sqrt

`math.sqrt (x)`

Returns the square root of x. (You can also use the expression x^0.5 to compute this value.)

#### math.tan

`math.tan (x)`

Returns the tangent of x (assumed to be in radians).

#### math.tointeger

`math.tointeger (x)`

If the value x is convertible to an integer, returns that integer. Otherwise, returns fail.

#### math.type

`math.type (x)`

Returns "integer" if x is an integer, "float" if it is a float, or fail if x is not a number.

#### math.ult

`math.ult (m, n)`

Returns the string `t` (ie. a boolean `true`) if and only if integer `m` is below integer n when they are compared as unsigned integers.

### Lua package library

The package library provides basic facilities for loading modules in Lua.
It exports one function directly in the global environment: `require`.
Everything else is exported in the `table` package.

#### require

`require (modname)`

Loads the given module.

If the `modname` is a **standard module id** (ex. `MyLibrary_Std.A.B.MyModule` - *tbd: document this*) a task is added to the [task graph](#task-model) to search for it.
The section [Custom Lua Modules](#custom-lua-modules) describes how to create your own modules.

As of the writing of this specification, only standard modules may be loaded.

Standard modules must be required with a specific version using one of two
equivalent forms:

- **Version-encoded form** (preferred): `require('Lib_Std.Mod.X_Y_Z')` where
  `X_Y_Z` is the semver version string with every `.` and `-` replaced by `_`.
  Example: `require('CommonsBase_Std.Extract.0_2_0')` loads version `0.2.0`.
  Note: the version suffix starting with a digit is valid as a `require`
  argument string even though it is not a valid bare Lua identifier.
- **`.at()` form** (deprecated): `require('Lib_Std.Mod').at('X.Y.Z')`.
  Example: `require('CommonsBase_Std.Extract').at('0.2.0')`.

Once imported with `require`, standard modules are enriched with constants as per
[Lua 5.1 module() convention](https://www.lua.org/manual/5.1/manual.html#pdf-module) and
[Lua module versioning conventions](http://lua-users.org/wiki/ModuleVersioning) and a `_build` field:

| Field      | Example                                                                    |
| ---------- | -------------------------------------------------------------------------- |
| `_NAME`    | `MyModule._NAME` would be `MyLibrary_Std.A.B.MyModule`                     |
| `_PACKAGE` | `MyModule._PACKAGE` would be `MyLibrary_Std.A.B`                           |
| `_VERSION` | `MyModule._VERSION` would be `1.0.0`                                       |
| `_M`       | (may be removed) `MyModule._M` would be a Lua reference to `MyModule`      |
| `_build`   | *described later in [Custom Lua Rules](#introduction-to-custom-lua-rules)* |

> Historical note: Even though the implementation of `module()` is deprecated after Lua 5.1, its conventions were never deprecated.

#### package.registrykey

`package.registrykey`

A opaque variable holding a key to an internal table of packages that are loaded.

### Lua request.rule library

This library is available through the `request.rule` field to [function rules](#function-rules) and to [UI Rules](#ui-rules).

For example:

```lua
local M = { id = '...' }
rules = build.newrules(M)
function rules.SomeRule(command,request)
  if command == "declareoutput" then
    -- use the [rule] library
    local id = request.rule.generatesymbol()
  end
end
return M
```

#### request.rule.generatesymbol

```lua
request.rule.generatesymbol(arg1, arg2, ...)
```

Generates a deterministic standard namespace term. For example, it may generate `X6pro7j57evsyymo36mehvpabhy` from a constant `X` followed by a lowercase base32-encoding of the BLAKE2s 128-bit digest of:

1. The rule's `MODULE` string from its `id = "MODULE@VERSION"`
2. The rule's `VERSION` string from its `id = "MODULE@VERSION"`
3. The `request.user` table
4. The arguments `arg1, arg2, ...`

Each Lua value has its digest calculated according to:

- `nil`: The `0x00` byte
- number: The little-endian IEEE 754 double-precision float representation of the number (even for integers).
- string: The bytes of the string
- function: An error is raised.
- userdata: An error is raised.
- a table with number or string keys:
  - Any number key is converted to an integer or it raises an error; then the integer is converted to a string.
  - The keys are lexographically sorted
  - The digest is calculated with depth-first traversal. That is the string `KEY1` then the Lua value `VALUE1`, then `KEY2` and `VALUE2`, until there are no more key values.

### Lua request.execution library

This library is available only to [UI Rules](#ui-rules) through the `request.execution` field.

An example for an execution specific UI rule:

```lua
local M = { id = '...' }
rules, uirules = build.newrules(M)
function uirules.SomeRule(command,request)
  if command == "submit" then
    -- use the [request.execution] library
    local osfamily = request.execution.OSFamily
    if osfamily == "macos" then
      print("Howdy mac users!")
    end
  end
end
return M
```

> [!TIP]
> [Function rules](#function-rules) do not have `request.execution`
> because function rule outputs must stay reproducible across execution platforms.
> When a function rule needs platform specific outputs, use
> `declareoutput.execution_slot` together with slot expansions like
> [${SLOTNAME.Release.execution_abi}](#slotnameslotname).

An example for an execution specific function rule that does *not* use `request.execution`:

```lua
function rules.F_Build(command, request)
  if command == "declareoutput" then
    return {
      declareoutput = {
        return_objects = {
          id = "UserLibrary_Std.A.B.FreeRule.OutputObject@1.0.0",
          slots = { "Release.Windows_x86_64", "Release.Darwin_arm64" },
          execution_slot = "Release.execution_abi"
        }
      }
    }
  elseif command == "submit" then
    return {
      values = {
        schema_version = { major = 1, minor = 0 },
        forms = {
          {
            id = p.outputid,
            function_ = {
              commands = {
                -- using one named slot (ex. SLOT.Release.Windows_x86_64) restricts the command to only Release.Windows_x86_64.
                -- when [env -u SOMETHING -- ...] runs, it delegates to the "..." part
                {
                    "$(get-object CommonsBase_Std.Coreutils@0.6.0 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f coreutils.exe -e '*')",
                    "env", "-u", "${SLOT.Release.Windows_x86_64}", "--",
                    "cmd", "/c", "\"call build.bat msvc64 & exit /b %ERRORLEVEL%\""
                },
                {
                    "$(get-object CommonsBase_Std.Coreutils@0.6.0 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f coreutils.exe -e '*')",
                    "env", "-u", "${SLOT.Release.Darwin_arm64}", "--",
                    "/bin/sh", "-c", "./configure && make && make install"
                }
              },
            },
            outputs = {
              assets = {
                -- files in common to all slots
                {
                  slots = { "Release.Windows_x86_64", "Release.Darwin_arm64" },
                  paths = { "LICENSE.txt" }
                },
                -- files specific to each ABI
                {
                  slots = { "Release.Windows_x86_64" },
                  paths = { "sample.exe" }
                },
                {
                  slots = { "Release.Darwin_arm64" },
                  paths = { "sample" }
                },
              }
            }
          }
        }
      }
    }
  end
end
```

#### request.execution.OSFamily

```lua
request.execution.OSFamily
```

The `OSFamily` of the execution platform. Values include `windows` and `macos`; they are defined in [OSFamily](#osfamily).

#### request.execution.ABIv3

```lua
request.execution.ABIv3
```

The third version of the DkML ABI of the execution platform described in [${SLOTNAME.SlotName}](#slotnameslotname). Examples include `Windows_x86_64` and `Darwin_arm64`.

#### request.execution.OSv3

```lua
request.execution.OSv3
```

The third version of the DkML OS of the execution platform. The set of OSv3 values is:

- `UnknownOS`
- `Android`
- `DragonFly`
- `FreeBSD`
- `IOS`
- `Linux`
- `NetBSD`
- `OpenBSD`
- `OSX`
- `Windows`

### Lua request.io library

This library is available to [function rules](#function-rules) and [UI Rules](#ui-rules) through the `request.io` field.

For example:

```lua
local M = { id = '...' }
rules = build.newrules(M)
function rules.SomeRule(command,request)
  if command == "submit" then
    -- use the [request.io] library
    local file = request.io.open("a/b/somefile", "w")
  end
end
return M
```

Capabilities are restricted so that:

- File objects can be created only to write to files in a directory unique to the rule and output key. The intent for these writable file objects is to allow creating [assets](#assets) and nothing else.
- File objects for reading can be obtained from value shell expressions given in response to [Function Rule Command - `submit`](#function-rule-command---submit)

Some build system implementations may sandbox the I/O operations.

#### request.io.open

```lua
file_or_directory = request.io.open(filename_or_dirname, mode)
```

This function opens a file or directory in the mode specified in the string mode. It returns a new file descriptor, or, in case of errors, nil plus an error message.

The mode string can be any of the following:

- "r" read-only mode.
- "w" write mode. `filename_or_dirname` must be a filename. Any parent directories required by `filename` will be created.

Unlike the C library function `fopen`, the file will be opened in binary mode rather than text mode. (Text mode adds CRLF on Windows systems and is non-reproducible when cross-compiling.)

The `filename_or_dirname` must be a [strictly relative path](#strictly-relative-path):

- An absolute path will raise an error.
- After the path is normalized, any path segments that start with `..` will raise an error.
- After the path is normalized, any path segments that contain a forward or backward slash will raise an error. For example, Unix filenames can contain backslashes, but they will raise errors.

Directory operations:

- For the `r` read-only mode when `filename_or_dirname` is a directory, the usable functions are `request.io.list` and `request.io.toasset`.

The file *may* be closed after the request is finished (ie. the `run-function` command is finished), but it is the author's responsibility to close the file with [request.io.close](#requestioclose) or with [request.io.toasset](#requestiotoasset).

#### request.io.read

```lua
request.io.read(file, format1, ...)
```

Reads the file `file` according to the given formats `format1, ...` which specify what to read. For each format, the function returns a string or a number with the characters read, or `nil` if it cannot read data with the specified format. (In this latter case, the function does not read subsequent formats.) When called without arguments, it uses a default format that reads the next line (see below).

The available formats are

- `a`, `all` or `*all`: reads the whole file, starting at the current position. On end of file, it returns the empty string; this format never fails unless the file does not exist or is unreadable
- `l`, `line` or `*line`: reads the next line skipping the end of line, returning `nil` on end of file. This is the default format.
- `L`: reads the next line keeping the end-of-line character (if present), returning `nil` on end of file.
- *number*: reads a string with up to this number of bytes, returning `nil` on end of file. If number is zero, it reads nothing and returns an empty string, or `nil` on end of file.

The formats `l` and `L` should be used only for text files.

This function behaves [Lua 5.4 io.read](https://www.lua.org/manual/5.4/manual.html#6.8) except the format `n` is not supported.

#### request.io.write

```lua
request.io.write(file, value1, ...)
```

Writes the value of each of its arguments to file `file`.
The arguments must be strings or numbers. To write other values, use [tostring](#lua-global-variable---tostring)
or [string.format](#stringformat) or [jsondk.encode](#jsondkencode).

#### request.io.list

```lua
request.io.list(dir, format1, ...)
```

List the contents of directory `dir` according to the given formats `format1, ...` which specify what to list.
For each format, the function returns a table (see below) with the directory contents read, or `nil` if it cannot list
the directory with the specified format.
(In this latter case, the function does not list with subsequent formats.) When called without arguments, it uses a default format that lists the whole directory (see below).

The available formats are

- `a` or `all`: list the entire directory, starting at the current position. On the end of directory, it returns the empty table; this format never fails unless the directory does not exist or is unreadable

The directory contents table has:

- keys that are index numbers: `1`, `2`, etc.
- values that are lazily-opened readonly file or subdirectories. Lazy-open means you do not need to [close](#requestioclose) it unless you [read](#requestioread) from it.

Use [request.io.isfile](#requestioisfile) and [request.io.isdir](#requestioisdir) to check the type of the directory entry.

#### request.io.isfile

```lua
request.io.isfile(file)
```

A truthy value if and only if `file` is a file object.
Any other Lua value will return a falsy value.

#### request.io.isdir

```lua
request.io.isdir(dir)
```

A truthy-value if and only if `dir` is a directory object.
Any other Lua value will return a falsy value.

#### request.io.realpath

```lua
request.io.realpath(file [, { relative = 1 }])
request.io.realpath(dir [, { ... }])
```

The path to the file or directory object.

The validity is only guaranteed inside:

- a [submit command](#function-rule-command---submit) until the next [continuation](#rule-argument---continue_).
- a [ui command](#ui-rule-command---ui) until the ui command is finished

In particular:

- The path may not exist immediately after `request.io.realpath`. A hermetic implementation is allowed to:

  1. Return dangling symlinks as the return value of `request.io.realpath`
  2. Bind those symlinks (ex. `ln -s -f` on Unix) to correct locations after the Lua rule function is finished but immediately before running rule expressions
  3. Bind those symlinks to dangling locations after the rule expressions are finished

If `relative` is truthy, the path is returned as a relative path from the project base directory.
Since, especially on Windows, not all paths can be made relative, the conversion to a relative path is best-effort.

Without `relative`, at the discretion of the build system implementation the returned path may be an absolute path or a relative path.

#### request.io.toasset

```lua
local origin, asset = request.io.toasset(file_or_dir, {
  path = "some/asset/path",
  origin_name = "..."
})
```

Converts the file or directory to an asset and closes the file.

In the options only `path` is mandatory.

`path`: Two assets at the same `path` is an error. Each `path` must be [strictly relative](#strictly-relative-path) or an error will be raised.

`origin_name`: The name of the origin. The origin is a label used to invalidate assets. Many assets can share the same origin.

The `origin` return value will be a table unique to the request. Multiple calls to `request.io.toasset` in the same request will give the same origin table:

```lua
{
  name = "SOME_IDENTIFIER",
  mirrors = { "selfasset://SOME_IDENTIFIER" }
}
```

The `asset` return value will be another table:

```lua
{
  -- from the `origin` table
  origin = "SOME_IDENTIFIER",
  -- from the request.io.toasset(file, {path}) argument
  path = "some/asset/path",
  -- the rest is calculated from the file
  size = 151,
  checksum = {
    sha256 = "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
  }
}
```

Security note: There is no protection against two `request.io.toasset` with the same `path` and `origin_name`. However, implementations are required to calculate the SHA256 checksum in the `asset` return value from the [request.io.write](#requestiowrite) rather than the filesystem. That means in a race multiple assets may be placed in the valuestore, but all will be valid assets and at most one will be the checksum recorded in the tracestore.

#### request.io.flush

```lua
request.io.flush()
```

Flushes the standard output that was buffered from [print](#lua-global-variable---print) and [printf](#lua-global-variable---printf).

#### request.io.close

```lua
request.io.close(file)
request.io.close(directory)
```

Closes the file or directory.

### Lua request.submit library

This library is available to [function rules](#function-rules) and [UI Rules](#ui-rules) through the `request.submit` field.

For example:

```lua
local M = { id = '...' }
rules = build.newrules(M)
function rules.SomeRule(command,request)
  if command == "submit" then
    -- use the [submit] library
    local id = request.submit.outputid
  end
end
return M
```

#### request.submit.outputid

```lua
request.submit.outputid
-- example: OurTest_Std.A.X6pro7j57evsyymo36mehvpabhy@0.1.0
```

This string is the form or asset identifier declared in [the "declareoutput" command](#function-rule-command---declareoutput).

It is only available to [function rules](#function-rules).

#### request.submit.outputmodule

```lua
request.submit.outputmodule
-- example: OurTest_Std.A.X6pro7j57evsyymo36mehvpabhy
```

This string is the module (`MODULE`) of the form or asset `MODULE@VERSION` declared in [the "declareoutput" command](#function-rule-command---declareoutput).

It is only available to [function rules](#function-rules).

#### request.submit.outputversion

```lua
request.submit.outputmodule
-- example: 0.1.0
```

This string is the version (`VERSION`) of the form or asset `MODULE@VERSION` declared in [the "declareoutput" command](#function-rule-command---declareoutput).

It is only available to [function rules](#function-rules).

### Lua request.ui library

This library is available to [UI Rules](#ui-rules) through the `request.ui` field.

For example:

```lua
local M = { id = '...' }
rules, uirules = build.newrules(M)
function uirules.SomeRule(command,request)
  if command == "ui" then
    -- use the [request.ui] library
    local bundle, getbundle, getasset = request.ui.glob { -[[ ... ]] }
  end
end
return M
```

#### request.ui.glob

```lua
bundle, getbundle, getasset = request.ui.glob {
  patterns = {"src/**/*.c"}
  [, cell = "root"]
  [, project = "OurProject_Std@0.1.0"]
  [, excludes = {"src/**/test*.c"} ]
  [, trace = 1]
}
```

Creates a [bundle](#bundles) of files from a project source directory for use when constructing a `values` inside a [Custom Lua Rule](#ui-rules).

The design intent is to allow user influenced change detection and reproducibility for project files:

- User-influenced change detection: In large projects (ex. monorepos), the project tree can be broken into smaller bundles. Only parts of the build that depend on smaller project bundles will be rebuilt when a project source file changes.
- Reproducibility: A build user does not access the project files directly; the project files are always checksummed and made available through this `request.ui` library.

The `project` argument is the identifier and version for the **end-user's** project. It defaults to `OurProject_Std@0.1.0`. The UI rule may be used by several projects, so the `project` argument is intended to be supplied by the end-user as a [request parameter](#rule-request-documents) to the UI rule. It may be a library id and version (ex. `OurProject_Std@1.0.0`) or a standard module id and version (ex. `OurProject_Std.A.B.SomeModule@1.0.0`). The `project` must belong to the [distribution package and version](#distributions) if the project is distributed. Using the `Our` vendor namespace means the project cannot be distributed, but the project does not need to have a [distribution with keys and version ranges](#distributions).

The `cell` argument is the name of the cell from the [project structure](#project-structure). It defaults to `root`. The generated bundle identifier is `PROJECT_MODULE.Cells.Xyyyyyyy@PROJECT_VERSION` where `PROJECT_MODULE` is the module or library identifier from `project`, `PROJECT_VERSION` is the project version from `project`, and `yyyy` is the lowercase, no-padding, base32-encoded SHA256 checksum of `cell`.

The `patterns` and `excludes` are glob expressions on project files that conform to [Language Server Protocol 3.18 patterns](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.18/specification/#patterns):

- `*` to match zero or more characters in a path segment
- `?` to match on one character in a path segment
- `**` to match any number of path segments, including none
- `{}` to group conditions (e.g. `**​/*.{ts,js}` matches all TypeScript and JavaScript files)
- `[]` to declare a range of characters to match in a path segment (e.g., `example.[0-9]` to match on `example.0`, `example.1`, …)
- `[!...]` to negate a range of characters to match in a path segment (e.g., `example.[!0-9]` to match on `example.a`, `example.b`,
  but not `example.0`)

`excludes` exclude *files* after they have been found by `patterns`.

Regardless of `patterns`, `request.ui.glob` never enumerates `dk0`'s build-state
directories: the signify keys directory (which holds the secret `build.sec`), the
data directory and the cache directory (`t/k`, `t/d` and `t/c` at their default
locations, or their configured locations). This ensures a rule can never route
`build.sec` or other build state into a content-addressed bundle. A user-supplied
`patterns`/`excludes` cannot override this exclusion.

The same project file may belong to different assets.

The specification does not mandate how change detection is implemented. An implementation may scan all the globs at startup, or cache the globbed files and only update them when an invalidation is given to the build system.

The project directory structure will be maintained in the asset. For example, given the project:

```text
src/
  main.c
  media/
    player.c
  db/
    sql.c
  platforms/
    windows.asm
    linux.s
    macos.s
test/
  test-db.c
```

and `patterns = {"src/**/*.c"}`, the asset will have the structure:

```text
src/
  main.c
  media/
    player.c
  db/
    sql.c
```

The return values are the *bundle*, *partial get-bundle command* and the *partial get-asset command*:

- *bundle*: The [bundle](#bundles). For example:

  ```lua
  {
    id = "PROJECT_MODULE.Sources.Xyyyyyyy@PROJECT_VERSION", -- derived bundle id
    listing = {
      origins = {
        {
          name = "...origin...", -- from `request.ui.glob {origin}` argument
          mirrors = { "." } -- `.` is the project directory
        }
      }
    },
    assets = {
      -- one asset per file matched by the glob patterns
      {
        origin = "...origin...",
        path = "<relative path to project file>",
        size = 123000, -- replaced with real size of project file
        checksum = {
          -- replaced with real SHA256
          sha256 = "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
        }
      }
    }
  }
  ```

- *partial get-bundle command*: The partially complete [value shell command](#value-shell-language-vsl) `get-bundle MODULE@VERSION` with `MODULE@VERSION` replaced with a real value. To use the command in subshells, the `-d :` must be added to complete the value shell command.
- *partial get-asset command*: The partially complete [value shell command](#value-shell-language-vsl) `get-asset MODULE@VERSION` with `MODULE@VERSION` replaced with a real value. To use the command in subshells, the `-p PROJECT_SOURCE_FILE -f :file:BASENAME` must be added to complete the value shell command.

Performance consideration: Using the *partial get-asset command* will almost always be more efficient for single file access than *partial get-bundle command*, as the latter may zip up the bundle and then unzip more files than are needed.

Using the bundle could look like the following, where the source code is given as an argument to a compiler:

```lua
function uirules.MyRule(command, request)
  if command == "submit" and continue_ == "start" then
    local bundle, getbundle = request.ui.glob {
      patterns = { "src/**/*.c" }
    }
    return {
      submit = {
        values = {
          forms = {
            {
              id = request.submit.outputid,
              -- ...
              function_ = {
                commands = {
                  "some-programming-language-compiler",
                  -- let's pretend that there is a `-c DIR` option
                  -- to compile everything in a directory
                  "-c",
                  -- using `getbundle` will copy/link all the globbed
                  -- files into an isolated directory
                  "$(" .. getbundle .. " -d :)"
                }
              }
            }
          },
          bundles = {
            bundle
          }
        }
      }
    }
  end
```

#### request.ui.spawn

```lua
request.ui.spawn {
  program = "/bin/echo"
  [, args = { "arg1", "arg2", "..." }]
  [, cwd = "/some/dir"]
  [, envmods = { "+DOTNET_ROOT=/some/dir", "..." }]
}
```

Runs the `program` with the arguments `args...`.
The program will have its environment modified by `envmods` in accordance to [Environment Modifications](#environment-modifications).

The default `cwd` is the user's working directory.

Build system implementations are required to have security controls.

The caller is expected to check the return values. Using the Lua convention `assert(request.ui.spawn { ... })` is sufficient to pass only on exit code zero.
The return values are:

- If the user rejected giving permission to the `spawn`, the three (3) return values are `nil`, an error mesage, and the string `denied`.
- On exit code 0, the two (2) return values are a truthy value and the number `0`.
- On any other exit code, the four (4) return values are `nil`, an error message, the string `exit`, and the exit code number.
- If terminated due to a signal, the four (4) return values are `nil`, an error message, the string `signal`, and the signal number.
- If stopped due to a signal, the four (4) return values are `nil`, an error message, the string `stop`, and the signal number.

That is:

```lua
nil, "The request to trust the rule to launch the program was denied", "denied"
"t", 0
nil, "The program exited with code 55", "exit"
nil, "The program terminated due to signal 15", "signal", 15
nil, "The program stopped due to signal 19", "stop", 19
```

#### request.ui.capture

```lua
result = request.ui.capture {
  program = "gh"
  [, args = { "auth", "status" }]
  [, cwd = "/some/dir"]
  [, envmods = { "+GH_HOST=github.com", "..." }]
  [, max_output_bytes = 16777211]
}
```

Runs the `program` with the arguments `args...`, captures stdout and stderr,
and returns a result table instead of streaming output to the terminal. The
program will have its environment modified by `envmods` in accordance to
[Environment Modifications](#environment-modifications).

`program` may be a `.cmd` or `.bat` file on Windows.

The default `cwd` is the user's working directory.

Like [request.ui.spawn](#requestuispawn), `request.ui.capture` runs a program, so
build system implementations are required to have the same security controls: an
interactive trust confirmation that fails closed (no program is run) when the
rule is not trusted or there is no interactive terminal. A `program` given as a
bare name (such as `gh`) is resolved through the `PATH` environment variable, and
the trust prompt warns about this.

The default maximum captured size is 16777211 bytes for each stream. If the user
rejected giving permission to run the program, the three return values are `nil`,
an error message, and the string `denied`. On process start failure, the three
return values are `nil`, an error message, and the string `error`. On captured
output limit failure, the three return values are `nil`, an error message, and
the string `output-limit`.

On process completion, one table is returned:

```lua
{
  status = "exit" | "signal" | "stop",
  code = 0,
  stdout = "...",
  stderr = "..."
}
```

#### request.ui.checksum

```lua
metadata = request.ui.checksum {
  path = "relative/project/file"
}
```

Calculates metadata for a project-local file. The `path` field must be a
[strictly relative](#strictly-relative-path) project path.

On success, returns a single table:

```lua
{
  sha256 = "...",
  size = 123
}
```

If the file does not exist, the three (3) return values are `nil`, an error
message, and the string `absent`, so a caller can map absence to
[`request.ui.writefile`](#requestuiwritefile)'s `expected_sha256 = false`. Using
the Lua convention `local meta = request.ui.checksum { path = ... }` treats a
missing file as `meta == nil`. On any other failure (a `path` that is not
strictly relative or an I/O error), the three (3) return values are `nil`, an
error message, and the string `error`.

#### request.ui.readfile

```lua
content = request.ui.readfile {
  path = "relative/project/file"
}
```

Reads a project-local file and returns its entire contents as a string. The
`path` field must be a [strictly relative](#strictly-relative-path) project
path, resolved against the user's project directory, the same base as
[`request.ui.checksum`](#requestuichecksum) and
[`request.ui.writefile`](#requestuiwritefile), and **not** the UI rule's
sandbox. Unlike [`request.io.read`](#requestioread), which reads the rule's
discarded working area, `request.ui.readfile` reads the checked-in project
tree.

The contents are returned verbatim as bytes; no newline translation is
performed.

The caller is expected to check the return values. Using the Lua convention
`assert(request.ui.readfile { ... })` is sufficient. The return values are:

- On success, the single return value is the file contents as a string.
- If the file does not exist or is a directory, the three (3) return values are
  `nil`, an error message, and the string `absent`.
- On any other failure (a `path` that is not strictly relative or escapes the
  project, exceeding the size cap, or an I/O error), the three (3) return
  values are `nil`, an error message, and the string `error`.

An optional `max_bytes` field caps the number of bytes read (default 16777211).

#### request.ui.writefile

```lua
request.ui.writefile {
  path = "relative/project/file",
  content = "file contents",
  expected_sha256 = "e3b0c44298fc1c14..."   -- or false to require the file is absent
}
```

Conditionally writes `content` to a project-local file. This is a
compare-and-swap: the write succeeds only if the file's current on-disk content
matches the caller's stated expectation, guarding against a concurrent writer
that changed the file since the caller last inspected it.

`path` must be a [strictly relative](#strictly-relative-path) project path,
resolved against the user's project directory, the same base as
[`request.ui.checksum`](#requestuichecksum) and
[`request.ui.signify`](#requestuisignify).
Missing parent directories are created. Unlike
[`request.io.write`](#requestiowrite), which stages bytes in the rule's
discarded working area, `request.ui.writefile` publishes into the
project tree.

The `expected_sha256` field is **required**; there is no unconditional write:

- A hex SHA-256 string requires the file to currently exist with exactly that
  SHA-256. Obtain it beforehand from [`request.ui.checksum`](#requestuichecksum).
- The boolean `false` or the literal string `false` requires the file to not
  currently exist (a create).

`content` is written verbatim as bytes (encoded UTF-8); no newline translation
is performed, so the caller controls line endings.

The caller is expected to check the return values. Using the Lua convention
`assert(request.ui.writefile { ... })` is sufficient. The return values are:

- If the user rejected giving permission to write the file, the three (3)
  return values are `nil`, an error message, and the string `denied`.
- If the current file does not satisfy `expected_sha256` (it exists with a
  different SHA-256, exists when `false` was required, or is absent when a
  SHA-256 was required), the three (3) return values are `nil`, an error
  message, and the string `conflict`.
- On success, the three (3) return values are a truthy value, the
  [strictly relative](#strictly-relative-path) project path written, and the
  SHA-256 of the newly written content.
- On any other failure (a `path` that is not strictly relative or escapes the
  project, or an I/O error), the three (3) return values are `nil`, an error
  message, and the string `error`.

#### request.ui.signify

```lua
signed = request.ui.signify {
  operation = "sign",
  message = "INDEX",
  signature = "INDEX.sig"
}

verified = request.ui.signify {
  operation = "verify",
  message = "INDEX",
  signature = "INDEX.sig"
}
```

Signs or verifies a project-local file using the [OpenBSD signify build keys](#openbsd-signify-keys)
local to your host and possibly shared with your team.

All path fields must be [strictly relative](#strictly-relative-path) project paths.
On signing, the signature is written to the `signature` path.
On verify failure, the three return values are `nil`, an error message, and the string `verify`.

#### request.ui.sleep

```lua
request.ui.sleep { seconds = 10 }
```

Suspends the UI rule for the requested number of seconds.

#### request.ui.buildpubkey

```lua
local key = request.ui.buildpubkey
```

The contents of the build public key, read in binary mode. This is the same
public key used by [`request.ui.signify`](#requestuisignify) to verify
signatures.

### Lua string library

This mostly Lua 5.4 compatible library provides generic functions for string manipulation, such as finding and extracting substrings, and pattern matching. When indexing a string in Lua, the first character is at position 1 (not at 0, as in C). Indices are allowed to be negative and are interpreted as indexing backwards, from the end of the string. Thus, the last character is at position -1, and so on.

The string library provides all its functions inside the table `string`. Unlike Lua 5.1+, it does *not* sets a metatable for strings where the __index field points to the string table. Therefore, you *cannot* use the string functions in object-oriented style. For instance, string.byte(s,i) *cannot* be written as s:byte(i).

The string library assumes one-byte character encodings.

#### string.byte

`string.byte (s [, i [, j]])`

Returns the internal numeric codes of the characters `s[i]`, `s[i+1]`, ..., `s[j]`. The default value for `i` is 1; the default value for `j` is `i`. These indices are corrected following the same rules of function `string.sub`.

Numeric codes are not necessarily portable across platforms.

#### string.find

`string.find (s, pattern [, init [, plain]])`

The `pattern` is a Lua 2.5 pattern; see [Lua 2.5 §6.2 Patterns](https://www.lua.org/manual/2.5/manual.html#6.2).

Looks for the first match of pattern in the string s. If it finds a match, then find returns the indices of s where this occurrence starts and ends; otherwise, it returns `fail`. A third, optional numeric argument `init` specifies where to start the search; its default value is 1 and can be negative. A true as a fourth, optional argument `plain` turns off the pattern matching facilities, so the function does a plain "find substring" operation, with no characters in pattern being considered magic.

If the pattern has captures, then in a successful match the captured values are also returned, after the two indices.

#### string.format

`string.format (formatstring, ···)`

Returns a formatted version of its variable number of arguments following the description given in its first argument, which must be a string. The format string follows the same rules as the ISO C function `sprintf`. The only differences are that the conversion specifiers and modifiers `F`, `n`, `*`, `h`, `L`, and `l` are not supported and that there is an extra specifier, `q`. Both width and precision, when present, are limited to two digits.

The specifier `q` formats booleans, nil, numbers, and strings in a way that the result is a valid constant in Lua source code. Booleans and nil are written in the obvious way (true, false, nil). Floats are written in hexadecimal, to preserve full precision. A string is written between double quotes, using escape sequences when necessary to ensure that it can safely be read back by the Lua interpreter. For instance, the call

```lua
     string.format('%q', 'a string with "quotes" and \n new line')
```

may produce the string:

```text
     "a string with \"quotes\" and \
      new line"
```

This specifier does not support modifiers (flags, width, precision).

The conversion specifiers `A`, `a`, `E`, `e`, `f`, `G`, and `g` all expect a number as argument. The specifiers `c`, `d`, `i`, `o`, `u`, `X`, and `x` expect an integer.

The specifier `s` expects a string; if its argument is not a string, it is converted to one following the same rules of [tostring](#lua-global-variable---tostring). If the specifier has any modifier, the corresponding string argument should not contain embedded zeros.

The specifier `p` formats the pointer returned by `lua_topointer` in Lua 5.1+, but in this specification an error is raised.

#### string.len

`string.len (s)`

Receives a string and returns its length. The empty string `""` has length 0. Embedded zeros are counted, so `"a\000bc\000"` has length 5.

#### string.lower

`string.lower (s)`

Receives a string and returns a copy of this string with all ASCII uppercase letters changed to lowercase. All other characters are left unchanged.

#### string.rep

`string.rep (s, n [, sep])`

Returns a string that is the concatenation of `n` copies of the string `s` separated by the string `sep`. The default value for `sep` is the empty string (that is, no separator). Returns the empty string if `n` is not positive.

(Note that it is very easy to exhaust the memory of your machine with a single call to this function.)

#### string.sub

`string.sub (s, i [, j])`

Returns the substring of `s` that starts at `i` and continues until `j`; `i` and `j` can be negative. If `j` is absent, then it is assumed to be equal to `-1` (which is the same as the string length). In particular, the call `string.sub(s,1,j)` returns a prefix of s with length `j`, and `string.sub(s, -i)` (for a positive `i`) returns a suffix of `s` with length `i`.

If, after the translation of negative indices, `i` is less than `1`, it is corrected to `1`. If `j` is greater than the string length, it is corrected to that length. If, after these corrections, `i` is greater than `j`, the function returns the empty string.

#### string.upper

`string.upper (s)`

Receives a string and returns a copy of this string with all ASCII lowercase letters changed to uppercase. All other characters are left unchanged.

### Lua stringdk library

#### stringdk.quote_posix_shell

`stringdk.quote_posix_shell (s)`

Returns a string that is quoted as a single word in a POSIX shell.

#### stringdk.quote_value_shell_literal

`stringdk.quote_value_shell_literal (s)`

Returns a string that is quoted as one literal word in the
[Value Shell Language](#vsl-lexical-rules). The returned word evaluates to
the exact text `s`, so any `${...}` sequences inside `s` are treated as data
rather than as VSL expansions.

Examples:

- `stringdk.quote_value_shell_literal ("hello world")` returns
  `'hello world'`
- `stringdk.quote_value_shell_literal ("abc${/}def")` returns
  `'abc${/}def'`

#### stringdk.quote_value_shell_evalable

`stringdk.quote_value_shell_evalable (s)`

Treats `s` as VSL code. It parses `s` as exactly one evalable Value Shell
Language term and then re-emits that term in canonical single-word VSL form.

This means the function validates the syntax of `s`, preserves expandable
constructs such as variables and subshells, and rejects strings that are not
one valid evalable VSL term. If the input is already a canonical evalable
term, the output may be unchanged.

Examples:

- `stringdk.quote_value_shell_evalable ("abc${/}def")` returns
  `abc${/}def`
- `stringdk.quote_value_shell_evalable ("${SRC}")` returns `${SRC}`
- `stringdk.quote_value_shell_evalable ("hello world")` returns two values:
  `nil` and a string explaining why it is not one evalable VSL term
- `stringdk.quote_value_shell_evalable ("${SRC")` returns two values:
  `nil` and a string indicating the VSL syntax is invalid

#### stringdk.quote_windows_batch

`stringdk.quote_windows_batch (s)`

Returns a string that can be inserted verbatim as one whitespace-delimited
word in a [Windows cmd.exe](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/cmd)
command line or Windows batch script command that launches an executable.
In other words, cmd.exe should parse the result as exactly one command word whose value is `s`.

This is intended for building command invocations such as executable paths and arguments. It is not a general-purpose escaping function for every batch-file construct such as `if`, `set`, or other contexts with additional cmd.exe parsing rules.

#### stringdk.sanitizesubpath

`stringdk.sanitizesubpath (s)`

Returns a normalization of the subpath `s` if and only if `s` is a strict subpath; that is:

- `s`, after file path normalization, does not begin with `..`
- `s` has no unsafe file characters

If `s` can't be sanitized, a `nil` and an error string is returned.

This function supports conventional Lua assert checks: `local sanitized = assert(stringdk.sanitizesubpath (s))`.

### Lua table library

This library provides generic functions for table manipulation. It provides all its functions inside the table table.

Remember that, whenever an operation needs the length of a table, all caveats about the length operator apply (see [§3.4.7](https://www.lua.org/manual/5.4/manual.html#3.4.7)).
All functions ignore non-numeric keys in the tables given as arguments.

#### table.concat

`table.concat (list [, sep [, i [, j]]])`

Given a list where all elements are strings or numbers, returns the string `list[i]..sep..list[i+1] ··· sep..list[j]`. The default value for `sep` is the empty string, the default for `i` is 1, and the default for j is the length of the list (ie. `#list` from Lua 5.1+). If `i` is greater than `j`, returns the empty string.

#### table.getn

`table.getn (table)`

**CAUTION** This Lua 5.0 function will be removed at a later date when the conventional Lua 5.1 table length operator `#` is introduced. Be prepared to change your Lua modules and rules when this happens.

Returns the size of a table, when seen as a list. If the table has an `n` field with a numeric value, this value is the size of the table. Otherwise, the size is one less the first integer index with a `nil` value.

Deprecated in Lua 5.1.

#### table.insert

`table.insert (list, [pos,] value)`

Inserts element `value` at position `pos` in `list`, shifting up the elements `list[pos]`, `list[pos+1]`, ···, `list[#list]`. The default value for `pos` is `#list+1`, so that a call `table.insert(t,x)` inserts `x` at the end of the list `t`.

#### table.move

`table.move (a1, f, e, t [,a2])`

Moves elements from the table `a1` to the table `a2`, performing the equivalent to the following multiple assignment: `a2[t],··· = a1[f],···,a1[e]`. The default for `a2` is `a1`. The destination range can overlap with the source range. The number of elements to be moved must fit in a Lua integer.

Returns the destination table `a2`.

Introduced in Lua 5.3.

#### table.pack

`table.pack (···)`

Returns a new table with all arguments stored into keys 1, 2, etc. and with a field "n" with the total number of arguments. Note that the resulting table may not be a sequence, if some arguments are `nil`.

#### table.remove

`table.remove (list [, pos])`

Removes from `list` the element at position `pos`, returning the value of the removed element. When `pos` is an integer between `1` and `#list`, it shifts down the elements `list[pos+1]`, `list[pos+2]`, ···, `list[#list]` and erases element `list[#list]`; The index `pos` can also be `0` when `#list` is `0`, or `#list + 1`.

The default value for `pos` is `#list`, so that a call `table.remove(l)` removes the last element of the list `l`.

#### table.unpack

`table.unpack (list [, i [, j]])`

Returns the elements from the given list. This function is equivalent to

```lua
     return list[i], list[i+1], ···, list[j]
```

By default, `i` is 1 and `j` is #list.

### Lua unified library

This library provides the functions available to unified scripts.

Almost all library functions return a *type constant* as their first
return value. That constant helps text or graphical user interfaces decide
how to print the remaining return values.

#### unified.existingoutput

```shell
% unified.existingoutput {}
'whatever is here like this number'
78
'is printed again including multiline table constructors like'
{
  a=1,
  b=2
}
```

`unified.existingoutput {}` is the contents of the output.

`unified.existingoutput {}`:

- reads the contents of the output as a sequence of Lua values
- returns the values from each Lua chunk

where each value is subject to the restrictions:

- there is at most one value per line
- each Lua value must be a data constructor (ie. `nil`, a string, a number or a table constructor)

In the example above, there are four (4) return values. The first is
a Lua string, the second is a Lua number, the third is a Lua string,
and the fourth is a Lua table.

When the output is a metadata markup block (see
[Metadata syntax](UNIFIED_SCRIPTS.md#metadata-syntax)) rather than Lua value
lines, such as the `\dk.asset(...)` block recorded by
[unified.asset](#unifiedasset):

- each markup converts to a Lua table with the reserved key `tag` bound to
  the markup name, each `name: "text"` attribute bound as a string field, each
  bare attribute bound to the number `1`, each nested attribute list bound to
  a nested table, and each `[argument]` added as a numbered entry holding the
  argument's plaintext
- the output block itself is echoed verbatim, not as the Lua table view, so
  the block is a fixed point under re-runs and `update` stays stable

Any Lua function can use `unified.existingoutput {}` to have a persistent memory of what
happened the last time the Lua function was called.

Or any Lua function can use `unified.existingoutput {}` to read what
was placed manually entered (ex. configuration values) in the output block.

#### unified.sections

```shell
# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0

% unified.sections {}
'sections'
'A literate script example - optional, ignored'
'OurLibrary_Std.A.B.C'
```

`unified.sections {}` is the type constant `section` followed by the ATX headers.

#### unified.scriptmodver

```shell
# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0

% unified.scriptmodver {}
'modver'
'OurLibrary_Std.A.B.C'
'1.0.0'
```

`scriptmodver` (the script module version) is three values:

1. The type constant `modver`.
2. The identifier for the script. Example: `OurLibrary_Std.A.B.C`
3. The version of the script. Example: `1.0.0`

The script module version comes from the previous level 2 ATX header:

```markdown
# A literate script example - optional, ignored

## OurLibrary_Std.A.B.C@1.0.0

... more of the unified script

## OurLibrary_Std.D.E.F@2.0.0

... more of the script that belongs to
... another package in the same library
```

The first word in the level 2 ATX header that is a valid `MODULE@VERSION` becomes the `scriptmodver`.

If there is no valid `MODULE@VERSION` then `scriptmodver` will be the values `nil` and an error message.

#### unified.asset

```shell
# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0

Assumes that the script is .../*.u/run.u and that
.../*.u/data/gawk-5.3.1.tar.gz exists.
  % unified.asset { name="GawkTarball", file="data/gawk-5.3.1.tar.gz" }
  \dk.asset(byteSize: "6264553", checksum: (sha256: "fa41b3a85413af87fb5e3a7d9c8fa8d4a20728c67651185bb49c38a7f9382b1e"))\;

Assumes that the script is .../*.u/run.u and that
.../*.u/user/share/gawk directory exists.
  % unified.asset { name="GawkShare", dir="usr/share/gawk" }
  \dk.asset(byteSize: "123456", checksum: (sha256: "0000003812089120bc2a5d84f9e65cd0c25e4a4d724c80075c357239c74ae904"))\;
```

`unified.asset` loads the local file or directory, and makes a singleton bundle from the asset. The local file or directory must be a [strictly relative path](#strictly-relative-path).

`unified.asset` is available in any unified script that has a section name `<standard module id>@<semantic version>`, including the [workspace script](#workspace-script).

- `name` must be a standard namespace term (ie. begins with a capital letter)
- the bundle id is `<scriptid>.<name>@<scriptver>` where `scriptid` and `scriptver` are from [unified.scriptmodver](#unifiedscriptmodver)
- the origin is named the library id of `scriptid` (ex. `OurLibrary_Std` if `scriptid = OurLibrary_Std.A.B.C`) and has mirrors set to the library cell (ex. `cell://OurLibrary_Std`).

On error, returns `nil` and an error message which are echoed as two Lua
value lines.

On success, the output block is a single `\dk.asset` metadata markup line
(see [Metadata syntax](UNIFIED_SCRIPTS.md#metadata-syntax)) with the
attributes:

1. `byteSize`: the size of the asset in bytes.
2. `checksum`: a nested attribute list with one attribute per checksum
   algorithm (ex. `sha256: "<hex>"`). A future revision may record
   additional checksum algorithms as additional attributes in the list.

The legacy output block form that predates the markup (the three Lua value
lines `'asset'`, the size, and the checksum or a numbered table of checksums)
is still accepted when reading existing output blocks, so already-published
packages keep working; `dk0 update` rewrites it to the markup form.

The singleton bundle in the above example would be:

```json
{
    "id": "OurLibrary_Std.A.B.C.GawkShare@1.0.0",
    "listing": {
        "origins": [
            {
                "name": "OurLibrary_Std",
                "mirrors": [
                    "cell://OurLibrary_Std"
                ]
            }
        ]
    },
    "assets": [
        {
            "origin": "OurLibrary_Std",
            "path": "data/gawk-5.3.1.tar.gz",
            "checksum": {
                "sha256": "fa41b3a85413af87fb5e3a7d9c8fa8d4a20728c67651185bb49c38a7f9382b1e"
            },
            "size": 6264553
        }
    ]
}
```

#### unified.envmod

**Not implemented yet.**

```shell
# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0
### function!

% unified.envmod "+YACC=$(get-object CommonsBase_GNU.Bison@3.8.2
          -s Release.execution_abi -m ./bin/yacc -e bin/yacc -f yacc)"
```

`unified.envmod` adds an [environment modification](#environment-modifications) to the
function `OurLibrary_Std.A.B.C.Fx@1.0.0` (the first valid word in the previous level 2 ATX header, with `Fx` added).

The level 3 ATX header must be `function!`.

#### unified.output

**Not implemented yet.**

```shell
# A literate script example - optional, ignored
## OurLibrary_Std.A.B.C@1.0.0
### function!

% unified.output {}
[Release.Darwin_Arm64 Release.Darwin_x86_64 Release.Windows_x86_64]
include/gawkapi.h

[Release.Darwin_Arm64 Release.Darwin_x86_64]
bin/awk bin/gawk "etc/profile.d/gawk.sh"
"lib/gawk/time.so" "libexec/awk/grcat"

[Release.Windows_x86_64]
bin/awk.exe bin/gawk.exe "lib/gawk/time.dll"
```

`unified.output` defines the files that will be generated in the slot directores by the
function `OurLibrary_Std.A.B.C.Fx@1.0.0` (the first valid word in the previous level 2 ATX header, with `Fx` added).

The example above would be the equivalent of the [form output](#frm---form) in JSON:

```json
"assets": [
  {
    "slots": [
      "Release.Darwin_arm64", "Release.Darwin_x86_64",
      "Release.Windows_x86_64"
    ],
    "paths": [
      "include/gawkapi.h"
    ]
  },
  {
    "slots": [
      "Release.Darwin_arm64", "Release.Darwin_x86_64"
    ],
    "paths": [
      "bin/awk", "bin/gawk",
      "etc/profile.d/gawk.sh",
      "lib/gawk/time.so", "libexec/awk/grcat"
    ]
  },
  {
    "slots": [
      "Release.Windows_x86_64"
    ],
    "paths": [
      "bin/awk.exe", "bin/gawk.exe",
      "lib/gawk/time.dll"
    ]
  }
]
```

The level 3 ATX header must be `function!`.

The shell output is the list of files. That is, the `output`
function slurps (reads all of) the shell output, partitions the output
based on `[Slot1 ... SlotN]` sections, and then splits each section into
whitespace separated words.

Double quotes are required for any filename that has either whitespace or a
double quote. An embedded double quote must be escaped with a second double
quote. That is, `"there is a middle "" double quote"` is the filename
`there is a middle " double quote`.

The return values are either a `nil` and an error message, or the values:

1. The type constant `outputpaths`.
2. A table whose keys are the slots and whose values are path lists. The key is a lexographically ordered list of slots (ex. `{"Release.Darwin_arm64", "Release.Darwin_x86_64"}`). The value is a path list (ex. `{"bin/awk", "bin/gawk"}`).

#### unified.assign

**Not implemented yet.**

```shell
# A literate script example - optional, ignored
## recipes!
% unified.assign { "coreutils",
    "$(get-object CommonsBase_Std.Coreutils@0.2.2 -s Release.Windows_x86_64 -m ./coreutils.exe -f : -e '*')"
  }
VAR:coreutils:=...
```

`unified.assign { "xyz": "a-value" }` sets the variable `xyz` to the value `a-value`
such that in subsequent shell commands in the `recipes!` section like:

```shell
## recipes!
...
### print-value
Prints the value of xyz
$ echo The value is ${VAR:xyz}.
The value is a-value.
```

the `${VAR:xyz}` expands to `a-value`.

The level 2 ATX header must be `recipes!`.

#### unified package target

**Not implemented yet.**

```shell
# A literate script example - optional, ignored
## OurTargets_Std.TheName@0.1.0
### targets!
% local ml = require("NotInriaCaml_Std.Build").at("1.0.0")
NotInriaCaml_Std.Build@1.0.0
% ml.jsonschema["library"]
url
https://www.schemastore.org/ocaml-library.json
% ml.library { name="something", ... }
```

The `ml.library { name="something", ... }` is interpreted as:

```lua
ml.target_library(command, request, continue_)
```

The `.target_library` behaves just like a [Function Rule Function](#function-rules)
except it does not need to be attached to "rules" in `rules, uirules = build.newrules(M)`.

The initial `command` is `submit` and the initial `request` is the table:

```lua
{
  user = {
    context = {
      sections={ "A literate script example", "OurTargets_Std.TheName@0.1.0", "targets!"},
      targetid="OurTargets_Std.TheName",
      targetver="0.1.0"
    },
    doc = { name="something", ... }
  }
}
```

If `.jsonschema` is present and the build implementation supports it, the
inputs to the action (ex. `.library`) will be validated during interpretation with the
JSON schema `.jsonschema["library"]`. The first return value is either `url` or `inline`.

If `.luadefinition` is present and the build implementation supports it, an
IDE (ex. a Visual Studio Code extension based on Markdown Language Server and
Lua Language Server) can load `.luadefinition` into a [Lua .d.lua definition file](https://luals.github.io/wiki/definition-files/) to provide auto-complete.

### Lua workspace globals

Global functions are available to unified scripts in the `workspace` section.

These functions return a *type constant* as their first
return value. That constant helps text or graphical user interfaces decide
how to print the remaining return values.

#### import

Imports a distribution.

```shell
## workspace

  %% import {
  ..   type="TYPE",
  ..   ... depends on TYPE ... }
  \dk.import(type: "TYPE", library: "CommonsBase_Std",
    version: "2.5.202603190707",
    checksum: (blake2b-256: "9d956430ebb347d46e0037e8094bb92b1fcbfa52603394b643685c40b489f7f0",
      sha256: "8e37f1d16259b643fbd3ce447d53e97ef321d96555bf9c3ba23a328108848ec6",
      sha1: "b82ae461dcc46e4aaa3381d9ada2a093e9aa1b49"))\;
```

The `import` will:

- download the distribution metadata
- validate the metadata
- place distribution metadata in the trace store (deprecated; <https://github.com/diskuv/dk/issues/101>)
- place distribution metadata in the source tree

On error, the output block is `nil` and an error message echoed as two Lua
value lines.

On success, the output block is one `\dk.import` metadata markup (see
[Metadata syntax](UNIFIED_SCRIPTS.md#metadata-syntax)) per imported library,
with the attributes:

1. `type`: the value 'TYPE' from `import { type="TYPE", ... }`.
2. `library`: the library that was distributed in the GitHub release.
3. `version`: the distributed version of the library.
4. `checksum`: a nested attribute list with one attribute per checksum
   algorithm of the distribution metadata
   (`.../<LIBRARY>-<VERSION>.values.json`).

The legacy output block form that predates the markup (the three Lua value
lines `'import'`, the type, and a table of records) is still accepted when
reading existing output blocks, so already-published packages keep working;
`dk0 update` rewrites it to the markup form.

However, *if* there is existing output (ie. `CommonsBase_Std@2.5.202603190707`)
and *all* of the `LIBRARY@VERSION` are present in the trace store or source tree,
then the import command is skipped.

> 📢 The import's `LIBRARY@VERSION` outputs in the workspace section, along with the distribution metadata in the source tree, behave like lock files in package managers like `npm` and `cargo`.

An implementation may also place:

1. *lazy* value files in the value store by default to avoid the time and space to download
   every binary artifact from the distribution

##### import type=github-l2

Imports a distribution from a GitHub release.

```shell
## workspace

  %% import {
  ..   type="github-l2",
  ..   repo="OWNER/REPO",
  ..   host="",
  ..   tag="" }
  \dk.import(type: "github-l2", library: "CommonsBase_Std",
    version: "2.5.202603190707",
    checksum: (blake2b-256: "9d956430ebb347d46e0037e8094bb92b1fcbfa52603394b643685c40b489f7f0",
      sha256: "8e37f1d16259b643fbd3ce447d53e97ef321d96555bf9c3ba23a328108848ec6",
      sha1: "b82ae461dcc46e4aaa3381d9ada2a093e9aa1b49"))\;
```

- `host` defaults to `github.com`.
- `repo` are the two URL segments after the host. For example, the `https://github.com/diskuv/dk.git` GitHub project has `repo=diskuv/dk`.
- `tag`, if unspecified, is the latest release tag.

`github-l2` will validate the release using GitHub's SLSA Level 2 attestations.

### Custom Lua Modules

Any Lua script that returns a table is a Lua module that can be imported by other Lua scripts.

The simplest module is:

```lua
-- values.lua
local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
function M.somefunc()
  print('inside somefunc()')
end
return M
```

The `id` field is required, and is the same `MODULE@VERSION` used throughout the build system.

The module above can be imported in another `values.lua` script as follows:

```lua
MyModule = require('MyLibrary_Std.A.B.MyModule')
MyModule = MyModule.at('1.0.0')
if build.is_building then MyModule.somefunc() end
```

See [Script Phases](#script-phases) for why `if build.is_building then ... end` is required to guard expressions.

To avoid conflicts with other modules, an error will be raised if a field is exported that is
a standard namespace term (ex. `SomeModule`).

Keeping your exports lowercased (or at least the first letter is lowercase) is sufficient to satify this restriction.

### Introduction to Custom Lua Rules

Lua rules are Lua functions inside modules that dynamically build other values.

A simple rule `MyRule` is:

```lua
-- values.lua
local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
rules, _ignore_ui_rules = build.newrules(M)
function rules.MyRule(command, request, continue_)
  print('ok')
end
return M
```

The rule above can be run from the command line:

```sh
${dk_build_system} run-function MyLibrary_Std.A.B.MyModule.MyRule@1.0.0 -s Some.Slot -- a=1 b=2
```

or from a subshell in a `values.json` build file:

```json
{
  // ...
  "forms": [
    "function": {
      "commands": [
        "echo",
        "$(run-function MyLibrary_Std.A.B.MyModule.MyRule@1.0.0 -s Some.Slot -- a=1 b=2)"
      ]
    }
  ]
}
```

or imported from another `values.lua` script:

```lua
MyRule = require('MyLibrary_Std.A.B.MyModule.MyRule')
MyRule = MyRule.at('1.0.0')
MyRule.use { a=1, b=2 }
```

### Function Rules

Function rules (ie. `M.fnrules`) are rules that are free to be used everywhere: in `values.json` files and directly by the end-user.

Function rules should be *pure* functions (ie. repeat and get the same results on a different machine) so they do *not* have direct access to changeable project source code directories.

When a distribution script runs a function rule from a `*.values.lua`
scriptmodule, the whole scriptmodule is included in the distributed package.
That is the normal way to distribute Lua-only rule modules that do not also
ship separate `*.values.json[c]` declarations.

By convention, function rule names are prefixed with `F_` (for example
`F_Run` or `F_Untar`). That keeps function rules distinct from interactive rule
names like `Run` in the same scriptmodule.

The form of a function rule function named `YourFreeRule` is:

```lua
local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
rules = build.newrules(M)
function rules.YourFreeRule(command, request, continue_)
  -- your rule here
end
return M
```

The response to a function rule function must match the [dk-rule-response.json schema](../etc/jsonschema/dk-rule-response.json).

The sequence of commands given to the function rule is:

```text
   [command == "declareoutput"]
             |
             |
             v
     [command == "submit"]
```

The next sections describe what each command does.

### Function Rule Command - `declareoutput`

The `declareoutput` command is the build system asking the function rule to declare
the output keys and any static input dependencies *before* the rule adds tasks
to the task graph.

The output keys can be [object](#objects) keys:

```lua
function rules.YourFreeRule(command, request)
  if command == "declareoutput" then
    return {
      -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
      declareoutput = {
        return_objects = {
          -- parse [request.user] to calculate `id` and `slots`
          id = "UserLibrary_Std.A.B.UserModule.OutputObject@1.0.0",
          slots = { "Release.Windows_x86_64", "Release.Darwin_arm64" },
          execution_slot = "Release.execution_abi"
        }
      }
    }
  end
end
```

or an [asset](#assets) key:

```lua
function rules.YourFreeRule(command, request)
  if command == "declareoutput" then
    return {
      -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
      declareoutput = {
        return_asset = {
          -- parse [request.asst] to calculate `id` and `path`
          id = "UserLibrary_Std.A.B.UserModule.OutputAsset@1.0.0",
          path = "some/file"
        }
      }
    }
  end
end
```

Static rule inputs are declared in the same `declareoutput` table:

```lua
function rules.YourFreeRule(command, request)
  if command == "declareoutput" then
    return {
      -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
      declareoutput = {
        return_asset = {
          id = "UserLibrary_Std.A.B.UserModule.OutputAsset@1.0.0",
          path = "some/file"
        },
        input_bundles = {
          { id = "UserLibrary_Std.A.B.SourceBundle@1.0.0" }
        },
        input_assets = {
          { id = "UserLibrary_Std.A.B.SourceBundle@1.0.0", path = "src.tar.gz" }
        },
        input_objects = {
          {
            id = "UserLibrary_Std.A.B.Tool@1.0.0",
            slots = { "Release.Windows_x86_64", "Release.Linux_x86_64" },
            execution_slot = "Release.execution_abi"
          }
        }
      }
    }
  end
end
```

For objects, the `execution_slot` is a literal or [wildcard](#slotnameslotname) slot that translates the current execution platform to one of the `return_object` slots.
Build system implementations will schedule the running of a task for the object that corresponds to the `execution_slot`.

In a [distribution](#distributions), the responsibility of the CI system is to ensure the graph is built on all execution platforms so all slots are available.
Consider, for example, a C language build. The CI system must provide virtual machines or cross-compilers so that C binary artifacts (executables and libraries) are created for all the ABIs.

`input_bundles`, `input_assets`, and `input_objects` are the static dependencies
of the rule itself.

Historical Note: This pattern of declaring the output *before* doing the building was inspired by [Buck2's dynamic dependencies](https://buck2.build/docs/rule_authors/dynamic_dependencies/).

### Function Rule Command - `submit`

The `submit` command is the entry point for the function rule to build artifacts by:

- add values to the valuestore and tasks to the task graph
- ask the build system for more information

All responses must set the field `submit`. That is:

```lua
return { submit = { values = ..., expressions = ..., commands = ..., andthen = ... } }
```

The fields that go into `submit.values` and `submit.andthen` are enumerated in the authoritative [dk-rule-response.json schema](../etc/jsonschema/dk-rule-response.json).

The `forms[].function` field may be written as `forms[].function_` to avoid conflicts with the Lua `function` keyword.

Consider the following response to a `submit` command:

```lua
return {
  submit = {
    values = {
      forms = {
        {
          id = "OurExample_Std.SomeModule@0.1.2",
          function_ = {
            commands = {
              "$(get-object CommonsBase_Std.Coreutils@0.2.2 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f coreutils.exe -e '*')",
              "sort",
              "--output",
              "${SLOT.request}/sorted-file",
              assert(request.user.filename, "please provide `filename=FILENAME`")
            }
          }
        }
      }
    },
    expressions = {
      files = {
        sorted_file =
          "$(get-object OurExample_Std.SomeModule@0.1.2 -s ${SLOTNAME.Release.execution_abi} -m ./sorted-file -f :file)"
      }
    },
    commands = {
      {"run-function", "OurExample_Std.SomeRule@3.4.5"}
    },
    andthen = {
      continue_ = {
        state = "have-sorted-file",
        passthrough = { someconstant = "the constant" }
      },
    }
  }
}
```

When the build system sees that response, the following sequence occurs:

1. the `values` are treated as if it were a new `values.json` file
2. all the `expressions.strings` are evaluated and will be made available as Lua strings in `request.continued`
3. all the `expressions.files` are evaluated and will be made available as readable file objects in `request.continued`
4. all the `expressions.dirs` are evaluated and will be made available as readable directory objects in `request.continued`
5. all the `commands` are evaluated
6. the rule function will get a callback (ie. `andthen`)

All four steps (`values`, `expressions`, `commands`, `andthen`) were optional.

The build system will perform the callback of the `andthen` with the following parameters:

```lua
-- YourFreeRule(command, request, continue_)
YourFreeRule(
  -- command
  "submit",
  -- request
  {
    user = ..., -- request.user table
    io = ..., -- request.io library
    continued = {
      -- anything in 'andthen.continue_.passthrough' is given literally to the rule
      someconstant = "the constant",
      -- anything in 'expressions.files', 'expresionss.dirs' and `expressions.strings`
      -- is evaluated and their responses given to the rule
      sorted_file = "...path to sorted-file..."
    }
  },
  -- continue_
  "have-sorted-file"
)
```

### UI Rules

UI rules (ie. `uirules`) are rules that:

- only an end-user can run these rules; using UI rules inside a `values.json[c]` file will fail the build
- the `dialog` subcommand runs UI rules, while `run-function` is reserved for function rules
- interact with the end-user through a console or a graphical user interface
- only one UI rule may run at a time even if the build system implementation parallelizes noninteractive rules
- have access to the project source code directories through the [request.ui](#lua-requestui-library) library

UI rules are *impure* functions that have outputs that are not reproducible because they have direct access to changing project source code:

- [request.ui.glob](#requestuiglob) reads from the project source tree
- [request.ui.spawn](#requestuispawn) and [request.ui.capture](#requestuicapture) default to running in the user's working directory unless a `cwd` is supplied

Because UI rules are impure, UI rules are never cached. With [request.ui.glob](#requestuiglob) these impure UI rules can take immutable snapshots of the project source code (ie. [assets](#assets)); these immutable assets can be used directly or passed to *pure* [function rules](#function-rules).

The form of a UI rule function named `YourUiRule` is:

```lua
local M = { id='MyLibrary_Std.A.B.MyModule@1.0.0' }
rules, uirules = build.newrules(M)
function uirules.YourUiRule(command, request, continue_)
  -- your rule here
end
return M
```

Please see [Function Rules](#function-rules) for a description of the `command`, `request` and `continue_` arguments.

The response to a UI rule function must match the [dk-rule-response.json schema](../etc/jsonschema/dk-rule-response.json).

The sequence of commands given to the UI rule is:

```text
   [command == "submit"]
             |
             |
             v
     [command == "ui"]
```

The next sections describe what each command does.

### UI Rule Command - `submit`

The `submit` command is the entry point for the UI rule to build artifacts by:

- adding values to the valuestore and tasks to the task graph
- ask the build system for more information

The responses to the command are the same as [Function Rule Command - `submit`](#function-rule-command---submit).

### UI Rule Command - `ui`

The `ui` command is executed after the `submit` command.

Its purpose is to let UI rules do something with the built artifacts from the `submit` command.

A typical action would be to run the built artifact or display a summary of the artifacts.

### Rule Argument - `request`

The details about the build request will be available as follows:

| Field                          | Commands Applicable To                                                | What                                                                                                          |
| ------------------------------ | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `request.user`                 | [Function Rule submit](#function-rule-command---submit)               | Rule request document translated from the arguments to `run-function`                                         |
|                                | [UI Rule submit](#ui-rule-command---submit)                           | ... The request document is described later in the [Rule Request Documents](#rule-request-documents) section. |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
|                                | *but not* [Embedded File Scripts](#embedded-file-scripts)             |                                                                                                               |
| `request.rule`                 | [Function Rule declareoutput](#function-rule-command---declareoutput) | [request.rule](#lua-requestrule-library)                                                                      |
|                                | [Function Rule submit](#function-rule-command---submit)               |                                                                                                               |
|                                | [UI Rule submit](#ui-rule-command---submit)                           |                                                                                                               |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
| `request.execution`            | [UI Rule submit](#ui-rule-command---submit)                           | [request.execution](#lua-requestexecution-library)                                                            |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
| `request.io`                   | [Function Rule submit](#function-rule-command---submit)               | [request.io](#lua-requestio-library)                                                                          |
|                                | [UI Rule submit](#ui-rule-command---submit)                           |                                                                                                               |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
| `request.continued`            | [Function Rule submit](#function-rule-command---submit)               | The last continuation. See [continue_ argument](#rule-argument---continue_)                                   |
|                                | [UI Rule submit](#ui-rule-command---submit)                           |                                                                                                               |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
|                                | [Embedded File Scripts](#embedded-file-scripts)                       |                                                                                                               |
| `request.ui`                   | [UI Rule submit](#ui-rule-command---submit)                           | [request.ui](#lua-requestui-library)                                                                          |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
| `request.ui`                   | [UI Rule submit](#ui-rule-command---submit)                           | [request.ui](#lua-requestui-library)                                                                          |
|                                | [UI Rule ui](#ui-rule-command---ui)                                   |                                                                                                               |
| `request.srcfile.id`           | [Embedded File Scripts](#embedded-file-scripts)                       | Asset id of the [Lua is embedded in it](#embedded-file-scripts), if any                                       |
| `request.srcfile.bundle`       | [Embedded File Scripts](#embedded-file-scripts)                       | [Bundle](#assets) of the [Lua is embedded in it](#embedded-file-scripts), if any                              |
| `request.srcfile.getasset`     | [Embedded File Scripts](#embedded-file-scripts)                       | The shell command [get-asset](#get-asset-moduleversion-file_path--f-file---d-dir)                             |
|                                |                                                                       | to get the asset in `request.srcfile.bundle`.                                                                 |
|                                |                                                                       | The `-f BASENAME` or `-f :` argument must be added.                                                           |
| `request.submit.outputid`      | [Function Rule submit](#function-rule-command---submit)               | `MODULE@VERSION` given by [Function Rule declareoutput](#function-rule-command---declareoutput)               |
| `request.submit.outputmodule`  | [Function Rule submit](#function-rule-command---submit)               | `MODULE` in `MODULE@VERSION` given by [Function Rule declareoutput](#function-rule-command---declareoutput)   |
| `request.submit.outputversion` | [Function Rule submit](#function-rule-command---submit)               | `VERSION` in `MODULE@VERSION` given by [Function Rule declareoutput](#function-rule-command---declareoutput)  |

It is important to check whether user provided arguments have been provided. Consider using expressions like the following to check that they are set:

```lua
assert(request.user.filename, "Please provide `filename=FILENAME`")
```

### Rule Argument - `continue_`

The `continue_` argument is the state of a request. A request's boundaries is the start and stop of a single `run-function` command submitted by a user or a [precommand](#precommands) or a [subshell](#subshell-run-function-moduleversion----cli_form_doc).

The `continue_` value will be:

| Value        | Commands Applicable To                                  | Notes                                       |
| ------------ | ------------------------------------------------------- | ------------------------------------------- |
| `start`      | [Function Rule submit](#function-rule-command---submit) | The first `submit` command will begin       |
|              | [UI Rule submit](#ui-rule-command---submit)             | in the `start` state.                       |
|              | [Embedded File Scripts](#embedded-file-scripts)         |                                             |
| *last value* | [Function Rule submit](#function-rule-command---submit) | Subsequent `submit` commands for the        |
|              | [UI Rule submit](#ui-rule-command---submit)             | same request will use the last `submit`     |
|              | [Embedded File Scripts](#embedded-file-scripts)         | response's `submit.andthen.continue_.state` |
|              |                                                         | field                                       |
| Lua `nil`    | [UI Rule ui](#ui-rule-command---ui)                     |                                             |

Since rules block the build system, [rules must be fast](#rule-requirements) and often rules are broken into small steps using a state machine.

For example, a rule may need to sort a large file. It would be terrible for performance if a build system capable of parallelism was blocked to sort a file. Instead, the following states can be used:

```text
       [start]
          |
          |
          v
   [have-sorted-file]
          |
          |
          v
       [done]
```

When the rule sees `continue_=="start"`, it can return [subshell](#subshells) expression to the build system to fetch a `sort` tool from the [uutils coreutils](https://uutils.github.io/coreutils/docs/utils/sort.html) project. Something like:

```lua
if command == "declareoutput" then
  local symbol = request.rule.generatesymbol()
  return {
    declareoutput = {
      return_asset = {
        id = "OurTest_Exec." .. symbol .. "@1.0.0",
        path = "SHA256.sig"
      }
    }
  }
elseif command == "submit" && continue_ == "start" then
  return {
    -- "$schema" = "https://diskuv.com/dk/schema/dk-rule-response-1.0.json",
    submit = {
      values = {
        forms = {
          {
            id = request.submit.outputid,
            function_ = {
              commands = {
                "$(get-object CommonsBase_Std.Coreutils@0.2.2 -s ${SLOTNAME.Release.execution_abi} -m ./coreutils.exe -f :exe:coreutils.exe)",
                "sort",
                "--output",
                "${SLOT.request}/sorted-file",
                -- the file to sort is provided by the user
                assert(request.user.filename, "provide `filename=FILENAME` on the command line")
              }
            }
          }
        }
      },
      expressions = {
        files = {
          sorted_file =
            "$(get-object " .. form_id .. " -s ${SLOTNAME.Release.execution_abi} -m ./sorted-file -f :file)"
        }
      },
      andthen = {
        continue_ = {
          state = "have-sorted-file"
        },
      }
    }
  }
end
```

The [Function Rule Command - `submit` section](#function-rule-command---submit) describes in detail how the build system interprets the response.
The salient part of the example is the `andthen` is a signal to the build system to call the rule function again.

The rule function will be called back with `continue_ = "have-sorted-file"`; when that happens, the rule should do something useful with the sorted file.
For this example we just print the file.

```lua
if continue_ == "start" then
  -- ...
elseif continue_ == "have-sorted-file" then
  printf("The sorted file is at: %s\n", request.continued.sorted_file)
  return {
    -- an empty submit table means the rule is done.
    submit = { }
  }
end
```

### Rule Request Documents

The request to [Custom Lua Rules](#introduction-to-custom-lua-rules) is always a JSON document.

In the introduction example of [Custom Lua Rules](#introduction-to-custom-lua-rules):

```lua
MyRule = require('MyLibrary_Std.A.B.MyModule.MyRule')
MyRule = MyRule.at('1.0.0')
MyRule.use { a=1, b=2 }
```

the JSON document was converted from the Lua table `{ a=1, b=2 }` into:

```json
{ "a": 1, "b": 2 }
```

See [jsondk.encode](#jsondkencode) for how Lua values are converted to JSON.
However, for rule requests the `jsondk.null` value is **never** encoded.
That means a Lua `nil` is considered equivalent to a missing value.

The introduction example also submitted a request to a rule through the command line:

```sh
${dk_build_system} run-function MyLibrary_Std.A.B.MyModule.MyRule@1.0.0 -s Some.Slot -- a=1 b=2
```

Those command line arguments `a=1 b=2` get converted into the same JSON document as before:

```json
{ "a": 1, "b": 2 }
```

The conversion of command line arguments follows the withdrawn but still useful [W3C HTML JSON Forms specification]:

- `... -- name=Jane` creates the request document `{"name":"Jane"}`
- `... -- pet[species]=Dahut kids[0]=Ashley` creates the request document `{"pets":{"species":"Dahut"},"kids":["Ashley"]}`

Build systems are free to accept the form document directly from a HTML form as defined in [W3C HTML JSON Forms specification] or directly from a JSON document.

[W3C HTML JSON Forms specification]: https://www.w3.org/TR/html-json-forms

### Embedded File Scripts

The body of a Lua [UI rule function](#ui-rules) (called the **guest Lua script**) may be embedded as comments at the bottom of a larger file (called the **host script**).

Consider the following C# *host* script:

```csharp
Console.WriteLine("This is running in C#!");
// local X = require("CommonsBase_Dotnet.SDK"); X = X.at("10.0.100-rc.2.25502.107")
// return X.run { ctx = ctx }
// !dk!s
```

It contains the *guest Lua script* extracted the bottom comments:

```lua
local X = require("CommonsBase_Dotnet.SDK"); X = X.at("10.0.100-rc.2.25502.107")
return X.run { ctx = ctx }
```

The guest Lua script is located using the `!dk!` marker that must appear on one of the last two *nonblank* lines within 16K of the end of the file.

> Aside: The `s` in `!dk!g` means to extract the guest Lua script from the surrounding **s**lash (`//`) comments. It is one of *many* possible codes to integrate with the most popular programming languages (see [Embedded Language Codes](#embedded-language-codes) section).

The build system is responsible for:

1. Creating an asset from the host script.
2. Converting the guest Lua script into a [UI rule function](#ui-rules):

   ```lua
   local M = { id = "... some generated id ..." }
   _rules, uirules = build.newrules(M)
   function uirules.EmbeddedFileScript(command, request, continue_)
     request.srcfile = { --[[ ... the host script asset ... ]] }
     local ctx = { command = command, request = request, continue_ = continue_, arg = arg }
     -- this is the guest Lua script
     local X = require("CommonsBase_Dotnet.SDK"); X = X.at("10.0.100-rc.2.25502.107")
     return X.run { ctx = ctx }
   end
   return M
   ```

   The [Behavior of Embedded Lua](#behavior-of-embedded-lua) describes `request.srcfile` in more detail.

3. Running the above UI rule function.

#### Behavior of Embedded Lua

The guast Lua script script will have the variables:

| Global      | What                                                                                  |
| ----------- | ------------------------------------------------------------------------------------- |
| `arg`       | Command line arguments (see below)                                                    |
| `command`   | At first [`submit`](#ui-rule-command---submit) and then [`ui`](#ui-rule-command---ui) |
| `request`   | The request table (see below)                                                         |
| `continue_` | The state. See [Rule Argument - `continue_`](#rule-argument---continue_)              |
| `ctx`       | The table:                                                                            |
|             | `{ command=command, request=request, continue_=continue_, arg=arg }`                  |

The command line arguments, if any, will be the global table named `arg` that conforms to the [Lua 5.4 "arg" library](https://www.lua.org/manual/5.4/manual.html#7).

The `request` table is available as:

- `request.io`: the [request.io](#lua-requestio-library)
- `request.submit`: This table will be empty.
- `request.user`: This table will be empty. This is in contrast to the [non-embedded UI rule](#rule-argument---request) where the command line arguments would be converted into [Rule Request Documents](#rule-request-documents).
- `request.srcfile`: An information table about the source file that contains the embedded Lua.
- `request.srcfile.id`: An asset identifer unique to the source file.
- `request.srcfile.basename`: The basename of the source file.
- `request.srcfile.bundle`: The [bundle](#bundles). For example:

  ```lua
  {
    id = "... value of request.srcfile.id ...",
    listing = {
      origins = {
        {
          name = "run",
          mirrors = { "selfasset://run" }
        }
      }
    },
    assets = {
      {
        origin = "run",
        path = "<basename_of_source_file>-<short_hash_of_source_file>",
        size = 151, -- replaced with real size
        checksum = {
          -- replaced with real SHA256
          sha256 = "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
        }
      }
    }
  }
  ```

- `request.srcfile.getasset`: The partially complete [value shell command](#value-shell-language-vsl) `get-asset MODULE@VERSION -p PATH` with `MODULE@VERSION` and `PATH` replaced with real values. To use the command in subshells, the `-f BASENAME` or `-f :` must be added to complete the value shell command.

The algorithm is:

1. Add a `values.lua` file to the valuestore that is a wrapper around the [recognized embedded Lua](#recognizing-embedded-lua):

   ```lua
   -- TheUniqueId replaced with a string based on the SHA-256 of the file
   -- that contains the embedded Lua.
   local M = { id = 'OurScript_Std.XTheUniqueId@0.1.0' }
   _rules, uirules = build.newrules(M)
   function uirules.Run(command,request,continue_)
     request.srcfile = request.srcfile or {}
     request.srcfile.id = "..."
     request.srcfile.bundle = {} -- ... it is populated
     -- embedded Lua goes here
   end
   return M
   ```

2. Add the `arg` table as a global variable populated with the actual command line arguments.
3. Add libraries and statements given to the Lua interpreter (`-e` and `-l` options)
4. Run the equivalent of `run-function OurScript_Std.XTheIdentifier.Run@0.1.0` with no arguments. That runs the rule.
5. If `-i` given to the Lua interpreter, start a REPL.

#### Embedded Language Codes

The code table below is organized with the following character meanings:

- vertical: `S` is start, `I` is interior, `E` is end
- horizontal: `S` is start, `E` is end

`<sp>` means the space (ASCII 32) character.
`<sp 6>` means six (6) spaces.
`(*)` means no whitespace at start of pattern.

Codes are named after the patterns not the programming language unless the latter is unambiguous or historically pervasive.

| Code | Languages        | SS               | IS                | EE             |
| ---- | ---------------- | ---------------- | ----------------- | -------------- |
| h    | PowerShell       |                  | `#<sp>`           |                |
|      | + Python + Perl  |                  |                   |                |
|      | + POSIX shell    |                  |                   |                |
|      | + Ruby + R       |                  |                   |                |
|      | + Assembly       |                  |                   |                |
|      | + PHP            |                  |                   |                |
| h    | same as above    |                  | `#`               |                |
| p    | OCaml, Pascal    | `(*`             |                   | `*)`           |
| jd   | Java, C, C++     | `/*`             | `*<sp>`           | `*/`           |
|      | + JavaScript     |                  |                   |                |
|      | + Typescript     |                  |                   |                |
| c    | C, C++, C#, Java | `/*`             |                   | `*/`           |
|      | + JavaScript     |                  |                   |                |
|      | + Go + SQL       |                  |                   |                |
|      | + Rust + Swift   |                  |                   |                |
|      | + Dart           |                  |                   |                |
|      | + Assembly       |                  |                   |                |
|      | + PHP + Kotlin   |                  |                   |                |
| s    | C#               |                  | `///`             |                |
| s    | C, C++, C#, Java |                  | `//<sp>`          |                |
|      | + JavaScript     |                  |                   |                |
|      | + Go + Dart      |                  |                   |                |
|      | + Rust + Swift   |                  |                   |                |
|      | + PHP + Kotlin   |                  |                   |                |
| s    | same as above    |                  | `//`              |                |
| db   | Lua              | `--[[`           |                   | `--]]`         |
| d    | Lua, SQL, Ada    |                  | `--<sp>`          |                |
|      | + Haskell        |                  |                   |                |
| d    | same as above    |                  | `--`              |                |
| hs   | Haskell          | `{-`             |                   | `-}`           |
| tq   | Python           | `"""`            |                   | `"""`          |
| tq   | Python           | `'''`            |                   | `'''`          |
| a    | Visual Basic     |                  | `'<sp>`           |                |
| a    | Visual Basic     |                  | `'`               |                |
| dos  | Visual Basic     |                  | `Rem<sp>`         |                |
|      | + Windows batch  |                  |                   |                |
| dos  | Windows batch    |                  | `@REM<sp>`        |                |
| dos  | Windows batch    |                  | `::<sp>`          |                |
| dos  | Windows batch    |                  | `::`              |                |
| pod  | Perl             | (*) `=begin<sp>` |                   | (*) `=end<sp>` |
| pod  | Perl             | (*) `=for<sp>`   |                   | (*) `=cut`     |
| rb   | Ruby             | (*) `=begin`     |                   | (*) `=end`     |
| x    | XML + HTML       | `<!--`           |                   | `-->`          |
|      | + Markdown       |                  |                   |                |
| sc   | Assembly         |                  | `;<sp>`           |                |
| sc   | Assembly         |                  | `;`               |                |
| e    | Fortran          |                  | `!<sp>`           |                |
| e    | Fortran          |                  | `!`               |                |
| sg   | COBOL            |                  | (*) `<sp 6>*<sp>` |                |
| sg   | COBOL            |                  | (*) `<sp 6>*`     |                |
| sg   | COBOL            |                  | (*) `<sp 6>/<sp>` |                |
| sg   | COBOL            |                  | (*) `<sp 6>/`     |                |
| sg   | COBOL            |                  | `*><sp>`          |                |
| sg   | COBOL            |                  | `*>`              |                |
| ps   | PowerShell       | `<#`             |                   | `#>`           |
| sh   | POSIX shell      | `: <<'END'`      |                   | (*) `END`      |

Pascal curly brace comment delimiters, Fortran 77 fixed source delimiters, and COBOL inline comments are not supported.
Nested comments are also not supported.

#### Recognizing Embedded Lua

The algorithm to recognize embedded Lua is:

- Strip any blank lines from the end of the file.
- For each line `L` of the source code, starting from the last line and going backwards to the second-last line (that is, only consider last two non-blank lines):
  - Let `Lafter` be the line after `L` (that is, `Lafter` is one line nearer to the end of the file), or the empty line if there `L` is the last line.
  - NEXT ROW: For each row `R` in the code table:
    - Let `N(s)` be a recognizer that matches leading whitespace followed by the literal `IS(R)` anchored to the beginning of the line `s`.
    - Let `O(s)` be a recognizer that matches the literal `EE(R)`. If there is a `(*)` marker with `EE(R)` the match must be anchored to the beginning of the line `s`.
    - Let `P(s)` be a recognizer that matches the concatenation of `!dk!` and the `Code(R)` cell` anywhere in the line `s`.
    - Let `Q(s)` be a recognizer that matches the literal `SS(R)`. If there is a `(*)` marker with `SS(R)` the match must be anchored to the beginning of the line `s`.
    - RECOGNIZE BLOCK: Does `P` recognize `L` and does `O` recognize either `S` or `Lafter`? That is, is `P(L) && (O(L) || O(Lafter))` true?
      - Let `Lq` be the line before `L` that is recognized by `Q` (that is, `Lq` is one line or more nearer to the start of the file). If no `Lq` found within 16KB of `L`, skip to RECOGNIZE LINES.
      - All lines prior to `L` but after the `Lq` are considered the rule body.
        - If `IS(R)` is set in the code table with `(*)` marker, the literal `IS(R)` is removed if at the start of the line.
        - If `IS(R)` is set in the code table without `(*)` marker, the lines are trimmed of leading whitespace and the literal `IS(R)` is removed if at the start of the line.
        - The lines are also stripped of a trailing carriage return, if any.
        - DONE.
    - RECOGNIZE LINES: Does `N` recognize `L`? That is, is `N(L)` true?
      - Let `Lnotn` be the line before `L` that is *not* recognized by `N` (that is, `Lnotn` is one line or more nearer to the start of the file). If no `Lnotn` found within 16KB, continue to NEXT ROW.
      - All lines after `Lnotn` before `L` are considered the rule body.
        - If `IS(R)` has `(*)` marker, the literal `IS(R)` is removed from the start of the line.
        - If `IS(R)` has no `(*)` marker, the lines are trimmed of leading whitespace and the literal `IS(R)` from the start of the line.
        - The lines are also stripped of a trailing carriage return, if any.
        - DONE.
- If not DONE at this point, the file does not have an embedded Lua UI function body.

This evaluation strategy can dedent one (1) space when there are multiple rows with the same code. For example, the code `h` (POSIX shell, etc.) has a row with `IS=#<sp>` and a lower row with `IS=#`.

That means the embedded Lua in the file:

```sh
#!/bin/sh
echo 'In the beginning'
exit 0
# return {}
# !dk!h
```

is the following line which has been correctly dedented by one space:

```lua
return {}
```

### Writing Lua Rules

#### Rule Requirements

A - RULE NAMING

Rule names must be standard namespace terms so they can be appended to
the module id to create a new, still-valid module id.

> Keeping the rule names with the first letter capitalized and
no underscores is sufficient to satify this restriction.

B - RETURNED FIELDS

The basic syntax for making a conventional Lua module is:

```lua
local M = {}

-- you: add things to "M". For example,
--   rules, uirules = build.newrules(M)

return M
```

The `M.id` field is required for all modules.

The `M.fnrules` field is populated by [build.newrules](#buildnewrules), and is required for all modules that export *free* rules.
By convention the local variable is named "rules".

Likewise, the `M.uirules` field is populated by [build.newrules](#buildnewrules), and is required for all modules that export *interactive* rules.
By convention the local variable is named "uirules".

C - PERFORMANCE

Rules **must be fast** as they block the build system. Use continuations to delegate all the I/O intensive work to the build system.

D - LEXICAL STRUCTURE

During the [`VALUESCAN` phase](#evaluation) the `values.lua` files are scanned for rules in the procedure described in the [Script Phases](#script-phases).

The practical implications are that the global scope should only be used for:

1. `require` function calls
2. defining functions:

   ```lua
   local M = {}
   function M.somefunc() print('VALUESCAN does not execute code inside functions') end
   ```

3. the final `return` statement

Anything that does not fit the above pattern should be guarded in a `build.is_building` condition:

```lua
if build.is_building then
  -- VALUESCAN will not execute code inside this code block
end
```

### Error Handling in Rules

Lua has both [assert(condition)](#lua-global-variable---assert) and [error "message"](#lua-global-variable---error) to raise errors. While using these functions are okay, especially for serious errors, these will expose Lua stack traces to your end-user.

The conventional way to indicate an error does not print a Lua stack trace. The convention is to return two values from a rule, the first of which is a `nil` and the second is the error message:

```lua
local M = {}
rules = build.define_rules(M)
function rules.MyRule(command,request)
  -- an error happened
  return nil, "this is the error message"
end
return M
```

Best Practices:

- Use `nil, "error ..."` for errors where the user was at fault (the person who submitted a request to the rule). The user forgetting to provide a required field is an example.
- Use `assert` and `error` defensively in preconditions, invariants and postconditions to catch programming errors. The resulting stack trace can be copy-pasted into a bug report by the user so you can fix the programmer error.

### Form Document

> 🚧 This section is still under construction.

Information is supplied to a rule as a JSON document.

The primary way today to supply this JSON document is through the command line syntax `run-function MODULE@VERSION -- CLI_FORM_DOC`, where **CLI_FORM_DOC** is a CLI-based recipe to construct a JSON document.

The form has a `options` JSON object to describe how the JSON document submitted to a form maps to command line options, arguments and variables. *nit: This should be "command==queryschema" given to rule ... it has nothing to do with the misnamed 'form' object in values.json!*

The top-level fields of the form document are available in variables:

- `${PARAM.fieldname}` is the text of the form field named `fieldname`, but it will error if the field is not a JSON string
- `${PARAMFILE.fieldname}` is the file path to the JSON value of the form field named `fieldname`

The form document also contributes to the command line invocation of the form's `function`, if it has one.

> Key Concept: The **group** is a layout of command line options and arguments that covers both the order of options and arguments, and also breaks like `--` or subcommand names in the command line.

#### Form Command Line

The *command* is one array item of the form `[<program>, <arg1>...]` in `"function": { "commands": [ [program1, args1...], [program2, args2...], ... ] }`.
Each *command line* is constructed as the concatenation of:

1. The `<program>, <arg1>...` for a single command.
2. The `{"options": "fields": [...]}` without any `group` field
3. The arguments in `groups[0]` (if any)
4. The `{"options": "fields": [...]}` with a `group: 0` field (if any)
5. The arguments in `groups[1]` (if any)
6. The `{"options": "fields": [...]}` with a `group: 1` field (if any)
7. ... and so on up to and including group 9
8. If `{"options": "document": {...}}` is present, an option and a location of a file containing the entire JSON form document

If `<program>` is a relative path, it is resolved against the function working
directory before spawning on all platforms.

On Windows, that command line is rendered using `cmd.exe` command-word quoting
and spacing rules because `CreateProcessW` takes one command-line string rather
than a native argv array. The `["--cmd.exe", "/c", "<string>"]` special form
described above is the exception: it builds one explicit `cmd.exe /c "..."`
payload from the supplied string with its own inner quoting rules.

#### Option Groups

Groups are necessary when you want some options and arguments to go before or after a `--` seperator:

```sh
cmake -E rm -f -- file1 file2
```

or if you want some options and arguments to go before or after a subcommand:

```sh
git -C some_directory log --oneline
```

or if you need to order some options like how `-L` is required to be first:

```sh
find /home/user -L -name "*.log" -type f -exec rm {} \;
```

Since Windows especially but all operating systems have limits on the size of the command line arguments, the schema may specify a `responsefile` which consolidates all of the command line arguments at the end into a single file that can be read by the program (the first argument of the function `args`). Both MSVC and clang support these responsefiles.

## Data Flow

### Task Model

The smallest unit of a build is a *task* identified by a *key* and, on successful build, resulting in a *value*.

The **keys** represent the parameters to [Value Shell Commands](#value-shell-language-vsl).

The **values** are discussed in [Values](#values).

**Tasks**, the computations that produce a value from a key, are all built into the build system except for [Custom Lua Rules](#introduction-to-custom-lua-rules).

A task may depend on zero or more keys. For example, the form task `CommonsBase_Shell.Pwsh@7.5.4` defined by:

```json
{
  "forms": [
    {
      "id": "CommonsBase_Shell.Pwsh@7.5.4",
      "function": {
        "commands": [
            "$(get-object CommonsBase_Dotnet.SDK@10.0.100-rc.2.25502.107 -s ${SLOTNAME.Release.execution_abi} -d :)/dotnet${.exe.execution}",
            "tool",
            "install",
            "PowerShell",
            //...
        ]
      }
    }
  ]
}
```

depends on the object key `CommonsBase_Dotnet.SDK@10.0.100-rc.2.25502.107`.

We say that the object key `CommonsBase_Dotnet.SDK@10.0.100-rc.2.25502.107` is an immediate dependency of object key `CommonsBase_Shell.Pwsh@7.5.4`.

The tasks and their dependencies form a **task graph**.

The shape of the task graph is:

1. *optional layer*. the incoming nodes of the task graph are impure rules ([UI rules](#ui-rules)). These nodes may depend on nodes in a lower layer.
2. *optional layers*. the interior nodes of the task graph are [objects](#objects) and **pure** rules (ie. [function rules](#function-rules)). These nodes may depend on nodes in the same layer or below.
3. the leaf nodes of the task graph are the immutable [bundles and assets](#assets). These nodes have no dependencies.

That shape is enforced through the edges (the dependencies) allowed in the task graph (todo: incomplete, inaccurate):

| Value Type From | Value Type To | Why                                                 |
| --------------- | ------------- | --------------------------------------------------- |
| `a`             | `j`           | Rebuild bundle if contents of `values.json` changes |
| `a`             | `v`           | Rebuild bundle if parsed `values.json` changes      |
| `o`             | `j`           | Rebuild form if contents of `values.json` changes   |
| `o`             | `v`           | Rebuild form if parsed `values.json` changes        |
| `p`             | `j`           | Rebuild asset if contents of `values.json` changes  |
| `p`             | `v`           | Rebuild asset if parsed `values.json` changes       |

### Trace Store

Each time a task is executed, the following items are captured into a single **trace**:

- the key of the task
- the successful [value](#values) of the task
- the keys of the task's immediate dependencies
- a SHA256 digest of the values of the task's immediate dependencies

The *key* is one of two types:

- A *module key* is what you -- the user -- specify in a shell command as the MODULE_ID and SLOT or PATH in the [Value Shell Language](#value-shell-language-vsl). The module key can be large for `run-function` since its parameters includes a JSON request.
- A *checksum key* is the SHA-256 of some content

The *value* is not directly stored in the trace. Instead, an identifier (the **value id**)
is stored in the trace, and the potentially large value is stored in the value store (more on that next section).

### Value Store

The value store is a key value table stored on disk.

The **value type** is a single letter that categorizes what the value is:

| Value Type | What                      | Docs                                     |
| ---------- | ------------------------- | ---------------------------------------- |
| `o`        | object                    | [Objects](#objects)                      |
| `b`        | bundle                    | [Bundles](#bundles)                      |
| `a`        | asset                     | [Assets](#assets)                        |
| `i`        | index files               | [Objects](#objects) or [Assets](#assets) |
| `j`        | values.json file          | [JSON Files](#json-files)                |
| `l`        | values.lua file           | [Lua Scripts](#scripts)                  |
| `v`        | (cache) parsed values AST | [JSON Files](#json-files)                |
| `c`        | built-in constants        | [Objects](#objects)                      |
| `s`        | source file               | FILLMEIN                                 |

All value types are *lowercase* for support on case-insensitive file systems.

Any value types with `(cache)` are stored in the local cache rather than the valuestore.

- A **value file** is a file whose content matches the value type
- A **value sha256** is a SHA-256 hex-encoded string of the value file. That is, if you ran `certutil` (Windows), `sha256sum` (Linux) or `shasum -a 256` (macOS) on the value file, the *value sha256* is what you would see.

The build system requests data in the form of a build key, and a [build task](#task-model) is responsible for resolving the build key into a value that will be persisted in the value store.
The value will be of a type that depends on the build key:

| Build Key                             | Value Type | Id Material                           | Value File                                                               |
| ------------------------------------- | ---------- | ------------------------------------- | ------------------------------------------------------------------------ |
| [asset](#assets)                      | `a`        | [ACI](#aci---asset-canonical-id)      | contents of asset                                                        |
|                                       | `i`        | digest of the `a` id (prefix swap)    | [index](#i---index-file)                                                 |
| [bundle](#bundles)                    | `b`        | [BCI](#bci---bundle-canonical-id)     | contents of zip archive file                                             |
| [object](#objects)                    | `o`        | [FRM](#frm---form)                    | output of form function                                                  |
|                                       |            | `::<SLOT>`                            |                                                                          |
|                                       | `i`        | digest of the `o` id (prefix swap)    | [index file](#i---index-file)                                            |
| [V256](#v256---sha256-of-values-file) | `j`        | [V256](#v256---sha256-of-values-file) | dos2unix json `{schema_version:,forms:,bundles:}`                        |
| [V256](#v256---sha256-of-values-file) | `l`        | [V256](#v256---sha256-of-values-file) | dos2unix lua script                                                      |
| [VCI](#vci---values-canonical-id)     | `v`        | [VCK](#vck---values-checksum)         | [parsed `{schema_version:,forms:,bundles:}`](#v---parsed-valuesjson-ast) |

The exact construction of each value id is given in [Value Id Formulas](#value-id-formulas).

To allow byte range optimized reading of zip files, special rules are in place when fetching assets and objects:

- When fetching an asset, the fetched build system value is a composite `Optional (zip)` and `Always (asset)`.
- When fetching an object, the fetched build system value is a composite `Optional (zip)` and `Always (object)`.
- If the object value is known¹ to be a zip file the build system implementation will produce both the `i` value type *and* the `o` value type.
- If the asset value is known¹ to be a zip file the build system implementation will produce both the `i` value type *and* the `a` value type.

A build system implementation may choose to lazily extract entries using the `Optional (zip)` value (if present) which serves as an index over the `Always (*)` value.
The `Optional (zip)` value avoids the overhead of preemptively downloading and unzip all of the possibly huge `Always (*)` zip file.
However, for the `Optional (zip)` value to provide a benefit, the value URL must support byte range lookups. For example, if assets are on a [web server](#web-assets) that does not support the [HTTP Range header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Range), the entire `Always (asset)` zip file will be downloaded.

¹: Currently only value store assets of [distributed value stores](#distributed-value-stores) are known to be zip files. Other asset files are [1-to-1 with their SHA256 identifiers](#asset-identity), and so can't be deterministically known to be zip files.

The *value id* is used to lookup or persist into the value store.

#### Value Id Formulas

Every value id is an explicit function of its inputs. The helper functions,
where `||` is byte concatenation:

```text
SHA256_HEX(s)   = lowercase hex encoding of the SHA-256 digest of the bytes s
BLAKE2B_RAW(s)  = raw (unencoded) BLAKE2b-256 digest of the bytes s
BASE32L(hex)    = RFC 4648 base32 of the hex-decoded bytes, using the lowercase
                  alphabet "abcdefghijklmnopqrstuvwxyz234567", without "="
                  padding. A 32-byte digest encodes to 52 characters.
BASE32L_RAW(b)  = the same base32 encoding applied directly to raw bytes b
CANON_JSON(x)   = the canonical compact JSON serialization of x: object fields
                  in the field order given in the definitions below, no
                  insignificant whitespace
STRIP_CR(s)     = s with every carriage return (ASCII 13) byte removed
```

Common inputs:

```text
VCI       = SHA256_HEX( canonical JSON of the parsed values file CST )
            (see "VCI - Values Canonical ID"; the values file is stripped of
             carriage returns before parsing)
MODVER    = MODULE "@" VERSION      -- no build metadata; see "BLD"
SLOT      = the value shell slot name, e.g. "Release.Windows_x86_64"
```

The formulas, per value type:

```text
-- o : object (the output of one form, for one slot)
FRM  = SHA256_HEX( VCI || "|form|" || MODVER )
o_id = "o" || BASE32L( SHA256_HEX( FRM || "::" || SLOT ) )

-- a : asset (one file of a bundle)
ACI_JSON = CANON_JSON( { checksum = { blake2b256?, sha1?, sha256? },
                         indexes?  = [ per-index records ],
                         path      = FILE_PATH,
                         size      = FILE_SIZE } )
ACI  = SHA256_HEX( ACI_JSON )
a_id = "a" || BASE32L( SHA256_HEX( ACI ) )

-- b : bundle (the zip of all files of a bundle)
BCI_JSON = CANON_JSON( { assets = [ ACI_JSON of each file, sorted by path ],
                         id     = MODVER } )
BCI  = SHA256_HEX( BCI_JSON )
b_id = "b" || BASE32L( SHA256_HEX( BCI ) )

-- i : index file over a zip-shaped object or asset payload
i_id = "i" || suffix          where ("o" || suffix) = o_id
                              or    ("a" || suffix) = a_id

-- j : values.json file        l : values.lua file
V256 = SHA256_HEX( STRIP_CR( values file bytes ) )
j_id = "j" || BASE32L( V256 )
l_id = "l" || BASE32L( V256 )

-- v : parsed values AST (local cache only)
VCK  = BLAKE2B_RAW( VCI || EXEC_ABI || TARGET_ABI || AST_SCHEMA_ID || CT )
v_id = "v" || BASE32L_RAW( VCK )
       where EXEC_ABI/TARGET_ABI are the ABI names of the current process,
             AST_SCHEMA_ID is a generated fingerprint of the AST type
             definitions (changes whenever the AST types change), and
             CT is the "CT - Compatibility Tag"

-- c : built-in constant        x : execution streams
c_id = "c" || BASE32L( SHA256_HEX( constant bytes ) )
x_id = "x" || BASE32L( SHA256_HEX( encoded execution streams ) )

-- k : key with a lazy value    s : source file (debugging)
k_id = "k" || KEY_ID
s_id = "s" || BASE32L( SHA256_HEX( source file bytes ) )
```

Two structural properties follow directly from the formulas:

1. **Asset and bundle ids pin exact bytes.** `ACI_JSON` embeds the content
   checksum and byte size of the file, so two assets with different bytes can
   never share an `a` id, and `BCI_JSON` inherits that property for `b` ids.
2. **Object ids do not hash the produced output.** `o_id` is derived only from
   the *recipe address*: the values file (via `VCI`), the form's module version and
   the slot. The bytes that the form's function writes into the output
   directory appear nowhere in the formula. The consequences are described in
   the next section.

#### Object Ids Hide Build Non-Determinism

An object id is a *recipe address* (the values file (via `VCI`), the form's
module version and the slot). Whichever build of the recipe
completes first has its output bytes persisted into the value store
under that id; every later build of the same recipe reuses (or republishes)
bytes under the same id, even if a fresh build would have produced different
bytes.

So object ids are designed to hide non-determinism present in
the underlying build. For a distribution script that means the use of
`$ get-object ...` will output a `\dk.object(...)` that contains a stable,
build-agnostic `value-id`.

An early version of the `CommonsLang_OCaml.DkML@4.14.3 -s Release.Windows_x86_64`
OCaml compiler object, for example, had independent builds with the same
value id. Even so, unique working directories were a primary source of non-determinism;
751 of the 2032 files inside any two newly built objects differed byte-for-byte:

- 671 OCaml typed-tree files (`.cmt`, `.cmti`) differed by a handful of bytes
  each: the OCaml compiler embeds the absolute source path, which contains the
  build system's ephemeral per-invocation working directory (for example
  `t\p\476\7vpw\...\src-ocaml\` in one build and `t\p\484\2vog\...\src-ocaml\`
  in the other).
- Linked binaries (`.exe`, `.dll`, `.lib`, `.obj`), which were generated by the
  MSVC compiler, differed because paths are recorded in debug information.
- Bytecode archives (`.cma`, `.cmo`) also differed.

> [!TIP]
> Nothing prevents the object recipe from setting the compiler options and
> environment to be bit-for-bit reproducible.
> For example, the OCaml compiler accepts
> a BUILD_PATH_PREFIX_MAP environment variable to adjust build paths, and
> the MSVC compiler accepts `/PDBALTPATH`, `/Brepro`, and other
> deterministic flags. And that **bit-for-bit reproducibility can be enforced**
> with distribution scripts that run a checksum object with
> a subshell that gets the object (`get-object`):
>
> ```sh
> $ run-object
> >   CommonsBase_Std.Coreutils@0.8.0 -s Release.execution_abi
> >   -m ./coreutils.exe -f coreutils.exe -e '*'
> >   --
> > sha256sum -b
> > $(get-object OBJECT -s SLOT -m MEMBER -f YOUR_FILENAME)
> \test(pass)
> \dk.target(abi: "Windows_x86_64")[SHA256 (YOUR_FILENAME) = 66d5f22136245be5e5c0aa3aba4cdc795d0c7315c23a5ccef3322fd008ac007d
> ]
> ```

#### v - parsed values.json AST

A `values.json` is parsed into an AST, and the AST is persisted directly from OCaml memory blocks and signed with the local build key.

The build system will verify the signature of the AST before loading the AST into memory.
If the signature does not match the local build key, or if the AST is incompatible with the memory layout of the current process (see [compatibility tag](#ct---compatibility-tag)), the `j` values.json file is fetched and re-parsed into a new AST.

#### i - index file

The value file is:

| Offset (bytes) | Size (bytes) | What                                        |
| -------------- | ------------ | ------------------------------------------- |
| 0              | n            | [Central Directory]                         |
| n              | 4            | Magic number. `44 4B 49 56` (DKIV)          |
| n+4            | 4            | Reserved. `00 00 00 00` for zipfile indexes |
| n+8            | 8            | offset of first local header                |

The offset of the first local header is typically zero (0) except in special cases like:

- self-extracting zips have the executable stub before the first local header

All sizes are little-endian.

The [Central Directory] includes:

1. All the [Central directory file headers (CDFH)](https://en.wikipedia.org/wiki/ZIP_(file_format)#Central_directory_file_header_(CDFH)) including their ZIP64 extra fields.
2. [Zip64 End of central directory record (EOCD64)](https://en.wikipedia.org/wiki/ZIP_(file_format)#ZIP64)
3. [20-byte End of Central Directory Locator](https://en.wikipedia.org/wiki/ZIP_(file_format)#ZIP64)
4. The classic [end of central directory record (EOCD)](https://en.wikipedia.org/wiki/ZIP_(file_format)#End_of_central_directory_record_(EOCD))

The following [Central Directory] fields are modified:

| Record                                         | Offset | Size | New Value                                                                |
| ---------------------------------------------- | ------ | ---- | ------------------------------------------------------------------------ |
| Zip64 End of central directory record (EOCD64) | 16     | 4    | 1 (Number of this disk.)                                                 |
| Zip64 End of central directory record (EOCD64) | 20     | 4    | 1 (Disk where central directory starts.)                                 |
| Zip64 End of central directory record (EOCD64) | 48     | 8    | CD (Offset of start of central directory, relative to start of archive.) |
| Zip64 End of Central Directory Locator         | 4      | 4    | 1 (Disk where EOCD64 starts.)                                            |
| Zip64 End of Central Directory Locator         | 8      | 8    | CD + EOCD64 (Offset to start of EOCD64, relative to start of archive.)   |
| Zip64 End of Central Directory Locator         | 16     | 4    | 2 (Total number of disks.)                                               |
| Classic EOCD                                   | 4      | 2    | 1 (Number of this disk)                                                  |
| Classic EOCD                                   | 6      | 2    | 1 (Disk where central directory starts)                                  |
| Classic EOCD                                   | 8      | 8    | CD (Offset of start of central directory, relative to start of archive)  |

The modifications mean the `i` value files are valid ZIP files. In other words:

- the `i` file becomes the second disk containing only the directory entries
- the original asset file, if downloaded in its entirety, is the first disk

[Central Directory]: https://en.wikipedia.org/wiki/ZIP_(file_format)#Structure

#### BLD - Build Metadata

The dot (`.`) separated build metadata from the semver version.

For example, `OurZip_Demo.S7z2.Windows7zExe@25.1.0+bn-20250101000000+diff` has build metadata `bn-20250101000000.diff`.

Build metadata is deliberately **not** part of any value id. The `a`, `b` and `o` value ids are content-addressed (from the asset/bundle/form canonical id and, for objects, the slot), so identical content gets the same value id regardless of the build number (the `bn-*` build metadata). This keeps distributions reproducible: changing the build number (for example via dk0's `-n` option or a git tag) does not change the object, bundle or asset ids.

Build metadata still participates in keys and versions (see [ID with Build Metadata](#object-id-with-build-metadata)); it is only excluded from value ids.

#### V256 - SHA256 of Values File

The SHA-256 (raw, not hex-encoded) of the `values.json` file that contains the bundle (or form or asset).

#### P256 - SHA256 of Asset

The hex-encoded SHA-256 of the asset. It is the `checksum.sha256` in the following asset:

```json
{
  "origin": "github-release",
  "path": "SHA256.sig",
  "size": 151,
  "checksum": {
    "sha256": "0d281c9fe4a336b87a07e543be700e906e728becd7318fa17377d37c33be0f75"
  }
}
```

#### Z256 - SHA256 of Zip Archive File

The hex-encoded SHA-256 of the zip archive generated from either:

- the output directory of a form
- the bundle directory for one or more bundle files

Z256 is a checksum of the stored *value file* (the payload).
In particular, the Z256 of an object
payload is not stable across rebuilds of the same object id (see
[Object Ids Hide Build Non-Determinism](#object-ids-hide-build-non-determinism)).

#### CT - Compatibility Tag

A string with the format `oc<OCAMLVERSION>_ws<OCAMLWORDSIZE>`.

For example, `oc414_wd64` is OCaml 4.14 with a 64-bit word size.

#### VCI - Values Canonical ID

The hex-encoded SHA256 of the `values.json` *canonicalized* JSON, stripped of all carriage returns (ASCII CR 13).

#### VCK - Values Checksum

The raw BLAKE2b-256 digest that keys the local cache of the parsed values AST:

```text
VCK = BLAKE2B_RAW( VCI || EXEC_ABI || TARGET_ABI || AST_SCHEMA_ID || CT )
```

where `EXEC_ABI` and `TARGET_ABI` are the ABI names of the current process,
`AST_SCHEMA_ID` is a generated fingerprint of the AST type definitions (it
changes whenever the AST types change, protecting the marshalled AST from
schema drift), and `CT` is the [compatibility tag](#ct---compatibility-tag).

The stripping of carriage returns occurs before the CST and AST parsing, so that any serialized AST uses the byte positions of the Unix-encoded JSON.

#### FRM - Form

The hex-encoded SHA-256 of the concatenation of:

- [VCI](#vci---values-canonical-id) for the `values.json` defining the form
- `|form|`
- `MODULE@VERSION` (without [build metadata](#bld---build-metadata))

That is, using the helpers of [Value Id Formulas](#value-id-formulas):

```text
FRM = SHA256_HEX( VCI || "|form|" || MODULE "@" VERSION )
```

#### ACI - Asset Canonical Id

The hex-encoded SHA-256 of the canonical compact JSON of one bundle file's
identity fields, in the field order:

```text
ACI_JSON = { "checksum": { "blake2b256"?, "sha1"?, "sha256"? },
             "indexes"?: [ per-index records ],
             "path": FILE_PATH,
             "size": FILE_SIZE }
ACI      = SHA256_HEX( CANON_JSON( ACI_JSON ) )
```

The `checksum` object contains whichever of the three checksums the bundle
declares. The [P256](#p256---sha256-of-asset) content checksum and the byte
size are both inside `ACI_JSON`, so an asset id pins the exact bytes of the
asset.

#### BCI - Bundle Canonical Id

The hex-encoded SHA-256 of the canonical compact JSON of the bundle
definition:

```text
BCI_JSON = { "assets": [ ACI_JSON of each file, sorted by path ],
             "id": MODULE "@" VERSION }
BCI      = SHA256_HEX( CANON_JSON( BCI_JSON ) )
```

The origins/mirrors listing is deliberately excluded so that re-mirroring a
bundle does not change its identity.

## Evaluation

| Phase        | What                                                |
| ------------ | --------------------------------------------------- |
| PRECONFIG    | (1) Resolve environment vars, directories and keys  |
| TRACELOCK    | Exclusive writer lock on the trace store            |
| TRACEREAD    | Read trace store                                    |
|              | Do quick value store integrity checks               |
| CONFIG       | (2) Create pid directory. And value store if needed |
| COMMANDPARSE | Parse the get-object, etc. command                  |
| STATERESTORE | (3) Initialize state from traces                    |
| VALUESCAN    | Scan values.json/.lua in include dirs               |
|              | Add parse-CST `j` tasks                             |
|              | Add built-in tasks                                  |
| VALUELOAD    | (4) Full value store integrity check.               |
|              | Run `j` tasks to get CST.                           |
|              | Parse CST into AST; validate; place in cache        |
|              | From ASTs add `d`,`f`,`b`,`a` tasks                 |
|              | Run `d` distribution tasks                          |
| USER         | Find command in task graph. Run user task.          |
| GRAPH        | Dump dependency/ancestor graphs if requested        |
| STATESAVE    | Update trace store                                  |

The number in parentheses is the classic phase number; those numbers are being phased out.