Design: unified, modular installation-knowledge manifest across platforms

[darkn3rd]

Motivation

Installation knowledge for this project is currently scattered across each
language's own README (Homebrew/Chocolatey/apt/pyenv/rbenv/SDKMAN
instructions, each written ad hoc). There's no single, organized,
machine-readable source of truth for "what needs to be installed, on which
platform, to run lesson X" - and no automated way to resolve cross-package-
manager dependencies (e.g. Groovy needs a JVM; corretto@17 via Homebrew,
Chocolatey, or apt-based Corretto repo all independently satisfy that).

Proposed design

1. Package manifest, organized by platform then by lesson directory

A YAML manifest, top-level keyed by platform (macos, windows, cygwin,
msys, ubuntu22, ...), nested under a lessons: key to mirror the actual
lessons/<category>/<language> directory structure. Each leaf lists
packages as pkg_manager: pkg_name (or a list of names), with optional
needs/meets/require keys for dependency resolution across package
managers - e.g. Groovy declares needs: java, and a Corretto package
declares meets: java to satisfy that regardless of which manager provided
it. A script: package type covers anything a plain package-manager line
can't handle (SDKMAN bootstrap, pyenv/rbenv setup, Chocolatey/Homebrew
bootstrapping, custom apt-repo setup) - defined once in a shared scripts:
block and referenced by name.

Small illustrative snippet (full manifest at the end of this issue):

macos:
  lessons:
    gen_scripts:
      groovy:
        packages:
          - brew: groovy
            needs: java
      perl:
        packages:
          - brew: perl
            meets: perl
ubuntu22:
  lessons:
    gen_scripts:
      python3:
        packages:
          - pyenv: 3.14.6
            cmd: pyenv global 3.10.4
scripts:
  macos_homebrew:
    type: bash
    cmd: |
      /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

2. Install resolution algorithm

• lesson_install <dotted.path> walks the manifest tree at that path,
  collecting every package declared at and below it.
• Parents are installed before children; for any package declaring
  require: [pkg_type: pkg_name, ...], those prerequisites are resolved
  and installed first (topological order).
• Uses flattened dot-path addressing to target either a single language
  (lesson_install lesson.gen_scripts.perl) or a whole category
  (lesson_install shell_scripts installs bash/zsh/dash/tcsh/ksh together).
• A cmd: step defaults to PowerShell on Windows, bash on POSIX, unless a
  step explicitly overrides the type.

lesson_install lesson.gen_scripts.perl  # install perl
lesson_install shell_scripts            # installs bash, zsh, dash, tcsh, ksh on mac
lesson_install cibox

3. Future extensibility: "yaps" (yet another package)

For cases too complex for a single package-manager line, an embedded
mini-package format modeled on Debian's .deb structure - preinst/
postinst/prerm hook scripts plus a data: tree of files to place.

yaps:
  yap_name:
    preinst: |
      command
    data:
      etc:
        apt:
          sources.list.d:
            microsoft-prod-list: deb [arch=amd64,arm64,armhf] https://packages.microsoft.com/ubuntu/22.04/prod jammy main

4. system_info() - normalized platform/version identification

Two-step detection: uname -s (or $PSVersionTable on native Windows) as
the first-pass discriminator, then an OS-specific follow-up query for the
actual distro/version, normalized to a consistent platform.version string:

uname -s / detection	Follow-up query	Normalized form
CYGWIN_NT-10.0-26200	-	cygwin_nt.10_0_26200
MINGW64_NT-10.0-26200	-	mingw64_nt.10_0_26200
Darwin	sw_vers	darwin.26_5
Linux	/etc/os-release	linux_ubuntu.26_04
Windows native	$PSVersionTable / [System.Environment]::OSVersion / Get-CimInstance Win32_OperatingSystem / Get-ComputerInfo	windows_nt.10_0_26200

5. Environment/platform support metadata

A separate block maps each manifest platform-name to the normalized
system_info() strings it's known to support:

environments:
  - platform-name: windows
    supports: [windows_nt.10_0_26200]
  - platform-name: darwin
    supports: [darwin.26_5]
  - platform-name: ubuntu22
    supports: [linux_ubuntu.26_04]
  - platform-name: cygwin
    supports: [cygwin_nt.10_0_26200]
  - platform-name: msys
    supports: [mingw64_nt.10_0_26200]

If the detected environment's normalized string isn't listed under any
platform's supports:, the tool aborts safely with an informative message
rather than guessing at a possibly-incompatible install path.

Open questions

• What actually implements lesson_install (Ruby, since testbox already
  depends on it? A standalone per-platform script?).
• How does this relate to the existing scattered manifests already in the
  repo (scriptbox's Brewfile/choco.config, cygwin_pkgs.txt,
  msys2_pkgs.txt) - wholesale replacement, or does this generate/subsume
  those?

Out of scope for now

Design/discussion only - no implementation until the shape of this is
settled.

---

Full draft manifest

Click to expand full YAML draft (macos, windows, cygwin, msys, ubuntu22, scripts)

macos:
  global:
    packages:
      - script: macos_homebrew
    lessions:
      gen_scripts:
        awk:
          packages:
            - brew: gawk
        groovy:
          packages:
            - brew: groovy
              needs: java
        perl:
          packages:
            - brew: perl
              meets: perl
        php:
          packages:
            - brew: php
        python3:
          packages:
            - brew: python3
        ruby:
          packages:
            - brew: ruby
        tcl:
          packages:
            - brew: tcl-tk
      shell_scripts:
        packages: 
            - brew: bc
            - brew: getopt
        bash:
            packages: 
              - brew: bash
        csh:
            packages: 
              - brew: tcsh
        ksh:
            packages: 
              - brew: ksh
        posix:
            packages: 
              - brew: dash-shell
              - needs: perl
        zsh:
            packages:
            - brew: zsh
      compiled_lang:
        cs:
          packages:
            - cask: dotnet-sdk
        go:
          packages:
            - brew: go
        java:
          packages:
            - cask: corretto@17
              require:
                - tap: homebrew/cask-versions
              meets: java
            - tap: homebrew/cask-versions
        rust:
          packages:
            - brew: rust
      win_scripts:
        powershell:
          packages:
            - brew: powershell
    cibox:
      packages: 
        - brew: act
        - brew: colima
        - brew: docker
    testbox:
      packages:
        - brew: powershell
        - brew: ruby
windows:
  global:
    packages:
      - choco: gsudo
        require:
         - script: windows_chocolatey
      -  script: windows_chocolatey
      - choco: make
        require:
         - script: windows_chocolatey
    lessons:
      gen_scripts:
        awk:
          packages:
            - choco: gawk
        groovy:
          packages:
            - choco: groovy
              needs: java
        perl:
          packages:
            - choco: strawberryperl
              meets: perl
        php:
          packages:
            - choco: php
        python2:
          packages:
            - choco: python2
        python3:
          packages:
            - choco: python3
        ruby:
          packages:
            - choco: ruby
        tcl:
          packages:
            - choco: magicsplat-tcl-tk
      shell_scripts:
        packages: 
          - choco: cygwin
          - choco: cyg-get
          - choco: msys2
          - choco_cyg: wget
            require:
              - choco: cyg-get
      compiled_lang:
        cs:
          packages:
            - choco: dotnet-sdk
        go:
          packages:
            - choco: go
        java:
          packages:
            - choco: corretto17jdk
              meets: java
        rust:
          packages:
            - choco: rust
      win_scripts:
        powershell:
          packages:
            - choco: powershell-core
      cibox:
        packages:
          - feature: Microsoft-Windows-Subsystem-Linux
            reboot: yes
          - feature: VirtualMachinePlatform
            reboot: yes
          - choco: act-cli
          - choco: docker-desktop
            require: 
              - feature: Microsoft-Windows-Subsystem-Linux
              - feature: VirtualMachinePlatform
      testbox:
        packages:
          - choco: powershell-core
          - choco: ruby
          - choco: psake
            require:
              - choco: powershell-core
  cygwin:
    global:
      gen_scripts:
        perl:
          packages:
            - cyg: perl
              meets: perl
        php:
          packages:
            - cyg: php
        python3:
          packages:
            - cyg: python312
        ruby:
          packages:
            - cyg: ruby
        tcl:
          packages:
            - cyg: tcl
      shell_scripts:
        packages:
          - cyg: bc
        csh:
          packages:
            - cyg: tcsh
        posix:
          packages:
            - cyg: perl
      compiled_lang:
        packages:
          - cyg: 
              - make
              - git
        cpp:
          packages:
            - cyg:
                - gcc-core
                - gcc-g++
                - automake
                - autoconf
                - libtool
                - binutils
                - pkg-config
                - patch
        rust:
          packages:
            - cyg: rust
    win_scripts:
    testbox:
      packages:
        - cyg: ruby
        - cyg: rubygems
msys:
  global:
    lessons:
      gen_scripts:
        awk:
          packages:
            - pacman: gawk
        perl:
          packages:
            - pacman: perl
              meets: perl
            - script: cpan_setup
              require:
                - pacman: perl
            - cpan: App::cpanminus
              require:
                - script: cpan_setup
            - cpanm: Switch
              require:
                - cpan: App::cpanminus
        python3:
          packages:
            - pacman: mingw-w64-ucrt-x86_64-python
        ruby:
          packages:
            - pacman: mingw-w64-ucrt-x86_64-ruby
        tcl:
          packages:
            - pacman: mingw-w64-ucrt-x86_64-tcl
      shell_scripts:
        packages:
          - pacman: bc
        csh:
          packages:
            - pacman: tcsh
        posix:
          packages:
            - pacman: dash
      compiled_lang:
        packages:
          - pacman: base-devel
          - pacman: mingw-w64-x86_64-toolchain
        go:
          packages:
            - packman: mingw-w64-ucrt-x86_64-go
        rust:
          packages:
            - packman: mingw-w64-ucrt-x86_64-rustup
      win_scripts: {}
    testbox:
      packages:
        - pacman: ruby
ubuntu22:
  global:
    packages:
      - script: ubuntu22_sdkman
      - apt: ['curl','git','make','wget']
    lessons:
      gen_scripts:
        packages:
          - apt:
              - autoconf
              - bison
              - build-essential
              - libbz2-dev
              - libdb-dev
              - libffi-dev
              - libgdbm6
              - libgdbm-dev
              - liblzma-dev
              - libncurses5-dev
              - libncursesw5-dev
              - libreadline6-dev
              - libreadline-dev
              - libsqlite3-dev
              - libssl-dev
              - libxml2-dev
              - libxmlsec1-dev
              - libyaml-dev
              - llvm
              - tk-dev
              - xz-utils
              - zlib1g-dev
          - apt: rbenv
            script: ubuntu22_post_rbenv
          - script: ubuntu22_pyenv
        awk:
          packages:
            - apt: gawk
        groovy:
          packages:
            - sdkman: groovy
              needs: java
        perl:
          packages:
            - apt: perl
              meets: perl
        php:
          packages:
            - apt: php
        python2:
          packages:
            - pyenv: 2.7.18
              cmd: pyenv global 3.10.4 2.7.18
              requires:
                - pyenv: 3.14.6
        python3:
          packages:
            - pyenv: 3.14.6
              cmd: pyenv global 3.10.4
        ruby:
          packages:
            - rbenv: 4.0.6
              cmd: rbenv global 4.0.6
        tcl:
          packages:
            - apt: tcl
      shell_scripts:
        packages: 
            - brew: bc
            - brew: getopt
        bash:
            packages: 
              - brew: bash
        csh:
            packages: 
              - brew: tcsh
        ksh:
            packages: 
              - brew: ksh
        posix:
            packages: 
              - brew: dash-shell
              - needs: perl
        zsh:
            packages:
            - brew: zsh
      compiled_lang:
        packages:
          apt:
            - curl
            - build-essential
            - make
            - gcc
        java:
          packages:
            script: ubuntu22_coretto_17
        go: 
          packages:
            script: ubuntu22_go
        rust:
          packages:
            script: ubuntu22_rust
      win_scripts:
        powershell:
          packages:
            - script: ubuntu22_powershell
    cibox:
      packages: 
        - brew: act
        - brew: colima
        - brew: docker
    testbox:
      packages:
        - script: ubuntu22_powershell
        - rbenv: 4.0.6
scripts:
  ubuntu22_powershell:
    type: bash
    cmd: |
      wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
      sudo dpkg -i packages-microsoft-prod.deb
      rm packages-microsoft-prod.deb
      
      # Add Microsoft packages list
      sudo apt-get update

      # Install PowerShell 7.x (pwsh)
      sudo apt-get install -y powershell
  ubuntu22_pyenv:
    type: bash
    cmd: |
      # Install Pyenv
      curl https://pyenv.run | bash 

      # Profile Setup
      echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
      echo 'command -v pyenv >/dev/null || export PATH="$PATH:$PYENV_ROOT/bin"' >> ~/.bashrc
      echo 'eval "$(pyenv init -)"' >> ~/.bashrc
      source ~/.bashrc

      # Update List
      pyenv update
  ubuntu22_post_rbenv:
     type: bash
     cmd: |
      # Profile Setup
      echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
      echo 'eval "$(rbenv init -)"' >> ~/.bashrc
      source ~/.bashrc

      # Update List
      pushd $HOME/.rbenv/plugins/ruby-build  && git pull && popd
  macos_homebrew:
    type: bash
    cmd: |
      script_url="https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh"
      /bin/bash -c "$(curl -fsSL "$script_url")"
  windows_chocolatey:
    type: powershell
    cmd: |
      Set-ExecutionPolicy Bypass -Scope Process -Force
      [System.Net.ServicePointManager]::SecurityProtocol = `
        [System.Net.ServicePointManager]::SecurityProtocol -bor 3072
      $WebClient = New-Object System.Net.WebClient
      $ScriptUrl = 'https://community.chocolatey.org/install.ps1'
      Invoke-Expression ($WebClient.DownloadString($ScriptUrl))
  ubuntu22_rust:
    type: bash
    cmd: |
      curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
      echo 'source "$HOME/.cargo/env' >> ~/.bashrc
      source ~/.bashrc
  ubuntu22_go:
    type: bash
    cmd: |
      sudo add-apt-repository ppa:longsleep/golang-backports
      sudo apt update && sudo apt install golang-go
      
      echo 'export GOPATH=$(go env GOPATH)' >> ~/.bashrc
      echo 'export PATH=$GOPATH/bin:$PATH' >> ~/.bashrc
      source ~/.bashrc
  cpan_local_setup:
      type: bash
      cmd: |
        PERL_MM_OPT="INSTALL_BASE=$HOME/perl5" cpan local::lib
        echo 'eval "$(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib=$HOME/perl5)"' >> ~/.bashrc
        source ~/.bashrc     
  ubuntu22_coretto_17:
      type: bash
      cmd:  |
        KEYRING="/usr/share/keyrings/corretto-keyring.gpg"
        wget -O - https://apt.corretto.aws/corretto.key \
          | sudo gpg --dearmor -o $KEYRING
        echo "deb [signed-by=$KEYRING] https://apt.corretto.aws stable main" \
          | sudo tee /etc/apt/sources.list.d/corretto.list
        sudo apt update && sudo apt install -y java-17-amazon-corretto-jdk

        echo "export JAVA_HOME=\$(readlink -f /usr/bin/java | sed 's|/bin/java||')" >> ~/.bashrc
        source ~/.bashrc    

Click to expand full "yaps" concept draft

yaps:
  yap_name:
    control:
    preinst: |
      command
      command
    postinst:
      command
      command
    prerm: |
      command
      command
    data:
      etc:
        apt:
          sources.list.d:
              microsoft-prod-list: deb [arch=amd64,arm64,armhf] https://packages.microsoft.com/ubuntu/22.04/prod jammy main

[darkn3rd]

This format will be used by a few tools that can do the following:

• create package manifests: Brewfile, choco.config, other
• generate install docs: INSTALL.md
• perform the installation

The format will indeed need to support pre and post operations before a package installation or copy of components. The proposed key words can borrow from terms in DEB package: preinst, postinst, prerm, postrm. As this is cross platform, a type can be provided if the scripting language is not posix, such as powershell. The powershell will try pwsh and powershell while posix can try: zsh, bash, dash and sh.

It may be useful utility operations for common chores to clean up embedded scripts:

• profile setup: .bashrc, .zshrc, .bash_profile, .profile
• env vars:
  • posix
  • windows: machine and user
• path
  • can be part of profile setup and install
  • pathmunger utility would be nice
• perms (permissions):
  • windows: iacls
  • posix: chmod

A data block for any configuration files if needed. This could just be part of any script or cmd block. A data block would need the artifacts (binaries are base64 encoded), perms, and path.