Migrate build system from Grunt to JS API with pre-build cache

[taylortom]

Context

The adapt_framework build is orchestrated by Grunt, invoked from the AAT via exec('npx grunt server-build:...') as a subprocess. This causes:

• False build failures: stderr output (e.g. browserslist warnings) is treated as fatal errors
• Slow builds: ~60s per course, with ~4s wasted on subprocess startup (Node + npx + grunt loading), sequential task execution when LESS/Handlebars/JS could run in parallel, and a full ~50MB framework directory copy per build
• No programmatic API: The AAT must shell out through adapt-cli to grunt, making caching, error handling, and integration unnecessarily complex

Key finding: most build output is not course-specific

Build output	Determined by	Course-specific?
adapt.min.js (Rollup bundle)	Plugin set only	No
templates.js (Handlebars)	Plugin set only	No
adapt.css (LESS)	Theme + menu	No
build.min.js (metadata)	Plugin set only	No
Plugin assets/fonts/libraries/HTML	Plugin set + theme	No
Course JSON files	Course content	Yes
Course assets (images/media)	Course content	Yes
HTML/XML replace output	Course + config data	Yes

---

Phase 1: Pre-built Compilation Cache for Previews

Goal: Eliminate ~55s of compilation from every preview build by caching grunt build output and reusing it across courses. No changes to adapt_framework needed.

Design constraints

• Cache logic must be portable to adapt-cli later
• Use adapt-cli's existing Project class where possible
• Only add new code for logic that doesn't exist or currently shells out to grunt
• Schema defaults are already handled by the jsonschema module at DB save time — no build-time reapplication needed

Cache strategy — two-tier, keyed by what determines the output

Artifact	Cache key	Invalidation trigger
JS bundle, templates, build.min.js, plugin assets/fonts/libraries	pluginHash (hash of all plugin names+versions)	Plugin or framework change
CSS (adapt.css + source map)	pluginHash + theme + menu	Plugin, framework, theme, or menu change

pluginHash is a content hash derived from sorted plugin name+version pairs (via adapt-cli's Project.getInstalledDependencies()), not the framework semver string. This catches any plugin change automatically.

Cache directory layout

{buildDir}/prebuilt-cache/
  {pluginHash}/                    ← shared artifacts
    adapt/js/adapt.min.js
    adapt/js/adapt.min.js.map
    adapt/js/build.min.js
    templates.js
    assets/
    libraries/
    index.html                     ← with @@placeholders still intact
    ...
  {pluginHash}_{theme}_{menu}/     ← CSS artifacts
    adapt/css/adapt.css
    adapt/css/adapt.css.map

Notes:

• index.html is cached before @@ replacement so it retains placeholders. This avoids caching a file that needs per-course modification.
• Cache lives under the configured temp/build dir for easy cleanup and disk space management.
• Parallel safety: On cache miss, populateCache writes to a temp dir first, then atomically renames into place. Concurrent misses that race will each produce identical content; last rename wins. On cache hit, each build copies into its own isolated timestamped build dir — no contention.

Build paths

Cache MISS (first build or after invalidation):

1. Run full grunt build via AdaptCli.buildCourse() as today
2. After grunt completes, extract shared artifacts into cache dirs (index.html saved with @@placeholders intact)
3. Continue with normal build flow

Cache HIT (subsequent preview builds):

1. Copy cached shared artifacts + CSS to build dir
2. Write course JSON (already in memory from loadCourseData())
3. Copy course assets (streams from storage)
4. Generate language manifest from known file list
5. Apply @@ replacements in index.html using in-memory config data
6. Done — skip AdaptCli.buildCourse() entirely

In-memory data flow (cache hit path)

On cache hit, grunt is skipped entirely. Post-processing is minimal:

• Schema defaults: NOT needed — jsonschema module applies defaults at DB save time
• Language manifest: Trivial ~5-line utility listing JSON filenames written to the language dir
• Replace patterns: Small utility applies 4 @@ substitutions in index.html (@@config._defaultLanguage, @@config._defaultDirection, @@build.type, @@build.timestamp) using config data already in memory

The only required filesystem writes on cache hit are: cached artifact copy, final course JSON, language manifest, course assets, modified index.html.

---

Implementation Step 1: New utility — lib/utils/computePluginHash.js

Computes a deterministic hash from the installed plugin set. Uses adapt-cli's Project class.

import { createHash } from 'node:crypto'
import Project from 'adapt-cli/lib/integration/Project.js'

export function computePluginHash (frameworkDir) {
  const project = new Project({ cwd: frameworkDir })
  const deps = project.getInstalledDependencies() // { name: version, ... }
  const sorted = Object.entries(deps).sort(([a], [b]) => a.localeCompare(b))
  return createHash('sha256').update(JSON.stringify(sorted)).digest('hex').slice(0, 16)
}

Test: tests/utils-computePluginHash.spec.js — table-driven with mock project dirs.

Implementation Step 2: New utility — lib/utils/prebuiltCache.js

Portable cache management (no AAT dependencies). Functions:

export function getCachePaths (cacheRoot, pluginHash, theme, menu)
// Returns { sharedDir, cssDir } paths

export async function hasCachedBuild (cacheRoot, pluginHash, theme, menu)
// Returns boolean — checks both shared + CSS dirs exist

export async function populateCache (buildOutputDir, cacheRoot, pluginHash, theme, menu)
// After a full grunt build, copies shared artifacts into cache dirs
// Shared: adapt/js/*, templates.js, plugin assets/*, libraries/*, index.html (with @@placeholders)
// CSS: adapt/css/*
// Uses atomic rename: writes to temp dir first, then renames into place (parallel-safe)
// NOTE: Must save index.html BEFORE replace ran, or restore placeholders from source template

export async function restoreFromCache (cacheRoot, pluginHash, theme, menu, destDir)
// Copies cached shared + CSS artifacts to destDir

export function invalidateCache (cacheRoot)
// Removes entire prebuilt-cache directory (called on plugin/framework change)

Test: tests/utils-prebuiltCache.spec.js — test populate/restore/invalidate with temp dirs.

Implementation Step 3: New utility — lib/utils/generateLanguageManifest.js

Generates the language_data_manifest.js file listing JSON filenames in a language dir. The framework runtime reads this to know which files to fetch.

export function generateLanguageManifest (jsonFileNames) {
  return jsonFileNames.filter(f => f !== 'language_data_manifest.js' && f !== 'assets.json')
}

~5 lines. Input is the list of filenames we just wrote (known in memory from this.courseData), so no filesystem glob needed.

Test: tests/utils-generateLanguageManifest.spec.js

Implementation Step 4: New utility — lib/utils/applyBuildReplacements.js

Applies @@ placeholder substitutions in index.html using in-memory data. Only 4 replacements needed:

export async function applyBuildReplacements (buildDir, { defaultLanguage, defaultDirection, buildType, timestamp }) {
  // Read index.html from buildDir
  // Replace @@config._defaultLanguage → defaultLanguage
  // Replace @@config._defaultDirection → defaultDirection
  // Replace @@build.type → buildType (e.g. 'development')
  // Replace @@build.timestamp → timestamp
  // Write back
}

~20 lines total. Values come from this.courseData.config.data (already in memory) and build metadata.

Test: tests/utils-applyBuildReplacements.spec.js — provide template with placeholders, verify substitution.

Implementation Step 5: Modify AdaptFrameworkModule.js

Add cache management and invalidation hooks.

// In init(), after framework install:
const contentplugin = await this.app.waitForModule('contentplugin')
contentplugin.postInsertHook.tap(() => this.invalidatePrebuiltCache())
contentplugin.postUpdateHook.tap(() => this.invalidatePrebuiltCache())
contentplugin.postDeleteHook.tap(() => this.invalidatePrebuiltCache())
this.postUpdateHook.tap(() => this.invalidatePrebuiltCache())

// New method:
invalidatePrebuiltCache () {
  const cacheRoot = path.join(this.getConfig('buildDir'), 'prebuilt-cache')
  invalidateCache(cacheRoot)
  this._pluginHash = null
}

// New getter (cached):
getPluginHash () {
  if (!this._pluginHash) {
    this._pluginHash = computePluginHash(this.getConfig('frameworkDir'))
  }
  return this._pluginHash
}

Implementation Step 6: Modify AdaptFrameworkBuild.js

Add cache-hit fast path for preview builds.

In build() method, after loadCourseData() and before the existing copyFrameworkSource() + AdaptCli.buildCourse() block:

// Check for cached preview build
if (this.isPreview) {
  const framework = await App.instance.waitForModule('adaptframework')
  const cacheRoot = path.join(framework.getConfig('buildDir'), 'prebuilt-cache')
  const pluginHash = framework.getPluginHash()
  const theme = this.courseData.config.data._theme
  const menu = this.courseData.config.data._menu

  if (await hasCachedBuild(cacheRoot, pluginHash, theme, menu)) {
    // CACHE HIT — fast path
    await restoreFromCache(cacheRoot, pluginHash, theme, menu, this.buildDir)
    await this.copyAssets()
    await this.preBuildHook.invoke(this)
    await this.writeContentJson()
    // Write language manifest
    const langDir = this.courseData.course.dir
    const manifest = generateLanguageManifest(
      Object.values(this.courseData).filter(d => d.dir === langDir).map(d => d.fileName)
    )
    await writeJson(path.join(langDir, 'language_data_manifest.js'), manifest)
    // Apply @@replacements in index.html
    await applyBuildReplacements(this.buildDir, {
      defaultLanguage: this.courseData.config.data._defaultLanguage ?? 'en',
      defaultDirection: this.courseData.config.data._defaultDirection ?? 'ltr',
      buildType: 'development',
      timestamp: Date.now()
    })
    await this.postBuildHook.invoke(this)
    this.location = path.join(this.dir, 'build')
    this.buildData = await this.recordBuildAttempt()
    return this
  }
}

// Existing flow (cache miss or non-preview)
// ... copyFrameworkSource, AdaptCli.buildCourse(), etc.

// After successful grunt build for preview, populate cache
if (this.isPreview) {
  await populateCache(this.buildDir, cacheRoot, pluginHash, theme, menu)
}

Implementation Step 7: Verify copyFrameworkSource.js

For cache-population builds, we include ALL plugins (no filtering). Check whether passing no enabledPlugins (or all plugins) to copyFrameworkSource() works. Current filter logic skips plugins not in the list — we need to ensure all are included for the initial build.

---

Files to modify

File	Change
lib/AdaptFrameworkModule.js	Cache invalidation hooks, getPluginHash(), invalidatePrebuiltCache()
lib/AdaptFrameworkBuild.js	Cache-hit fast path in build(), cache population after grunt
New: lib/utils/computePluginHash.js	Deterministic hash from plugin set (uses adapt-cli Project)
New: lib/utils/prebuiltCache.js	Cache CRUD operations (portable, no AAT deps)
New: lib/utils/generateLanguageManifest.js	Language manifest generation (~5 lines)
New: lib/utils/applyBuildReplacements.js	@@ placeholder substitution in index.html
lib/utils.js	Barrel export updates
New: tests/utils-computePluginHash.spec.js
New: tests/utils-prebuiltCache.spec.js
New: tests/utils-generateLanguageManifest.spec.js
New: tests/utils-applyBuildReplacements.spec.js

adapt-cli portability

The new utilities are designed to be moved to adapt-cli later:

• computePluginHash — already uses adapt-cli's Project class
• prebuiltCache — pure filesystem operations, no AAT dependencies
• generateLanguageManifest — pure function, no dependencies
• applyBuildReplacements — standalone function, plain data inputs, no AAT dependencies

When later phases extract grunt tasks into a JS API in adapt_framework, these utilities can be consumed by adapt-cli's buildCourse() to offer the same caching for standalone CLI builds.

Expected preview build times

Step	Time
Copy pre-built artifacts	~0.5s
Write course JSON	~0.1s
Copy course assets	~0.5-2s (depends on asset count)
Language manifest + replace	~0.1s
Total	~1.5-3s

Down from ~60s.

Verification

1. Unit tests for all new utilities
2. Cache miss: first preview build takes ~60s (same as today) and populates cache
3. Cache hit: second preview of same or different course takes ~2-5s
4. Cache invalidation: install/update/remove a plugin → next preview is a cache miss
5. Publish builds: unaffected (always run full grunt)
6. Diff verification: diff -r between cache-hit and full-grunt build output should be identical (except timestamp)
7. Multi-language builds (which call buildCourse() multiple times with outputDir) work with cache

Resolved questions

1. Schema defaults: NOT needed at build time. The jsonschema module applies defaults when data is saved to the DB via Schema.validate() with useDefaults: true. Data already has defaults by the time the build reads it.

2. Language manifests: Trivial utility (~5 lines) kept in adaptframework since it's part of the standard build flow. Lists JSON filenames in a language dir, excluding manifest itself and assets.json.

3. Replace pattern format: Uses @@path.to.value dot-notation. Template is src/core/required/index.html. Only 4 placeholders: @@config._defaultLanguage, @@config._defaultDirection, @@build.type, @@build.timestamp. Simple string replacement, no complex pattern engine needed.

4. Plugin scripts (adaptpostcopy/adaptpostbuild): Only adapt-contrib-spoor uses adaptpostcopy — copies SCORM files based on config. Only relevant for publish/export builds. Since we only cache for preview builds, this is not a concern.

---

Future Phases (out of scope for this issue)

Phase 2: Extract Build Functions from Grunt (adapt_framework)

Extract each grunt task into a standalone async function with a JS API in adapt_framework/lib/build/. Keep grunt tasks intact during transition.

Phase 3: BuildPipeline + AAT Direct Integration

Replace AdaptCli.buildCourse() subprocess with direct BuildPipeline call. Eliminates subprocess overhead (~2-4s), enables parallel LESS + HBS + JS (~20-30% speedup), eliminates framework directory copy.

Phase 4: Advanced Optimisations

• Persistent in-memory Rollup cache across builds
• Concurrent build deduplication (multiple languages share compilation)

Phase 5: Remove Grunt (adapt_framework)

Delete Gruntfile.js and grunt/ directory, remove ~15 grunt dependencies, update adapt-cli to use JS API.

Expected Speed Improvements Summary

Optimisation	Saving	Phase
Pre-built cache for previews (cache hit)	~60s → ~1.5-3s	1
No subprocess (npx + grunt startup)	~2-4s on publish	3
Parallel LESS + HBS + JS	~20-30% on publish	3
No framework dir copy	~1-5s on all builds	3
In-memory Rollup cache (cache miss)	~0.5-2s	4
Concurrent build dedup	~N-1 redundant passes	4