From a13933b8a790d317d5df3bb9a048e19e30d235b0 Mon Sep 17 00:00:00 2001 From: avivkeller Date: Tue, 4 Aug 2026 12:09:13 -0400 Subject: [PATCH] chore: make doc-kit friendly --- .changeset/bootstrap-command.md | 5 + .changeset/install-command.md | 5 + .changeset/serve-command.md | 5 + README.md | 100 +++- docs/cli.md | 94 +++ docs/commands.md | 185 ------ docs/configuration.md | 188 +++--- docs/creating-generators.md | 475 +++++++++++++++ docs/customization.md | 174 ++++++ docs/generators.md | 558 ++---------------- docs/publishing.md | 71 +++ docs/troubleshooting.md | 77 +++ docs/writing-docs.md | 180 ++++++ packages/core/README.md | 42 +- packages/core/bin/cli.mjs | 2 +- packages/core/bin/commands/bootstrap.mjs | 186 ++++++ packages/core/bin/commands/generate.mjs | 57 +- packages/core/bin/commands/index.mjs | 5 +- packages/core/bin/commands/install.mjs | 55 ++ packages/core/bin/commands/options.mjs | 39 ++ packages/core/bin/commands/serve.mjs | 96 +++ .../core/src/server/__tests__/index.test.mjs | 186 ++++++ packages/core/src/server/index.mjs | 223 +++++++ .../src/utils/__tests__/bootstrap.test.mjs | 125 ++++ .../core/src/utils/__tests__/install.test.mjs | 150 +++++ packages/core/src/utils/bootstrap.mjs | 164 +++++ packages/core/src/utils/install.mjs | 187 ++++++ packages/legacy/README.md | 38 ++ packages/node/README.md | 31 + packages/react/README.md | 43 ++ playwright.config.js | 2 +- scripts/build-docs-content.mjs | 15 +- www/doc-kit.config.mjs | 81 ++- www/pages/getting-started.md | 156 ++--- www/pages/index.md | 77 +-- 35 files changed, 2999 insertions(+), 1078 deletions(-) create mode 100644 .changeset/bootstrap-command.md create mode 100644 .changeset/install-command.md create mode 100644 .changeset/serve-command.md create mode 100644 docs/cli.md delete mode 100644 docs/commands.md create mode 100644 docs/creating-generators.md create mode 100644 docs/customization.md create mode 100644 docs/publishing.md create mode 100644 docs/troubleshooting.md create mode 100644 docs/writing-docs.md create mode 100644 packages/core/bin/commands/bootstrap.mjs create mode 100644 packages/core/bin/commands/install.mjs create mode 100644 packages/core/bin/commands/options.mjs create mode 100644 packages/core/bin/commands/serve.mjs create mode 100644 packages/core/src/server/__tests__/index.test.mjs create mode 100644 packages/core/src/server/index.mjs create mode 100644 packages/core/src/utils/__tests__/bootstrap.test.mjs create mode 100644 packages/core/src/utils/__tests__/install.test.mjs create mode 100644 packages/core/src/utils/bootstrap.mjs create mode 100644 packages/core/src/utils/install.mjs create mode 100644 packages/legacy/README.md create mode 100644 packages/node/README.md create mode 100644 packages/react/README.md diff --git a/.changeset/bootstrap-command.md b/.changeset/bootstrap-command.md new file mode 100644 index 00000000..813f6f9a --- /dev/null +++ b/.changeset/bootstrap-command.md @@ -0,0 +1,5 @@ +--- +'@nodejs/doc-kit': minor +--- + +Add `doc-kit bootstrap [generators...]` diff --git a/.changeset/install-command.md b/.changeset/install-command.md new file mode 100644 index 00000000..245351a9 --- /dev/null +++ b/.changeset/install-command.md @@ -0,0 +1,5 @@ +--- +'@nodejs/doc-kit': minor +--- + +Add `doc-kit install [generators...]` diff --git a/.changeset/serve-command.md b/.changeset/serve-command.md new file mode 100644 index 00000000..db0b9572 --- /dev/null +++ b/.changeset/serve-command.md @@ -0,0 +1,5 @@ +--- +'@nodejs/doc-kit': minor +--- + +Add `doc-kit serve` diff --git a/README.md b/README.md index b24f5c68..f7df5b5d 100644 --- a/README.md +++ b/README.md @@ -46,16 +46,26 @@ $ node packages/core/bin/cli.mjs --help ``` Usage: @nodejs/doc-kit [options] [command] -CLI tool to generate the Node.js API documentation +CLI tool to generate API documentation Options: - --log-level Log level (choices: "debug", "info", "warn", "error", - "fatal", default: "info") - -h, --help display help for command + --log-level Log level (choices: "debug", "info", + "warn", "error", "fatal", default: + "info") + -h, --help display help for command Commands: - generate [options] Generate API docs - help [command] display help for command + bootstrap [options] [generators...] Set up a project for doc-kit: a + configuration file wired to your + package.json, a documentation directory, + and the generator packages + generate [options] Generate API docs + install [options] [generators...] Install the packages providing built-in + generators (defaults to the targets in + your configuration file) + serve [options] Generate API docs, serve them locally, + and regenerate on changes + help [command] display help for command ``` ### `generate` @@ -78,14 +88,14 @@ Options: (json-simple, legacy-html, legacy-html-all, man-page, legacy-json, legacy-json-all, addon-verify, api-links, orama-db, llms-txt, - sitemap, web) or an import specifier for a custom - generator + sitemap, html) or an import specifier for a + custom generator --ignore Ignore file patterns (glob) -o, --output The output directory -p, --threads Number of threads to use (minimum: 1) --chunk-size Number of items to process per worker thread (minimum: 1) - -v, --version Target Node.js version + -v, --version Target project version -c, --changelog Changelog URL or path --git-ref Git ref --index index.md URL or path @@ -94,6 +104,76 @@ Options: -h, --help display help for command ``` +### `serve` + +Generates the documentation, serves it locally, and regenerates whenever the +input files change — the fastest way to preview docs while writing them. It +accepts the same configuration options as `generate`, plus: + +``` +Options: + --port Preferred port (falls back to the next available + one) (default: 3000) + --static Serve the existing output as-is, without + generating or watching +``` + +### `bootstrap` + +Sets a project up end to end: a `doc-kit.config.mjs` wired to your +`package.json`, a documentation directory (detected, or created with a starter +page), a `.gitignore` entry for the output, and the generator packages +installed. Prompts for its few decisions on a TTY; `--yes` accepts the +defaults. A new project only needs: + +```sh +npx doc-kit bootstrap && npx doc-kit serve +``` + +``` +Usage: @nodejs/doc-kit bootstrap [options] [generators...] + +Set up a project for doc-kit: a configuration file wired to your package.json, a +documentation directory, and the generator packages + +Arguments: + generators Built-in generator names (json-simple, legacy-html, + legacy-html-all, man-page, legacy-json, + legacy-json-all, addon-verify, api-links, orama-db, + llms-txt, sitemap, html) + +Options: + -y, --yes Accept all defaults without prompting + --dir The documentation directory + -o, --output The output directory (default: "out") + --force Overwrite an existing configuration file + -h, --help display help for command +``` + +### `install` + +Installs the packages providing built-in generators (e.g. `doc-kit install html` +installs `@nodejs/doc-kit-generator-react`). Without arguments, it installs +whatever the `target` in your configuration file needs. The package manager is +detected from the project lockfile. + +``` +Usage: @nodejs/doc-kit install [options] [generators...] + +Install the packages providing built-in generators (defaults to the targets in +your configuration file) + +Arguments: + generators Built-in generator names (json-simple, legacy-html, + legacy-html-all, man-page, legacy-json, legacy-json-all, + addon-verify, api-links, orama-db, llms-txt, sitemap, + html) + +Options: + --config-file Config file + -h, --help display help for command +``` + ## Examples ### Legacy @@ -126,5 +206,5 @@ npx doc-kit generate \ > In order to use the search functionality, you _must_ serve the output directory. > > ```sh -> npx serve out +> npx doc-kit serve --static -o out > ``` diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 00000000..3564cb54 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,94 @@ +# CLI + +The `doc-kit` command-line interface. Every option that maps to +configuration can also live in a [configuration file](./configuration.md); +when both are present, CLI flags win (rule of specifity). + +```sh +npx doc-kit [command] [options] +``` + +One option applies to every command: + +| Option | Description | +| --------------------- | ------------------------------------------------------ | +| `--log-level ` | `debug`, `info` (default), `warn`, `error`, or `fatal` | + +## `doc-kit bootstrap` + +```sh +npx doc-kit bootstrap [options] [generators...] +``` + +Sets a project up end to end: a `doc-kit.config.mjs` wired to your +`package.json`, a documentation directory (detected, or created with a +starter page), a `.gitignore` entry for the output, and the generator +packages installed. On a terminal it prompts for its few decisions; +otherwise (or with `--yes`) it accepts the defaults. + +| Option | Description | +| ---------------------- | --------------------------------------------- | +| `[generators...]` | Built-in generator names. **Default:** `html` | +| `-y`, `--yes` | Accept all defaults without prompting | +| `--dir ` | The documentation directory | +| `-o`, `--output ` | The output directory. **Default:** `out` | +| `--force` | Overwrite an existing configuration file | + +## `doc-kit generate` + +```sh +npx doc-kit generate [options] +``` + +Runs the generators and writes their output. Requires a `target` and an +`input`, from flags or the configuration file. + +| Option | Description | +| ----------------------------- | ------------------------------------------------------------- | +| `--config-file ` | Use a specific configuration file instead of searching | +| `-i`, `--input ` | Input file patterns (glob) | +| `-t`, `--target ` | Generator name(s), or import specifiers for custom generators | +| `--ignore ` | Input patterns to skip | +| `-o`, `--output ` | The output directory | +| `-v`, `--version ` | The version of the project being documented | +| `-c`, `--changelog ` | Changelog URL or path (release history for version selectors) | +| `--git-ref ` | Git ref used in source links | +| `--index ` | `index.md` URL or path | +| `--minify` | Minify the output | +| `--type-map ` | Type map URL or path (custom type-name → URL links) | +| `-p`, `--threads ` | Worker threads to use (minimum 1) | +| `--chunk-size ` | Items per worker thread (minimum 1) | + +## `doc-kit serve` + +```sh +npx doc-kit serve [options] +``` + +Generates the documentation, serves it locally, and regenerates whenever +input files change — the writing loop. Accepts every `generate` option, +plus: + +| Option | Description | +| ----------------- | ---------------------------------------------------------------------- | +| `--port ` | Preferred port, falling back to the next free one. **Default:** `3000` | +| `--static` | Serve the existing output as-is, without generating or watching | + +`--static` is also the production preview: it serves any built output +directory over HTTP, which the `html` output requires. + +## `doc-kit install` + +```sh +npx doc-kit install [generators...] +``` + +Installs the packages providing built-in generators — `doc-kit install html` +installs `@nodejs/doc-kit-generator-react`. With no arguments, it installs +whatever the `target` in your configuration file needs. The package manager +is detected from your lockfile (npm, pnpm, yarn, or bun). + +| Option | Description | +| ---------------------- | ------------------------------------------------------ | +| `[generators...]` | Built-in generator names | +| `--config-file ` | Use a specific configuration file instead of searching | diff --git a/docs/commands.md b/docs/commands.md deleted file mode 100644 index 21785977..00000000 --- a/docs/commands.md +++ /dev/null @@ -1,185 +0,0 @@ -# Creating Commands - -## Command Structure - -Commands in `doc-kit` are defined as modules that export a command object conforming to the `Command` interface: - -```typescript -interface Command { - name: string; - description: string; - options: { [key: string]: Option }; - action: (options: any) => Promise; -} -``` - -Each command consists of: - -- **name**: The command name used in the CLI (e.g., `generate`) -- **description**: A short description shown in help text -- **options**: An object mapping option names to their definitions -- **action**: The async function that executes when the command is run - -## Creating a New Command - -### Step 1: Create the Command File - -Create a new file in `packages/core/bin/commands/` with your command name: - -```javascript -// packages/core/bin/commands/my-command.mjs -import logger from '../../src/logger/index.mjs'; - -/** - * @type {import('./types').Command} - */ -export default { - name: 'my-command', - description: 'Does something useful', - - options: { - // Define your options here (see next section) - }, - - async action(opts) { - logger.info('Starting my-command', opts); - - // Your command logic here - - logger.info('Completed my-command'); - }, -}; -``` - -### Step 2: Register the Command - -Add your command to the exports in `packages/core/bin/commands/index.mjs`: - -```javascript -import generate from './generate.mjs'; -import myCommand from './my-command.mjs'; // Add this - -export default [ - generate, - myCommand, // Add this -]; -``` - -### Step 3: Update CLI Entry Point - -The CLI in `packages/core/bin/cli.mjs` automatically loads commands from `packages/core/bin/commands/index.mjs`, so no changes are needed there if you followed step 2. - -## Command Options - -Options define the flags and parameters your command accepts. Each option has: - -```typescript -interface Option { - flags: string[]; // CLI flags (e.g., ['-i', '--input ']) - desc: string; // Description for help text -} -``` - -### Defining Options - -```javascript -options: { - input: { - flags: ['-i', '--input '], - desc: 'Input file patterns (glob)', - prompt: { - type: 'text', - message: 'Enter input glob patterns', - variadic: true, - required: true, - }, - }, - - force: { - flags: ['-f', '--force'], - desc: 'Force overwrite existing files', - prompt: { - type: 'confirm', - message: 'Overwrite existing files?', - initialValue: false, - }, - }, - - mode: { - flags: ['-m', '--mode '], - desc: 'Operation mode', - prompt: { - type: 'select', - message: 'Choose operation mode', - options: [ - { label: 'Fast', value: 'fast' }, - { label: 'Thorough', value: 'thorough' }, - ], - }, - }, -} -``` - -### Flag Syntax - -- `` - Required argument -- `[value]` - Optional argument -- `` - Variadic (multiple values) -- `[values...]` - Optional variadic - -### Option Types - -#### `text` - -Single-line text input. - -```javascript -prompt: { - type: 'text', - message: 'Enter a value', - initialValue: 'default', - required: true, -} -``` - -#### `confirm` - -Yes/no confirmation. - -```javascript -prompt: { - type: 'confirm', - message: 'Are you sure?', - initialValue: false, -} -``` - -#### `select` - -Single choice from a list. - -```javascript -prompt: { - type: 'select', - message: 'Choose one', - options: [ - { label: 'Option 1', value: 'opt1' }, - { label: 'Option 2', value: 'opt2' }, - ], -} -``` - -#### `multiselect` - -Multiple choices from a list. - -```javascript -prompt: { - type: 'multiselect', - message: 'Choose multiple', - options: [ - { label: 'Choice A', value: 'a' }, - { label: 'Choice B', value: 'b' }, - ], -} -``` diff --git a/docs/configuration.md b/docs/configuration.md index 214d2857..19c3664b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,143 +1,91 @@ # Configuration -`doc-kit` uses [`cosmiconfig`](https://github.com/cosmiconfig/cosmiconfig) to -discover and load configuration. Run the CLI from your project directory and it -will automatically look for a `doc-kit` property in `package.json`, rc files -such as `.doc-kitrc.yml`, and module files such as `doc-kit.config.mjs`. - -Use `--config-file ` to load a specific file instead of searching. - -## Configuration File Format - -Configuration files can be either: - -- **JavaScript** (`.js`, `.mjs`, `.cjs`) -- **TypeScript** (`.ts`, when `typescript` is installed in the project) -- **JSON** (`.json`) -- **YAML** (`.yaml`, `.yml`, or an extensionless rc file) - -JavaScript and TypeScript configuration files export the configuration object. -JSON and YAML files contain the object directly. A `package.json` configuration -uses the `doc-kit` property: - -```json -{ - "doc-kit": { - "target": ["json"], - "global": { - "input": "doc/api/*.md", - "output": "out" - } - } -} -``` - -### Basic Example - -```javascript +`doc-kit` discovers configuration with +[`cosmiconfig`](https://github.com/cosmiconfig/cosmiconfig): run the CLI +from your project directory and it looks for a `doc-kit.config.mjs` (or +`.js`/`.cjs`/`.ts`), a `.doc-kitrc` file (JSON or YAML, with any of the +usual extensions), or a `doc-kit` property in `package.json`. Use +`--config-file ` to load a specific file instead of searching. + +```mjs displayName="doc-kit.config.mjs" +/** @type {import('@nodejs/doc-kit/utils/configuration/types').Configuration} */ export default { - // Targets, alternatively supplied by command line flags. Each entry is - // either a built-in shorthand name or an import specifier resolving to a - // generator module (e.g. '@my-scope/my-package/my-generator'). - target: ['orama-db', 'html'], - global: { - version: '20.0.0', - minify: true, - repository: 'nodejs/node', - ref: 'main', - baseURL: 'https://nodejs.org/docs/', - input: 'src/', - output: 'dist/', - ignore: ['node_modules/', 'test/'], - changelog: - 'https://raw.githubusercontent.com/nodejs/node/main/CHANGELOG.md', - index: - 'https://raw.githubusercontent.com/nodejs/node/main/doc/api/index.md', - }, - - threads: 4, - chunkSize: 10, + // Which generators to run. Built-in names, or import specifiers + // resolving to custom generator modules. + target: ['html', 'orama-db'], - // Generator-specific configurations - json: { - format: 'json', - minify: false, // Override global setting + global: { + input: ['docs/**/*.md'], + output: 'out', + version: '1.2.0', + baseURL: 'https://example.com/docs', + changelog: [], }, + // Generator-specific sections, keyed by generator name html: { - format: 'html', - }, - - metadata: { - typeMap: { - String: 'string', - Number: 'number', - Boolean: 'boolean', - }, + project: 'My Project', }, }; ``` -## Configuration Structure +## How values merge -### Global Configuration +Three sources, in order of precedence: -The `global` object contains settings that apply to all generators unless overridden: +1. **CLI flags** (see the [CLI reference](./cli.md)) override +2. **the configuration file**, which overrides +3. **built-in defaults**. -| Property | Type | Description | Default | -| ------------ | ------------------ | ------------------------------------------ | -------------------------------------------------- | -| `version` | `string \| SemVer` | Documentation version | `process.version` | -| `minify` | `boolean` | Whether to minify output | `true` | -| `repository` | `string` | GitHub repository in `owner/repo` format | `'nodejs/node'` | -| `ref` | `string` | Git reference (branch, tag, or commit SHA) | `'HEAD'` | -| `baseURL` | `string \| URL` | Base URL for documentation | `'https://nodejs.org/docs'` | -| `input` | `string[]` | Input directory path | - | -| `output` | `string` | Output directory path | - | -| `ignore` | `string[]` | Patterns to ignore | `[]` | -| `changelog` | `string \| URL` | Changelog URL | Auto-generated URL based on `ref` and `repository` | -| `index` | `string \| URL` | Index URL | - | +Each generator's section starts from its own defaults, then inherits every +`global` value it doesn't override. So `global.minify` applies to all +targets, while `'legacy-json': { minify: false }` exempts one. -### Generator-Specific Configuration +## Global options -Each generator (e.g., `json`, `html`, `markdown`) can have its own configuration that overrides global settings: +| Property | Type | Description | Default | +| ------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | +| `input` | `string \| string[]` | Glob patterns for the source Markdown files. Required (with `target`) to run. | — | +| `output` | `string` | The directory generated files are written to. | — | +| `ignore` | `string \| string[]` | Glob patterns excluded from `input`. | — | +| `version` | `string` | The version of the project being documented (coerced to semver). | `process.version` | +| `changelog` | `string \| URL \| Array` | Release history used to build version selectors. A URL or path to a `CHANGELOG.md` to parse, or a pre-parsed array — `[]` disables versioning (and the network fetch). | The Node.js `CHANGELOG.md` | +| `index` | `string \| URL \| Array` | An `index.md` listing section titles, or a pre-parsed array. | — | +| `baseURL` | `string \| URL` | The public URL of the published site; used wherever absolute links are needed (sitemaps, `llms.txt`, social metadata). | `'https://nodejs.org/docs'` | +| `repository` | `string` | GitHub repository in `owner/repo` form, used for source and edit links. | `'nodejs/node'` | +| `ref` | `string` | Git ref (branch, tag, or SHA) used in source links. | `'HEAD'` | +| `minify` | `boolean` | Minify the output, in whatever form it takes. | `true` | +| `pathsToCopy` | `Array` | Extra files or directories copied into the output. A string copies to `output/`; a `{ source: destination }` object controls the target path. Missing paths are skipped. | `['assets', 'public', 'static']` | -```javascript +> The defaults still reflect doc-kit's Node.js origins: `changelog`, +> `repository`, and `baseURL` point at the Node.js project unless you set +> them. `doc-kit bootstrap` writes a configuration that overrides the ones +> your project needs. + +## Execution options + +Top-level, alongside `target` and `global`: + +| Property | Type | Description | Default | +| ----------- | -------- | ----------------------------------- | -------------- | +| `threads` | `number` | Worker threads used for generation. | Your CPU count | +| `chunkSize` | `number` | Items processed per worker thread. | `10` | + +## Generator options + +Each generator documents its own options on its reference page — see the +[generators overview](./generators.md). Two commonly configured ones: + +```js export default { - global: { - version: '20.0.0', - minify: true, + html: { + project: 'My Project', }, - 'legacy-json': { - minify: false, // Override: JSON output won't be minified + metadata: { + typeMap: { + MyThing: 'https://example.com/docs/my-thing.html', + }, }, }; ``` - -## Configuration Merging - -Configurations are merged in the following order (higher sources take -precedence): - -1. **CLI options** (command-line arguments) -2. **Configuration file** (discovered or selected with `--config-file`) -3. **Default values** (built-in defaults) - -## CLI Options Mapping - -CLI options map to configuration properties: - -| CLI Option | Config Property | Example | -| ---------------------- | ------------------ | ------------------------- | -| `--input ` | `global.input` | `--input src/` | -| `--output ` | `global.output` | `--output dist/` | -| `--ignore ` | `global.ignore[]` | `--ignore test/` | -| `--minify` | `global.minify` | `--minify` | -| `--git-ref ` | `global.ref` | `--git-ref v20.0.0` | -| `--version ` | `global.version` | `--version 20.0.0` | -| `--changelog ` | `global.changelog` | `--changelog https://...` | -| `--index ` | `global.index` | `--index file://...` | -| `--type-map ` | `metadata.typeMap` | `--type-map file://...` | -| `--target ` | `target` | `--target json` | -| `--threads ` | `threads` | `--threads 4` | -| `--chunk-size ` | `chunkSize` | `--chunk-size 10` | diff --git a/docs/creating-generators.md b/docs/creating-generators.md new file mode 100644 index 00000000..3ec11000 --- /dev/null +++ b/docs/creating-generators.md @@ -0,0 +1,475 @@ +# Creating Generators + +This guide explains how to create new documentation generators for `@nodejs/doc-kit`. + +## Generator Concepts + +Generators in `doc-kit` transform API documentation through a pipeline. Each generator: + +1. **Takes input** from a previous generator or raw files +2. **Processes the data** into a different format +3. **Yields output** for the next generator or final output + +### Generator Pipeline + +``` +Raw Markdown Files + ↓ + [ast] - Parse to MDAST + ↓ + [metadata] - Extract structured metadata + ↓ + [jsx-ast] - Convert to JSX AST + ↓ + [html] - Generate HTML/CSS/JS bundles +``` + +Each generator declares its dependency using the `dependsOn` field, allowing automatic pipeline construction. + +## Generator Structure + +A generator is defined as a module exporting an object conforming to the `GeneratorMetadata` interface. + +## Creating a Basic Generator + +### Step 1: Create the Generator Files + +Create a new directory in your project: + +``` +/ +├── index.mjs # Generator metadata (required) +├── generate.mjs # Generator implementation (required) +├── constants.mjs # Constants (optional) +├── types.d.ts # TypeScript types (required) +└── utils/ # Utility functions (optional) + └── formatter.mjs +``` + +### Step 2: Define Types + +Create a `types.d.ts` file containing a `Generator` export. Use this when typing your generator. + +```ts +export type Generator = GeneratorMetadata< + { + // If your generator supports a custom configuration, + // define it here + myCustomOption: string; + }, + Generate>, + // If your generator supports parallel processing: + ProcessChunk< + InputToMyParallelProcessor, + OutputOfMyParallelProcessor, + DependenciesOfMyParallelProcessor + > +>; +``` + +### Step 3: Define Generator Metadata + +A generator module's default export is a plain object with its metadata and +implementation. Create it in `index.mjs`: + +```javascript +import { generate } from './generate.mjs'; + +/** + * Generates output in MyFormat. + * + * @type {import('./types').Generator} + */ +export default { + name: 'my-format', + + description: 'Generates documentation in MyFormat', + + // This generator depends on the metadata generator. Dependencies are + // declared as import specifiers, so they can live in any package. + dependsOn: '@nodejs/doc-kit/metadata', + + defaultConfiguration: { + // If your generator supports a custom configuration, define the defaults here + myCustomOption: 'myDefaultValue', + + // All generators support options in the GlobalConfiguration object + // To override the defaults, they can be specified here + ref: 'overriddenRef', + }, + + generate, +}; +``` + +### Step 4: Implement the Generator Logic + +Create the generator implementation in `generate.mjs`: + +```javascript +import { writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; + +import getConfig from '../../utils/configuration/index.mjs'; + +/** + * Main generation function + * + * @type {import('./types').Generator['generate']} + */ +export async function generate(input, worker) { + const config = getConfig('my-format'); + + // Transform input to your format + const result = transformToMyFormat(input, config.version); + + // Write to file if output directory specified + if (config.output) { + await writeFile( + join(config.output, 'documentation.myformat'), + result, + 'utf-8' + ); + } + + return result; +} + +/** + * Transform metadata entries to MyFormat + * @param {Array} entries + * @param {import('semver').SemVer} version + * @returns {string} + */ +function transformToMyFormat(entries, version) { + // Your transformation logic here + return entries + .map(entry => `${entry.api}: ${entry.heading.data.name}`) + .join('\n'); +} +``` + +### Step 5: Make the Generator Loadable + +Generators are loaded dynamically by import specifier. Anything that resolves +to a module whose default export is a generator works as a `--target`: + +```bash +# A package (subpath) export +doc-kit generate -t @my-scope/my-package/my-format ... + +# A local file +doc-kit generate -t ./generators/my-format/index.mjs ... +``` + +Built-in generators additionally get a shorthand alias in +`packages/core/src/generators/index.mjs`, which maps the name users type to +the import specifier it resolves to: + +```javascript +export const publicGenerators = { + 'json-simple': '@nodejs/doc-kit/json-simple', + 'my-format': '@nodejs/doc-kit/my-format', // Add this + // ... other generators +}; +``` + +If the generator lives in this repository, also add a matching subpath to the +`exports` map of its package's `package.json`. + +## Parallel Processing with Workers + +For generators processing large datasets, implement parallel processing using worker threads. + +### Implementing Worker-Based Processing + +First, define the generator metadata in `index.mjs`: + +```javascript +import { generate, processChunk } from './generate.mjs'; + +/** + * @type {import('./types').Generator} + */ +export default { + name: 'parallel-generator', + + description: 'Processes data in parallel', + + dependsOn: '@nodejs/doc-kit/metadata', + + // Indicates this generator has a processChunk implementation + hasParallelProcessor: true, + + generate, + processChunk, +}; +``` + +Then, implement both `processChunk` and `generate` in `generate.mjs`: + +```javascript +import getConfig from '../../utils/configuration/index.mjs'; + +/** + * Process a chunk of items in a worker thread. + * This function runs in isolated worker threads. + * + * @type {import('./types').Generator['processChunk']} + */ +export async function processChunk(fullInput, itemIndices, deps) { + const results = []; + + // Process only the items at specified indices + for (const idx of itemIndices) { + const item = fullInput[idx]; + const result = await processItem(item, deps); + results.push(result); + } + + return results; +} + +/** + * Main generation function that orchestrates worker threads + * + * @type {import('./types').Generator['generate']} + */ +export async function* generate(input, worker) { + // Configuration for this generator is based on its name + const config = getConfig('my-format'); + + // Prepare serializable dependencies + const deps = { + version: config.version, + // ...other config + }; + + // Stream chunks as they complete + for await (const chunkResult of worker.stream(input, deps)) { + // Process chunk result if needed + yield chunkResult; + } +} +``` + +### Key Points for Worker Processing + +1. **`processChunk` executes in worker threads** - No access to main thread state +2. **Only serializable data** can be passed to workers (no functions, classes, etc.) +3. **`fullInput` and `itemIndices`** - Workers receive full input but only process specified indices +4. **`deps` must be serializable** - Pass only JSON-compatible data + +### When to Use Workers + +Use parallel processing when: + +- Processing many independent items (files, modules, entries) +- Each item takes significant time to process +- Operations are CPU-intensive + +Don't use workers when: + +- Items have dependencies on each other +- Output must be in specific order +- Operation is I/O bound rather than CPU bound + +## Streaming Results + +Generators can yield results as they're produced using async generators. + +Define the generator metadata in `index.mjs`: + +```javascript +import { generate, processChunk } from './generate.mjs'; + +/** + * @type {import('./types').Generator} + */ +export default { + name: 'streaming-generator', + + description: 'Streams results as they are ready', + + dependsOn: '@nodejs/doc-kit/metadata', + + hasParallelProcessor: true, + + generate, + processChunk, +}; +``` + +Implement the generator in `generate.mjs`: + +```javascript +/** + * Process a chunk of data + * + * @type {import('./types').Generator['processChunk']} + */ +export async function processChunk(fullInput, itemIndices, deps) { + // Process chunk + return results; +} + +/** + * Generator function that yields results incrementally + * + * @type {import('./types').Generator['generate']} + */ +export async function* generate(input, worker) { + // Stream results as workers complete chunks + for await (const chunkResult of worker.stream(input, {})) { + // Yield immediately - downstream can start processing + yield chunkResult; + } +} +``` + +### Benefits of Streaming + +- **Reduced memory usage** - Process data in chunks +- **Earlier downstream starts** - Next generator can begin before this one finishes +- **Better parallelism** - Multiple generators can work simultaneously + +### Non-Streaming Generators + +Some generators must collect all input before processing. + +Generator metadata in `index.mjs`: + +```javascript +import { generate } from './generate.mjs'; + +/** + * @type {import('./types').Generator} + */ +export default { + name: 'batch-generator', + + description: 'Requires all input at once', + + dependsOn: '@nodejs/doc-kit-generator-react/jsx-ast', + + generate, +}; +``` + +Implementation in `generate.mjs`: + +```javascript +/** + * Non-streaming - returns Promise instead of AsyncGenerator + * + * @type {import('./types').Generator['generate']} + */ +export async function generate(input, worker) { + // Collect all input (if dependency is streaming, this waits for completion) + const allData = await collectAll(input); + + // Process everything together + const result = processBatch(allData); + + return result; +} +``` + +Use non-streaming when: + +- You need all data to make decisions (e.g., code splitting, global analysis) +- Output format requires complete dataset +- Cross-references between items need resolution + +## Generator Dependencies + +### Declaring Dependencies + +In `index.mjs`: + +```javascript +import { generate } from './generate.mjs'; + +export default { + name: 'my-generator', + + // This generator requires the metadata generator's output. The dependency + // is an import specifier, so it may point at any installed package. + dependsOn: '@nodejs/doc-kit/metadata', + + // ... other metadata + + generate, +}; +``` + +In `generate.mjs`: + +```javascript +export async function generate(input, worker) { + // input contains the output from 'metadata' generator +} +``` + +## File Output + +### Writing Output Files + +In `generate.mjs`: + +```javascript +import { mkdir, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; + +import getConfig from '../../utils/configuration/index.mjs'; + +export async function generate(input, worker) { + const config = getConfig('my-format'); + + if (!config.output) { + // Return data without writing + return result; + } + + // Ensure directory exists + await mkdir(config.output, { recursive: true }); + + // Write single file + await writeFile(join(config.output, 'output.txt'), content, 'utf-8'); + + // Write multiple files + for (const item of items) { + await writeFile( + join(config.output, `${item.name}.txt`), + item.content, + 'utf-8' + ); + } + + return result; +} +``` + +### Copying Assets + +```javascript +import { cp } from 'node:fs/promises'; +import { join } from 'node:path'; + +import getConfig from '../../utils/configuration/index.mjs'; + +export async function generate(input, worker) { + const config = getConfig('my-format'); + + if (config.output) { + // Copy asset directory + await cp( + new URL('./assets', import.meta.url), + join(config.output, 'assets'), + { recursive: true } + ); + } + + return result; +} +``` diff --git a/docs/customization.md b/docs/customization.md new file mode 100644 index 00000000..e476243b --- /dev/null +++ b/docs/customization.md @@ -0,0 +1,174 @@ +# Customizing the site + +This page walks the customizations most projects make, in the +order they usually make them. The complete option list lives in the +[`html` generator reference](./generators/html.md). + +All of it goes in the `html` section of your configuration file: + +```js +// doc-kit.config.mjs +export default { + target: ['html'], + global: { + input: ['docs/**/*.md'], + output: 'out', + }, + + html: { + // ... everything on this page + }, +}; +``` + +## Name and titles + +```js +html: { + project: 'My Project', // Used in page titles and the version selector + title: '{project} Documentation', // Browser-tab title template + // Suppress the Node.js announcement banner fetched at runtime + remoteConfigUrl: '', +}, +``` + +`title` supports `{project}` and `{version}` placeholders. Projects that +bootstrap with `doc-kit bootstrap` get `project` wired to their +`package.json` automatically. + +## The `` + +`head` controls the markup injected into every page's ``: + +```js +html: { + head: { + meta: [ + { name: 'description', content: 'My project documentation' }, + { property: 'og:image', content: 'https://example.com/og.png' }, + ], + links: [{ rel: 'icon', href: '/favicon.ico' }], + html: [ + ``, + ], + }, +}, +``` + +The ``, - ], - }, - // use a custom logo instead of the Node.js logo - // our logo.jsx file like this, just for the demo - // export default Logo = () => - // ; - imports: { - '#theme/Logo': join(import.meta.dirname, './logo.jsx'), - // You can also change things such as `#theme/Layout`, - // and more! - }, + target: ['html'], + + global: { + input: ['docs/**/*.md'], + output: 'out', }, }; ``` -Re-build the project, serve, and you'll see how quickly you can change the -experience, preserving core functionality. +`doc-kit serve` and `doc-kit generate` read that configuration; every option can +also be passed as a CLI flag. See the +[configuration reference](./configuration.html) for everything the file accepts. ## Next steps -- Explore [Configuration](./configuration.html) — consider moving your `-t` - target flags into a `doc-kit.config.mjs` file. -- [Further customize the `html` generator](./generators/html.html) — give it a - custom sidenav or footer. -- [Read the full input specification](./specification.html) — the full Markdown - contract. +- [Writing documentation](./writing-docs.html) — the Markdown conventions that + make `doc-kit` more than a static-site generator. +- [Customizing the site](./customization.html) — your name, logo, navigation, + and components instead of the defaults. +- [Publishing your docs](./publishing.html) — production builds, base URLs, and + hosting. +- [Troubleshooting](./troubleshooting.html) — if something didn't work. diff --git a/www/pages/index.md b/www/pages/index.md index 7ee686d5..84a10fd5 100644 --- a/www/pages/index.md +++ b/www/pages/index.md @@ -1,71 +1,30 @@ # `doc-kit` -`doc-kit` is an opinionated Markdown parsing tool to structurally transform API -documentation. It's the documentation toolchain behind the Node.js API -reference, found at https://nodejs.org/docs/latest/api/. This site is built by -`doc-kit`, from its own repository. The pages you are reading were produced by -the `html` generator. +`doc-kit` turns API-shaped Markdown into documentation. -> 📣 `doc-kit` is in beta. We'd like feedback within the -> [issue log](https://github.com/nodejs/doc-kit/issues) or by visiting the -> [#nodejs-website Slack Channel](https://openjs-foundation.slack.com/archives/CVAMEJ4UV) +> 📣 `doc-kit` is in beta. We'd love feedback in the +> [issue tracker](https://github.com/nodejs/doc-kit/issues) or the +> [#nodejs-website channel](https://openjs-foundation.slack.com/archives/CVAMEJ4UV) > on [the OpenJS Slack](http://slack.openjsf.org/). -## `doc-kit` is a pipeline, not a Markdown converter +## Quick Start -`doc-kit` parses Markdown source files once, emitting output according to -configured generators. You run any subset of them in one command, customize -their logic, or even build your own generator. - -``` -Raw Markdown Files - │ - [ast] parse to MDAST - │ - [metadata] extract structured API metadata - │ - ├─► [jsx-ast] ─► [html] server-rendered site - ├─► [legacy-html] ─► …-all classic HTML - ├─► [legacy-json] ─► …-all JSON - ├─► [json-simple] simplified JSON - ├─► [llms-txt] llms.txt - ├─► [man-page] man pages - ├─► [orama-db] search index - └─► [sitemap] sitemap.xml +```bash +npx doc-kit bootstrap +npx doc-kit serve ``` -Only some of these are things you ask for by name. `ast`, `metadata`, and -`jsx-ast` are internal stages — they run because something downstream depends on -them, and they are not valid `-t` targets. Everything in the fan-out below -`metadata` is a target you can pass to `-t`, and passing several at once reuses -the one shared parse rather than repeating it. The full list is in the -[generators reference](./generators/html.html). - -## The input contract - -Because `metadata` is looking for an API document, the shape of your Markdown -matters more than it would in a typical static-site generator. The most -important rule: - -> **Every page must begin with a level-one heading.** The first `#` becomes the -> page's identity — its sidebar label and its output filename. A file without -> one produces no page at all, and the build still exits `0`. - -See [the specification](./specification.html) for the full input format. - -## What's next - -- [Getting started](./getting-started.html) — render your first document. -- [Commands](./commands.html) — the `doc-kit` CLI surface. -- [Configuration](./configuration.html) — `doc-kit.config.mjs` reference. -- [Creating generators](./generators.html) — extend the pipeline. +`bootstrap` sets your project up; `serve` renders the site at `localhost:3000` +and rebuilds as you edit. [Getting started](./getting-started.html) walks you +through it. ## Showcase -A couple places `doc-kit` is already in use. Feel free to PR yours. +A few places `doc-kit` is already in use. Feel free to PR yours. -- - `legacy-json` -- - `html` -- - `html` -- - `html` -- - `html` +- - `legacy-html`, `legacy-json` +- - `llms-txt` +- - `html`, `orama-db`, `llms-txt` +- - `html`, `orama-db`, `llms-txt` +- - `html`, `orama-db`, `llms-txt` +- - `html`, `orama-db`, `llms-txt`