- Enhance documentation with titles and descriptions for various pages using YAML frontmatter.

- Introduce a custom build system for generating llms.txt and llms-full.txt from documentation files.
- Add openapi.json to .gitignore to prevent it from being tracked.
- Remove outdated llms.txt file as it is now generated dynamically.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Author: peterschmidt85

.gitignore

@@ -26,3 +26,4 @@ uv.lock
 /src/dstack/_internal/server/statics
 
 profiling_results.html
+docs/docs/reference/api/rest/openapi.json

contributing/DOCS.md

@@ -47,3 +47,106 @@ If you want to build static files, you can use the following command:
 ```shell
 uv run mkdocs build -s
 ```
+
+## Documentation build system
+
+The documentation uses a custom build system with MkDocs hooks to generate various files dynamically.
+
+### Build hooks
+
+The build process is customized via hooks in `scripts/docs/hooks.py`:
+
+#### 1. Example materialization
+
+Example pages like `examples/single-node-training/trl/index.md` are stubs that reference `README.md` files in the repository root:
+- **Stub location**: `docs/examples/single-node-training/trl/index.md`
+- **Content source**: `examples/single-node-training/trl/README.md`
+
+During the build, the hook reads the README content and uses it for rendering the HTML page.
+
+#### 2. Schema reference expansion
+
+Files in `docs/reference/**/*.md` can use `#SCHEMA#` placeholders that are expanded with generated schema documentation during the build.
+
+#### 3. llms.txt generation
+
+Two files are generated for LLM consumption:
+
+- **llms.txt**: Structured overview of documentation with titles and descriptions
+  - Generated from mkdocs nav structure
+  - Includes sections: Getting started, Concepts, Guides, Examples
+  - Excludes: Reference section
+  - Configuration: `scripts/docs/gen_llms_files.py` (INCLUDE_SECTIONS, EXCLUDE_SECTIONS)
+
+- **llms-full.txt**: Full concatenation of all pages from llms.txt
+  - Contains complete markdown content of all included pages
+
+The generation logic is in `scripts/docs/gen_llms_files.py` and uses:
+- `site_name`, `site_description`, `site_url` from `mkdocs.yml`
+- Page titles from mkdocs nav structure
+- Page descriptions from markdown frontmatter
+
+**Adding descriptions**: To add descriptions to pages, add YAML frontmatter:
+
+```yaml
+---
+title: Page Title
+description: Short description of what this page covers
+---
+```
+
+For examples, add frontmatter to the `README.md` files in the repository root (e.g., `examples/single-node-training/trl/README.md`).
+
+#### 4. Skills discovery
+
+The build creates `.well-known/skills/` directory structure for skills discovery:
+- Reads `skills/dstack/SKILL.md`
+- Parses name and description from frontmatter
+- Generates `.well-known/skills/index.json`
+- Copies SKILL.md to both `.well-known/skills/dstack/` and site root
+
+### File structure
+
+```
+docs/
+├── docs/                    # Main documentation content
+│   ├── index.md            # Getting started
+│   ├── installation.md
+│   ├── quickstart.md
+│   ├── concepts/           # Concept pages
+│   ├── guides/             # How-to guides
+│   └── reference/          # API reference (schema expansion)
+├── examples/               # Example stub files (index.md)
+│   └── single-node-training/
+│       └── trl/
+│           └── index.md    # Stub referencing root README
+└── overrides/              # Theme customization
+
+examples/                    # Example content (repository root)
+└── single-node-training/
+    └── trl/
+        ├── README.md       # Actual content with frontmatter
+        └── train.dstack.yml
+
+scripts/docs/
+├── hooks.py                # MkDocs build hooks
+├── gen_llms_files.py       # llms.txt generation
+├── gen_schema_reference.py # Schema expansion
+└── gen_cli_reference.py    # CLI reference generation
+
+skills/
+└── dstack/
+    └── SKILL.md            # Skills discovery content
+```
+
+### Testing changes
+
+When modifying the build system:
+
+1. Test local build: `uv run mkdocs build -s`
+2. Check generated files in `site/`:
+   - `site/llms.txt`
+   - `site/llms-full.txt`
+   - `site/.well-known/skills/index.json`
+3. Verify example pages render correctly
+4. Check that descriptions appear in llms.txt

docs/docs/concepts/backends.md

@@ -1,3 +1,8 @@
+---
+title: Backends
+description: Configuring cloud providers and Kubernetes clusters
+---
+
 # Backends
 
 Backends allow `dstack` to provision fleets across GPU clouds or Kubernetes clusters.

docs/docs/concepts/dev-environments.md

@@ -1,3 +1,8 @@
+---
+title: Dev environments
+description: Provisioning remote instances for cloud-based development
+---
+
 # Dev environments
 
 A dev environment lets you provision an instance and access it with your desktop IDE.

docs/docs/concepts/events.md

@@ -1,3 +1,8 @@
+---
+title: Events
+description: Auditing resource state changes and operations
+---
+
 # Events
 
 Events provide a chronological record of notable state changes and operations affecting `dstack` resources. They are designed for auditing, debugging, and understanding the lifecycle of runs, jobs, fleets, and other resources.

docs/docs/concepts/fleets.md

@@ -1,3 +1,8 @@
+---
+title: Fleets
+description: Managing pools of compute instances
+---
+
 # Fleets
 
 Before submitting runs, you must create a fleet. Fleets act as both pools of instances and templates for how those instances are provisioned.

docs/docs/concepts/gateways.md

@@ -1,3 +1,8 @@
+---
+title: Gateways
+description: Managing ingress traffic and endpoints for services
+---
+
 # Gateways
 
 Gateways manage ingress traffic for running [services](services.md), handle auto-scaling and rate limits, enable HTTPS, and allow you to configure a custom domain. They also support custom routers, such as the [SGLang Model Gateway](https://docs.sglang.ai/advanced_features/router.html#).

docs/docs/concepts/metrics.md

@@ -1,3 +1,8 @@
+---
+title: Metrics
+description: Tracking and monitoring system metrics
+---
+
 # Metrics
 
 `dstack` automatically tracks essential metrics, which you can access via the CLI and UI.

docs/docs/concepts/projects.md

@@ -1,3 +1,8 @@
+---
+title: Projects
+description: Organizing teams and isolating resources
+---
+
 # Projects
 
 Projects enable the isolation of different teams and their resources. Each project can configure its own backends and

docs/docs/concepts/secrets.md

@@ -1,3 +1,8 @@
+---
+title: Secrets
+description: Managing sensitive values and credentials
+---
+
 # Secrets
 
 Secrets allow centralized management of sensitive values such as API keys and credentials. They are project-scoped, managed by project admins, and can be referenced in run configurations to pass sensitive values to runs in a secure manner.