diff --git a/README.md b/README.md index 082b833..072a690 100644 --- a/README.md +++ b/README.md @@ -1,132 +1,18 @@ -# md-docrs-proxy +# md-docrs -`md-docrs-proxy` resolves docs.rs rustdoc JSON URLs and renders rustdoc JSON as Markdown. +`md-docrs` is a CLI that renders docs.rs rustdoc JSON as Markdown. -## Workspace - -Rust crates under `crates/`: - -- `md-docrs-core` — shared spec parsing, docs.rs resolution, rustdoc JSON rendering, cache traits -- `md-docrs-fetch-http` — native HTTP fetcher for docs.rs -- `md-docrs-cli` — native CLI that prints Markdown to stdout -- `md-docrs-server` — native HTTP server -- `md-docrs-worker` — Cloudflare Worker crate -- `md-docrs-rust-wasm` — Rust `wasm32-unknown-unknown` export layer -- `md-docrs-wasm-compare` — host-side WASM comparison harness - -Other top-level directories: - -- `zig/` — Zig implementation of the minimal `resolve_url` ABI, plus its Worker wrapper -- `wasm/` — staged WASM artifacts and the repo-level build script - -## What each path owns - -- `crates/` owns the Rust implementation -- `zig/` owns the minimal Zig implementation -- `wasm/` owns artifact staging for Rust/Zig WASM comparison - -The top-level `wasm/` directory is not a Cargo crate. - -## Build and test - -Build the Rust workspace: - -```sh -cargo build --workspace -``` - -Run the Rust tests: - -```sh -cargo test --workspace -``` - -Run the Zig tests from the repo root: - -```sh -zig build test --build-file zig/lib/build.zig -``` - -## Release packaging - -This repository is configured for [`cargo-dist`](https://axodotdev.github.io/cargo-dist/) -releases of the `md-docrs` CLI. With `cargo-dist` 0.31, the generated release -configuration lives in [dist-workspace.toml](/Users/thomas/dev/perso/github/md_docrs_proxy/dist-workspace.toml) -and the CI workflow lives in [.github/workflows/release.yml](/Users/thomas/dev/perso/github/md_docrs_proxy/.github/workflows/release.yml). - -Install the release tooling locally: - -```sh -cargo install cargo-dist --locked -cargo install cargo-release --locked -``` - -Validate what the release workflow will build: - -```sh -dist plan --tag vX.Y.Z -``` - -Build the current platform's release artifacts and installers locally: - -```sh -dist build --tag vX.Y.Z -``` - -Release tags should use the unified workspace form (`v0.1.0`, `v0.2.3`, ...). -This workspace distributes only the `md-docrs-cli` package with `cargo-dist`, -which ships the `md-docrs` binary. - -Homebrew releases are published through the `ThomAub/homebrew-tap` tap managed -by `cargo-dist`. There is intentionally no checked-in `HomebrewFormula/` -directory in this repository. - -This repository configures `cargo-release` to: - -- update the shared Rust workspace version in one place -- create a single unified release tag as `v{{version}}` -- publish the crates needed by the CLI release -- keep the final remote push explicit - -For this virtual workspace, preview and execute releases from the repository root: - -```sh -# preview -cargo release 0.2.0 - -# version bump + release commit + tag + crates.io publish, but keep the push explicit -cargo release 0.2.0 --execute --no-push - -# push the release commit and tag after validation -git push origin main -git push origin v0.2.0 -``` - -Repository prerequisites: - -- Create the tap repository `ThomAub/homebrew-tap`. -- Add a GitHub personal access token with `repo` scope as the - `HOMEBREW_TAP_TOKEN` secret in this repository. - -Once a release has been published, install with Homebrew: - -```sh -brew install ThomAub/tap/md-docrs -``` - -## Native CLI - -The CLI binary comes from `md-docrs-cli`. - -Spec grammar: +It lets you query a crate, module, type, trait, function, or other documented item with a compact spec: ```text crate[@version][::path::to::item] ``` -Examples: +## Quick start + +Run the CLI directly from the workspace: -```sh +```bash cargo run -p md-docrs-cli -- anyhow cargo run -p md-docrs-cli -- anyhow::Error cargo run -p md-docrs-cli -- tokio::sync::Mutex @@ -134,91 +20,32 @@ cargo run -p md-docrs-cli -- tokio@1.52.1::sync::Mutex cargo run -p md-docrs-cli -- --target x86_64-unknown-linux-gnu tokio::sync::Mutex ``` -Output is Markdown on stdout. +Output is written to stdout as Markdown. -## Native server +## What this repository contains -The HTTP server binary comes from `md-docrs-server`. +This repository is a Rust workspace centered on the `md-docrs` CLI, plus related server, worker, and WASM targets for some explorative work on wasm and zig. -Start it locally: +### Crates -```sh -cargo run -p md-docrs-server -- --port 8080 --bind 127.0.0.1 -``` - -Example requests: - -```sh -curl -sS http://127.0.0.1:8080/anyhow -curl -sS http://127.0.0.1:8080/anyhow/latest/anyhow/struct.Error.html -curl -sS http://127.0.0.1:8080/tokio/latest/tokio/sync/struct.Mutex.html -curl -sS http://127.0.0.1:8080/healthz -``` - -Response behavior: +- `crates/md-docrs-cli` — the `md-docrs` command-line tool +- `crates/md-docrs-core` — spec parsing, docs.rs resolution, rendering, and cache abstractions +- `crates/md-docrs-fetch-http` — native HTTP fetcher for docs.rs rustdoc JSON +- `crates/md-docrs-server` — native HTTP server that serves Markdown +- `crates/md-docrs-worker` — Cloudflare Worker for a hosted HTTP version +- `crates/md-docrs-rust-wasm` — Rust WASM export layer +- `crates/md-docrs-wasm-compare` — host-side comparison harness for WASM builds -- `200` with `Content-Type: text/markdown; charset=utf-8` -- `400` for invalid specs -- `404` for missing items -- `502` for upstream, decode, or JSON errors +### Other directories -Optional disk-backed cache support is available behind the `hybrid-cache` feature on `md-docrs-server`. +- `zig/` — minimal Zig URL-resolution implementation for fun and compare WASM size +- `wasm/` — WASM artifacts and comparison build flow -## Rust WASM - -The Rust WASM crate lives at `crates/md-docrs-rust-wasm`. - -Minimal build, ABI-compatible with the Zig module: - -```sh -cargo build --profile wasm-release --target wasm32-unknown-unknown \ - -p md-docrs-rust-wasm --no-default-features -``` +## Hosted MCP -Default build adds `render_markdown`: +There is also a Cloudflare Worker-based hosted path in `crates/md-docrs-worker`. +Ongoing work! -```sh -cargo build --profile wasm-release --target wasm32-unknown-unknown \ - -p md-docrs-rust-wasm -``` - -Full build adds `render_markdown` and `render_spec`: - -```sh -cargo build --profile wasm-release --target wasm32-unknown-unknown \ - -p md-docrs-rust-wasm --no-default-features --features full -``` - -## Zig - -The Zig subtree implements the minimal `resolve_url` path. - -Common commands: - -```sh -zig build --build-file zig/lib/build.zig -zig build cli --build-file zig/lib/build.zig -zig build test --build-file zig/lib/build.zig -``` - -See `zig/README.md` for details. - -## WASM comparison - -Stage artifacts, then run the comparison harness: - -```sh -./wasm/build.sh -cargo run -p md-docrs-wasm-compare -- --offline -``` - -See `wasm/README.md` for the workflow and supported flags. - -## Notes - -Current limits: +## Release packaging -- in-memory cache by default for native paths -- no disk cache unless `md-docrs-server` is built with `hybrid-cache` -- partial rendering coverage; not all rustdoc surfaces are rendered yet -- Zig currently covers URL resolution only, not fetch/decompress/render +This repository is configured thanks to [`cargo-dist`](https://axodotdev.github.io/cargo-dist/) and [`cargo-release`](https://github.com/crate-ci/cargo-release) diff --git a/crates/md-docrs-cli/README.md b/crates/md-docrs-cli/README.md new file mode 100644 index 0000000..b24a78c --- /dev/null +++ b/crates/md-docrs-cli/README.md @@ -0,0 +1,93 @@ +# md-docrs-cli + +`md-docrs-cli` provides the `md-docrs` command-line interface for rendering docs.rs Rust API documentation as Markdown. + +It resolves a crate or item spec, fetches the corresponding rustdoc JSON from docs.rs, and prints Markdown to stdout. + +## Install + +From the workspace root: + +```bash +cargo install --path crates/md-docrs-cli +``` + +Run without installing: + +```bash +cargo run -p md-docrs-cli -- +``` + +## Spec format + +```text +crate[@version][::path::to::item] +``` + +Examples: + +```bash +md-docrs anyhow +md-docrs anyhow::Error +md-docrs tokio::sync::Mutex +md-docrs tokio@1.52.1::sync::Mutex +md-docrs --target x86_64-unknown-linux-gnu tokio::sync::Mutex +``` + +## Usage + +```text +md-docrs [--target ] +``` + +Arguments: + +- `` — crate or item to render +- `--target ` — optional target triple to resolve a target-specific rustdoc JSON build + +## Examples + +Render a crate landing page as Markdown: + +```bash +cargo run -p md-docrs-cli -- anyhow +``` + +Render a specific item: + +```bash +cargo run -p md-docrs-cli -- anyhow::Error +``` + +Render a version-pinned item: + +```bash +cargo run -p md-docrs-cli -- tokio@1.52.1::sync::Mutex +``` + +Render using a specific target triple: + +```bash +cargo run -p md-docrs-cli -- --target x86_64-unknown-linux-gnu tokio::sync::Mutex +``` + +Pipe the Markdown into a file: + +```bash +cargo run -p md-docrs-cli -- serde::Serialize > serde-serialize.md +``` + +## Output + +The command writes Markdown to standard output. + +On invalid input or upstream failures, it exits with an error and prints a message to standard error. + +## Related crates + +- `md-docrs-core` — spec parsing, resolution, rendering, and cache abstractions +- `md-docrs-fetch-http` — native HTTP fetcher used by the CLI + +## Workspace + +This crate lives in the `md-docrs-proxy` workspace under `crates/md-docrs-cli`. \ No newline at end of file diff --git a/crates/md-docrs-core/README.md b/crates/md-docrs-core/README.md new file mode 100644 index 0000000..49cecf3 --- /dev/null +++ b/crates/md-docrs-core/README.md @@ -0,0 +1,58 @@ +# md-docrs-core + +Core library for `md-docrs`. + +This crate owns the shared pipeline used by the CLI, server, Worker, and WASM builds: + +- parse `crate[@version][::path::to::item]` +- resolve the matching docs.rs rustdoc JSON URL +- validate and load rustdoc JSON +- resolve an item inside the rustdoc crate +- render the result as Markdown +- provide cache traits and in-memory cache support + +## What this crate contains + +Main areas of responsibility: + +- `spec` — parse and validate item specs +- `resolve` — find items inside a loaded rustdoc JSON crate +- `render` — turn resolved rustdoc items into Markdown +- `fetch` — shared docs.rs URL building and format validation helpers +- `cache` — cache abstractions plus the default in-memory cache + +This crate is intentionally runtime-agnostic. It does not perform platform-specific HTTP itself. + +## Public API + +Typical entry points: + +- `ItemSpec` — parsed spec representation +- `render_spec` — high-level fetch + resolve + render flow +- `load_crate` — cache-aware crate loading +- `render_loaded_crate` — render from an already loaded rustdoc crate +- `RustdocFetcher` — fetcher trait implemented by host-specific crates + +## Feature flags + +- `hybrid-cache` — enables the optional foyer-backed hybrid cache support + +Default features are empty. + +## Used by + +- `md-docrs-cli` +- `md-docrs-server` +- `md-docrs-worker` +- `md-docrs-rust-wasm` + +## Notes + +This crate focuses on shared logic only: + +- no native CLI surface +- no HTTP server +- no Worker bindings +- no docs.rs HTTP client implementation + +Those integrations live in sibling crates in the workspace. \ No newline at end of file diff --git a/crates/md-docrs-fetch-http/README.md b/crates/md-docrs-fetch-http/README.md new file mode 100644 index 0000000..f89373d --- /dev/null +++ b/crates/md-docrs-fetch-http/README.md @@ -0,0 +1,88 @@ +# md-docrs-fetch-http + +HTTP fetcher for `md-docrs`. + +This crate is the native transport layer used to download rustdoc JSON from docs.rs, decompress it, deserialize it, and hand a validated `rustdoc_types::Crate` back to the rest of the system. + +## Purpose + +`md-docrs-fetch-http` owns the network-facing part of the pipeline for native Rust targets: + +- build the docs.rs rustdoc JSON URL +- fetch the `.zst` payload over HTTPS +- decompress the response +- deserialize rustdoc JSON +- validate the rustdoc format version + +It is used by: + +- `md-docrs-cli` +- `md-docrs-server` + +It is not used by the Cloudflare Worker path, which has its own worker-specific fetch implementation. + +## Scope + +This crate does: + +- HTTP fetching from docs.rs +- zstd decompression +- rustdoc JSON decoding +- format-version validation + +This crate does not do: + +- spec parsing +- item resolution +- Markdown rendering +- caching policy +- HTTP server routing +- CLI argument parsing + +Those responsibilities live in other crates, primarily `md-docrs-core`, `md-docrs-cli`, and `md-docrs-server`. + +## Dependency relationship + +`md-docrs-fetch-http` depends on `md-docrs-core` for shared types and validation helpers. + +Typical flow: + +1. another crate parses a spec into `ItemSpec` +2. core/cache logic decides whether a fetch is needed +3. this crate downloads and decodes the rustdoc JSON crate +4. `md-docrs-core` resolves the requested item and renders Markdown + +## Native-only role + +This crate is intended for native environments. + +It currently uses `ureq` for HTTP and `zstd` for decompression, which makes it a good fit for the CLI and server binaries but not for the Worker/WASM deployment targets. + +## Build + +From the repository root: + +```bash +cargo build -p md-docrs-fetch-http +``` + +## Test + +From the repository root: + +```bash +cargo test -p md-docrs-fetch-http +``` + +## Related crates + +- `md-docrs-core` — shared spec parsing, URL building helpers, resolution, rendering, cache traits +- `md-docrs-cli` — native CLI that prints Markdown to stdout +- `md-docrs-server` — native HTTP server +- `md-docrs-worker` — Cloudflare Worker implementation with its own fetch path + +## Notes + +This crate is intentionally small and focused. + +If you need to change how docs.rs payloads are fetched on native targets, this is the crate to update. \ No newline at end of file diff --git a/crates/md-docrs-rust-wasm/README.md b/crates/md-docrs-rust-wasm/README.md index 61ce638..a144316 100644 --- a/crates/md-docrs-rust-wasm/README.md +++ b/crates/md-docrs-rust-wasm/README.md @@ -1,201 +1,95 @@ # md-docrs-rust-wasm -Rust `wasm32-unknown-unknown` export layer for this workspace. +Rust `wasm32-unknown-unknown` export layer for `md-docrs`. -This crate wraps the shared Rust logic from `md-docrs-core` behind a small C-style ABI for host environments. +This crate packages the core `md-docrs` logic as a WebAssembly module so host environments can call a small, stable ABI. It is primarily used for WASM artifact comparison and for browser/edge-style hosts that want Rust-based docs.rs resolution and Markdown rendering. -## What it exports +## What this crate owns -### Always available +- Rust WASM exports for the `md-docrs` pipeline +- ABI-compatible exports for comparison with the Zig WASM module +- optional render-only and fetch-enabled WASM builds +- the Rust side of the staged artifacts under `wasm/` -- `alloc(len: u32) -> *mut u8` -- `free(ptr: *mut u8, len: u32)` -- `resolve_url(spec_ptr, spec_len, target_ptr, target_len, out_ptr, out_cap) -> u32` +It does not own: -These three exports match the Zig minimal WASM surface. - -### With `render` - -- `render_markdown(json_ptr, json_len, spec_ptr, spec_len, target_ptr, target_len, len_out) -> *mut u8` - -This lets a host pass rustdoc JSON into the module and receive rendered Markdown back. - -### With `render` + `fetch` - -- `render_spec(spec_ptr, spec_len, target_ptr, target_len, buf_ptr_out, buf_len_out) -> i32` - -This is the full in-module pipeline: - -1. parse the spec -2. build the docs.rs rustdoc JSON URL -3. call the host-provided `fetch_bytes` -4. zstd-decode the response -5. parse rustdoc JSON -6. resolve the requested item -7. render Markdown +- the native CLI (`crates/md-docrs-cli`) +- the native HTTP server (`crates/md-docrs-server`) +- the Cloudflare Worker app (`crates/md-docrs-worker`) +- artifact staging orchestration (`wasm/build.sh`) +- the Zig comparison implementation (`zig/`) ## Features -| Feature | Default | Purpose | -| --- | --- | --- | -| `render` | yes | Enables JSON-to-Markdown rendering and exports `render_markdown` | -| `fetch` | no | Enables host-imported fetch + in-WASM zstd decode used by `render_spec` | -| `full` | no | Convenience alias for `render` + `fetch` | +This crate has three feature modes: -## Build modes +- default: `render` +- `fetch` +- `full` -### Minimal +Feature behavior: -Exports only: +- `render` + - enables JSON-to-Markdown rendering + - expects the host to provide rustdoc JSON bytes +- `fetch` + - enables in-WASM fetching and zstd decoding support +- `full` + - convenience alias for `render + fetch` -- `alloc` -- `free` -- `resolve_url` +## Build variants -Build: +Minimal ABI-compatible build: -```/dev/null/minimal.sh#L1-2 +```bash cargo build --profile wasm-release --target wasm32-unknown-unknown \ -p md-docrs-rust-wasm --no-default-features ``` -### Default - -Exports: - -- `alloc` -- `free` -- `resolve_url` -- `render_markdown` +Default render build: -Build: - -```/dev/null/default.sh#L1-2 +```bash cargo build --profile wasm-release --target wasm32-unknown-unknown \ -p md-docrs-rust-wasm ``` -### Full - -Exports: - -- `alloc` -- `free` -- `resolve_url` -- `render_markdown` -- `render_spec` +Full build with render + fetch support: -Build: - -```/dev/null/full.sh#L1-2 +```bash cargo build --profile wasm-release --target wasm32-unknown-unknown \ -p md-docrs-rust-wasm --no-default-features --features full ``` -Output path: - -```/dev/null/output.txt#L1-1 -target/wasm32-unknown-unknown/wasm-release/md_docrs_rust_wasm.wasm -``` - -## ABI notes - -### Memory - -- `alloc` returns a pointer in WASM linear memory -- `free` must be called with the exact pointer and length originally allocated -- `alloc(0)` returns null -- most failures are reported as `0`, null, or a negative status code depending on the export - -### `resolve_url` - -`resolve_url` parses: - -```/dev/null/spec.txt#L1-1 -crate[@version][::path::to::item] -``` - -If `target_len == 0`, no explicit target triple is used. +## Relationship to the Zig WASM module -On success it writes the docs.rs rustdoc JSON URL into the caller-provided output buffer and returns the number of bytes written. +This crate is designed to match the minimal ABI surface used by the Zig implementation where possible so the host-side comparison harness can exercise both modules with the same calling convention. -It returns `0` on failure, including: +That comparison flow is managed from the repository root: -- invalid UTF-8 -- invalid spec -- output buffer too small - -### `render_markdown` - -`render_markdown` expects the host to provide: - -- rustdoc JSON bytes -- a spec -- an optional target triple -- a writable `len_out` - -On success it returns a newly allocated Markdown buffer and writes its size to `*len_out`. - -The caller owns the returned buffer and must release it with `free(ptr, len)`. - -It returns null on failure. - -### `render_spec` - -`render_spec` requires a host import: - -```/dev/null/fetch-bytes.txt#L1-5 -fetch_bytes( - url_ptr: *const u8, - url_len: u32, - buf_ptr_out: *mut u32, - buf_len_out: *mut u32, -) -> i32 +```bash +./wasm/build.sh +cargo run -p md-docrs-wasm-compare -- --offline ``` -The host is expected to: +## Typical role in the workspace -1. fetch the URL -2. allocate a buffer inside WASM memory using exported `alloc` -3. write the response body into that buffer -4. store the pointer and length into the provided out-slots +Common flows: -Return `0` for success and non-zero for failure. +- build Rust WASM artifacts for size and behavior comparison +- export URL resolution in a minimal WASM-compatible ABI +- export Markdown rendering for host-driven integrations +- serve as the Rust-side module for staged WASM artifacts in `wasm/artifacts/` -On success, `render_spec` writes an allocated Markdown buffer to `*buf_ptr_out` and `*buf_len_out`, then returns `0`. +## Related crates and directories -### `render_spec` status codes +- `crates/md-docrs-core` — shared spec parsing, resolution, rendering, cache traits +- `crates/md-docrs-wasm-compare` — host-side comparison harness +- `wasm/` — staged artifact workflow +- `zig/` — Zig implementation with a matching minimal comparison target -| Code | Meaning | -| --- | --- | -| `0` | Success | -| `-1` | Allocation failure | -| `-2` | Host fetch failed | -| `-3` | zstd decode failed | -| `-4` | JSON parse failed | -| `-5` | Spec parse failure, resolve miss, or URL too long | -| `-6` | Output pointer or length could not be written | +## Notes -## Relationship to the rest of the repo - -- `crates/md-docrs-core` contains the shared Rust parsing, resolution, and rendering logic -- `crates/md-docrs-wasm-compare` contains the host-side comparison harness -- `zig/` contains the independent Zig implementation of the minimal ABI -- `wasm/` contains the repo-level staging script and staged artifacts - -This crate should stay focused on the Rust WASM ABI layer. - -## Typical workflow - -Build and stage artifacts from the repo root: - -```/dev/null/workflow.sh#L1-2 -./wasm/build.sh -cargo run -p md-docrs-wasm-compare -- --offline -``` - -## Tests - -```/dev/null/tests.sh#L1-1 -cargo test -p md-docrs-rust-wasm -``` +- target is `wasm32-unknown-unknown` +- the workspace defines a dedicated `wasm-release` profile for optimized builds +- this crate is not published independently +- use `wasm/build.sh` when you want the repo-level staged artifact workflow instead of a single direct build \ No newline at end of file diff --git a/crates/md-docrs-server/README.md b/crates/md-docrs-server/README.md new file mode 100644 index 0000000..37b3315 --- /dev/null +++ b/crates/md-docrs-server/README.md @@ -0,0 +1,123 @@ +# md-docrs-server + +`md-docrs-server` is the native HTTP server for `md-docrs`. + +It fetches rustdoc JSON from docs.rs, resolves requested items, renders them as Markdown, and serves the result over HTTP. + +## What it does + +- accepts crate and item requests over HTTP +- fetches rustdoc JSON from docs.rs +- renders Markdown responses +- exposes a simple health endpoint +- supports in-memory caching by default +- supports an optional memory + disk hybrid cache via the `hybrid-cache` feature + +## Run locally + +From the workspace root: + +```bash +cargo run -p md-docrs-server -- --port 8080 --bind 127.0.0.1 +``` + +Then query it with `curl`: + +```bash +curl -sS http://127.0.0.1:8080/anyhow +curl -sS http://127.0.0.1:8080/anyhow/latest/anyhow/struct.Error.html +curl -sS http://127.0.0.1:8080/tokio/latest/tokio/sync/struct.Mutex.html +curl -sS http://127.0.0.1:8080/healthz +``` + +## Routes + +Supported request forms: + +- `GET /` +- `GET //` +- `GET /////struct.Name.html` +- `GET /?spec=` +- `GET /?spec=&target=` +- `GET /healthz` + +The server also accepts `?target=` on path-based requests. + +Examples: + +```bash +curl -sS http://127.0.0.1:8080/serde +curl -sS http://127.0.0.1:8080/tokio/1.52.1/tokio/sync/struct.Mutex.html +curl -sS 'http://127.0.0.1:8080/?spec=anyhow::Error' +curl -sS 'http://127.0.0.1:8080/tokio/latest/tokio/sync/struct.Mutex.html?target=x86_64-unknown-linux-gnu' +``` + +## CLI options + +```bash +md-docrs-server --port --bind [--cache-dir ] + [--cache-disk-bytes ] + [--cache-memory-bytes ] +``` + +Options: + +- `--port` — listen port, defaults to `8080` +- `--bind` — bind address, defaults to `127.0.0.1` +- `--cache-dir` — enable the foyer-backed hybrid cache and store the disk tier in this directory +- `--cache-disk-bytes` — disk tier capacity in bytes +- `--cache-memory-bytes` — memory tier weight budget in bytes + +## Response behavior + +- `200` with `Content-Type: text/markdown; charset=utf-8` for successful item renders +- `200` with plain text for `/healthz` +- `400` for invalid specs +- `404` for missing items +- `502` for upstream fetch, decode, format, or JSON errors + +Successful Markdown responses also include: + +- `Vary: Accept` +- `X-Markdown-Tokens` — a rough token estimate derived from output size + +## Caching + +By default, the server uses an in-memory cache. + +With the default crate features, you can enable a foyer-backed hybrid cache by passing `--cache-dir`: + +```bash +cargo run -p md-docrs-server -- --cache-dir .cache/md-docrs +``` + +If the binary is built without the `hybrid-cache` feature, `--cache-dir` is rejected. + +Build without that feature: + +```bash +cargo build -p md-docrs-server --no-default-features +``` + +Build explicitly with the feature: + +```bash +cargo build -p md-docrs-server --features hybrid-cache +``` + +## Relationship to the workspace + +This crate is the native HTTP-serving entry point. + +Related crates: + +- `md-docrs-core` — spec parsing, resolution, rendering, cache traits +- `md-docrs-fetch-http` — native docs.rs fetcher +- `md-docrs-cli` — native CLI that prints Markdown to stdout +- `md-docrs-worker` — Cloudflare Worker version + +## Notes + +- this crate is not published +- it is intended for native server deployments +- item path parsing strips rustdoc kind prefixes like `struct.` and the `.html` suffix from the final path segment \ No newline at end of file diff --git a/crates/md-docrs-wasm-compare/README.md b/crates/md-docrs-wasm-compare/README.md new file mode 100644 index 0000000..cec8dc6 --- /dev/null +++ b/crates/md-docrs-wasm-compare/README.md @@ -0,0 +1,77 @@ +# md-docrs-wasm-compare + +Host-side comparison harness for the staged Rust and Zig WASM artifacts. + +This crate is used to compare behavior and artifact characteristics across the WASM modules built elsewhere in the workspace. It does not build the `.wasm` files itself; it consumes artifacts staged under the repo-level `wasm/` directory. + +## What it does + +- loads staged WASM artifacts from `wasm/artifacts/` +- runs the same host-side checks against each available artifact +- compares outputs across implementations +- supports offline runs against locally staged artifacts + +This crate is intended for verification and comparison work, not for serving production traffic. + +## Scope + +This crate owns: + +- the native comparison binary +- host-side loading and execution of staged WASM modules +- result comparison across Rust and Zig artifacts + +This crate does not own: + +- WASM artifact building +- artifact staging scripts +- the Rust WASM implementation itself +- the Zig implementation itself + +## Binary + +The package exposes the `wasm-compare` binary. + +Run it from the repository root after staging artifacts: + +```bash +./wasm/build.sh +cargo run -p md-docrs-wasm-compare -- --offline +``` + +## Inputs + +The harness looks for staged files in `wasm/artifacts/`. + +Expected filenames include: + +- `zig-minimal.wasm` +- `zig-full.wasm` +- `rust-minimal.wasm` +- `rust-minimal-opt.wasm` +- `rust-full.wasm` +- `rust-full-opt.wasm` + +Missing artifacts are skipped. + +## Typical workflow + +From the repository root: + +```bash +./wasm/build.sh +cargo run -p md-docrs-wasm-compare -- --offline +``` + +## Related paths + +- `wasm/README.md` — repo-level staging workflow +- `wasm/build.sh` — artifact build and staging script +- `crates/md-docrs-rust-wasm` — Rust WASM module under test +- `zig/README.md` — Zig implementation and Worker wrapper + +## Notes + +- this crate is host-side only +- it is not published +- it is mainly useful when validating parity and size/runtime tradeoffs across WASM variants \ No newline at end of file diff --git a/crates/md-docrs-worker/README.md b/crates/md-docrs-worker/README.md new file mode 100644 index 0000000..648205c --- /dev/null +++ b/crates/md-docrs-worker/README.md @@ -0,0 +1,116 @@ +# md-docrs-worker + +`md-docrs-worker` is the hosted Markdown rendering entrypoint for the `md-docrs` workspace. + +It runs as a Cloudflare Worker and serves Markdown generated from docs.rs rustdoc JSON. It is the hosted counterpart to the native `md-docrs` CLI and `md-docrs-server`. + +## What it does + +This crate: + +- accepts crate specs and docs.rs-style paths over HTTP +- fetches rustdoc JSON from docs.rs +- decompresses and parses the payload inside the Worker +- renders the requested crate or item as Markdown +- caches decoded rustdoc crates in Cloudflare KV + +This makes it suitable for a hosted MCP-friendly deployment where the Worker provides a simple HTTP interface and Cloudflare handles the edge runtime. + +## Routes + +The Worker supports these request forms: + +- `GET /?spec=` +- `GET /?spec=&target=` +- `GET /` +- `GET //` +- `GET /////.../struct.Name.html` +- `GET /healthz` +- `GET /kv` + +The `target` query parameter is optional and overrides the docs.rs target triple. + +## Examples + +Render a crate root: + +```/dev/null/worker-readme-curl-1.sh#L1-1 +curl 'http://127.0.0.1:8787/?spec=anyhow' +``` + +Render a specific item from a query spec: + +```/dev/null/worker-readme-curl-2.sh#L1-1 +curl 'http://127.0.0.1:8787/?spec=tokio::sync::Mutex' +``` + +Render using a docs.rs-style path: + +```/dev/null/worker-readme-curl-3.sh#L1-1 +curl 'http://127.0.0.1:8787/tokio/latest/tokio/sync/struct.Mutex.html' +``` + +Render for a specific target: + +```/dev/null/worker-readme-curl-4.sh#L1-1 +curl 'http://127.0.0.1:8787/?spec=anyhow::Error&target=x86_64-unknown-linux-gnu' +``` + +Health check: + +```/dev/null/worker-readme-curl-5.sh#L1-1 +curl 'http://127.0.0.1:8787/healthz' +``` + +## Responses + +Successful render responses return: + +- status `200` +- `Content-Type: text/markdown; charset=utf-8` + +Error behavior: + +- `400` for invalid specs +- `404` for missing items +- `502` for upstream fetch, decode, format, or JSON errors + +The root route `/` returns a short plain-text usage message. + +## Cache + +The Worker uses Cloudflare KV for crate-level caching. + +Cache keys are derived from: + +- crate name +- version +- optional target triple + +Cached values contain the decoded rustdoc crate payload, which avoids repeating fetch and decode work for hot entries. + +## Local development + +This crate is intended to run in the Cloudflare Worker environment. + +Typical local flow from the repository root: + +```/dev/null/worker-readme-local.sh#L1-2 +cargo test --workspace +cargo build --workspace +``` + +Then run the Worker with your usual Cloudflare local workflow for this repository. + +## Relationship to other crates + +- `md-docrs-core` provides spec parsing, URL resolution, rendering, and shared error types +- `md-docrs-worker` provides the hosted HTTP interface and KV-backed cache +- `md-docrs-cli` is the native CLI for local use +- `md-docrs-server` is the native server alternative + +## Notes + +- this crate is not published independently +- it is designed for Cloudflare Worker deployment +- it is the main hosted path to mention when describing a remotely deployed MCP-compatible setup \ No newline at end of file diff --git a/wasm/README.md b/wasm/README.md index 1f222f9..1606bcb 100644 --- a/wasm/README.md +++ b/wasm/README.md @@ -20,7 +20,7 @@ The comparison binary lives in `crates/md-docrs-wasm-compare`. From the repo root: -```/dev/null/wasm-compare.sh#L1-2 +```bash ./wasm/build.sh cargo run -p md-docrs-wasm-compare -- --offline ``` diff --git a/zig/README.md b/zig/README.md index 7a1149e..fdb7e9d 100644 --- a/zig/README.md +++ b/zig/README.md @@ -49,14 +49,14 @@ It does not own: From `zig/`: -```/dev/null/zig-build-npm.sh#L1-2 +```bash npm install npm run build:wasm ``` From `zig/lib/`: -```/dev/null/zig-build-lib.sh#L1-3 +```bash zig build zig build cli zig build test @@ -64,7 +64,7 @@ zig build test From the repo root: -```/dev/null/zig-build-root.sh#L1-1 +```bash zig build test --build-file zig/lib/build.zig ``` @@ -72,14 +72,14 @@ zig build test --build-file zig/lib/build.zig Build: -```/dev/null/zig-cli-build.sh#L1-2 +```bash cd zig/lib zig build cli ``` Run: -```/dev/null/zig-cli-run.sh#L1-4 +```bash ./zig-out/bin/md-docrs-zig serde ./zig-out/bin/md-docrs-zig 'tokio@1.52.1::sync::Mutex' ./zig-out/bin/md-docrs-zig 'anyhow::Error' --target x86_64-unknown-linux-gnu @@ -88,13 +88,13 @@ zig build run -- 'tokio@1.52.1::sync::Mutex' --target x86_64-unknown-linux-gnu Usage: -```/dev/null/zig-cli-usage.txt#L1-1 -md-docrs-zig [--target TRIPLE] +```text +md-docrs-zig <SPEC> [--target TRIPLE] ``` Spec grammar: -```/dev/null/spec.txt#L1-1 +```text crate[@version][::path::to::item] ``` @@ -106,7 +106,7 @@ Behavior: Examples of output: -```/dev/null/zig-cli-output.txt#L1-3 +```text https://docs.rs/crate/serde/latest/json/57.zst https://docs.rs/crate/tokio/1.52.1/json/57.zst https://docs.rs/crate/anyhow/latest/x86_64-unknown-linux-gnu/json/57.zst @@ -118,7 +118,7 @@ The Worker is a thin host around the Zig WASM module. Setup and run: -```/dev/null/zig-worker-dev.sh#L1-4 +```bash cd zig npm install npm run build:wasm @@ -127,7 +127,7 @@ npm run dev Deploy: -```/dev/null/zig-worker-deploy.sh#L1-1 +```bash npm run deploy ``` @@ -142,7 +142,7 @@ GET /?spec=&target= Examples: -```/dev/null/zig-worker-curl.sh#L1-4 +```bash curl localhost:8787/serde curl localhost:8787/tokio@1.52.1::sync::Mutex curl 'localhost:8787/tokio::sync::Mutex?target=x86_64-unknown-linux-gnu' @@ -186,7 +186,7 @@ That lets the comparison harness swap Rust and Zig artifacts with the same host- Use the repo-level comparison flow from the repository root: -```/dev/null/wasm-compare.sh#L1-2 +```bash ./wasm/build.sh cargo run -p md-docrs-wasm-compare -- --offline ```