Blog post: how other link checkers handle recursion

content/2026/lychee-recursion/index.md

@@ -113,9 +113,7 @@ In September 2021 we decided to do a bigger rewrite: a stream-based architecture
 
 [PR #165](https://github.com/lycheeverse/lychee/pull/165) was closed in December 2021. The stream refactor landed and gave us a 35–50% speedup. Nice! Tradeoffs, I guess.
 
-{% info() %}
-
-## Takeaways
+{% info(title="Takeaways") %}
 
 - **Counting outstanding work in an async pipeline is fragile.** An off-by-one in distributed counting means a deadlock or an early exit.
 - **Big refactors and feature branches don't get along.** The stream rewrite made the recursion branch stale before it was ever ready.
@@ -176,9 +174,7 @@ I took the problem to the Tokio Discord, and the advice that came back was: "Sto
 
 Even ignoring the deadlock, there was a second issue. The new `from_chan` method benchmarked roughly 30% slower than the existing `from` method. The extra channel indirection cost something, and it cost it even in the non-recursive case, which is the case basically everyone uses.
 
-{% info() %}
-
-## Takeaways
+{% info(title="Takeaways") %}
 
 - **Channels are the wrong tool for cyclic pipelines.** Their close-on-last-sender-drop semantics are fundamentally at odds with a feedback loop.
 - **`for_each_concurrent` looks perfect and isn't.** It processes a stream concurrently but gives you no way to feed items back in.
@@ -228,9 +224,7 @@ A semaphore solves the concurrency-limiting problem. It does nothing for the *te
 
 There's a subtlety with the permits, too. Swapping `for_each_concurrent` for raw `tokio::spawn` loses the bounded concurrency that channels gave us for free. The semaphore adds it back, but you have to manage permits carefully. If a task acquires a permit, spawns a child, and transfers the permit, the parent can't do more work. If it clones the permit, you can blow past your concurrency limit. Getting the permit lifecycle exactly right is fiddly.
 
-{% info() %}
-
-## Takeaways
+{% info(title="Takeaways") %}
 
 - **Semaphores solve concurrency, not termination.** You still need something to tell you "all the work is done."
 - **`Arc<RwLock<State>>` is a code smell in async Rust.** When you start wrapping everything in locks, you're fighting the ownership model instead of working with it. That can leave a lot of performance on the table since every access is a lock acquisition across all threads.
@@ -314,9 +308,7 @@ After a burst of energy in January 2025, things slowed. Merge conflicts piled up
 
 I didn't want them to apologize. They got further than anyone, on a hard feature, in a complex async codebase, as a volunteer. My own note on the PR a while later was just the sober truth:
 
-{% info() %}
-
-## Takeaways
+{% info(title="Takeaways") %}
 
 - **The atomic counter is a manual counter in a trenchcoat.** It had the same failure modes. 
 - When you're adding `vec![]` and `0` to every `Response::new()` call, that's a leaky abstraction.

static/css/main.css

@@ -527,15 +527,32 @@ details {
 
 .info {
     margin: 40px 0 20px;
+    padding: 35px;
     border-style: dotted;
+    font-size: 0.9em;
 }
 
-.info > *:first-child,
-.info > *:last-child {
+.info > *:first-child {
     margin-top: 0;
+}
+
+.info > *:last-child {
     margin-bottom: 0;
 }
 
+.info-title {
+    margin: 0 0 24px;
+    font-weight: 700;
+    font-size: 1.25em;
+    line-height: 1.2;
+}
+
+.info ul,
+.info ol {
+    padding-left: 1.3em;
+    margin: 0;
+}
+
 details summary {
     width: 100%;
     margin: -20px;
@@ -567,6 +584,52 @@ th {
     font-weight: 600;
 }
 
+/* Wide tables: break out of the narrow reading column and scroll
+   horizontally on small screens. Apply via the `wide_table` shortcode. */
+.wide-table {
+    width: 94vw;
+    max-width: 1200px;
+    position: relative;
+    left: 50%;
+    transform: translateX(-50%);
+    overflow-x: auto;
+    margin: 24px 0;
+}
+
+.wide-table table {
+    width: auto;
+    min-width: 100%;
+    max-width: none;
+    margin: 0;
+    font-size: 0.82em;
+}
+
+/* Keep cells readable: wrap at spaces only, never split words or inline
+   code mid-token (`main article` sets `overflow-wrap: anywhere`). The
+   table widens to fit its content and scrolls horizontally if needed. */
+.wide-table th,
+.wide-table td {
+    overflow-wrap: normal;
+    word-break: normal;
+    hyphens: none;
+}
+
+.wide-table code {
+    white-space: nowrap;
+}
+
+/* On small screens the reading column is already near full width, so the
+   breakout is unnecessary and its transform can push the page wider than
+   the viewport. Keep the table inside the column and scroll it instead. */
+@media (max-width: 920px) {
+    .wide-table {
+        width: 100%;
+        max-width: 100%;
+        left: auto;
+        transform: none;
+    }
+}
+
 /* Homepage styles */
 .homepage h1 {
     font-size: 1.5em;

static/js/mermaid-init.js

@@ -0,0 +1,195 @@
+// Renders Mermaid diagrams and keeps them in sync with the site's
+// light/dark theme toggle (driven by the `data-theme` attribute on <html>).
+//
+// Mermaid replaces the original <pre class="mermaid"> source with an SVG once
+// it has rendered, so we stash the source in `data-source` to be able to
+// re-render with a different theme when the user flips the toggle.
+(function () {
+  let rendering = false;
+  let rerenderQueued = false;
+  const SVG_NS = "http://www.w3.org/2000/svg";
+
+  // Single ink color for the whole diagram, following the light/dark theme.
+  function ink() {
+    return document.documentElement.getAttribute("data-theme") === "dark"
+      ? "#e6e6e6"
+      : "#1a1a1a";
+  }
+
+  function createSvgEl(name, attrs) {
+    const el = document.createElementNS(SVG_NS, name);
+    for (const key in attrs) el.setAttribute(key, attrs[key]);
+    return el;
+  }
+
+  // Hand-drawn "squiggle": a turbulence-driven displacement filter applied to
+  // the box and line strokes (not the text labels). The filter is injected
+  // into each diagram's *own* <svg> with a unique id and referenced via a
+  // presentation attribute, because Safari refuses to resolve a CSS
+  // `filter: url(#id)` that points at a filter defined in a different (or
+  // zero-sized) <svg> element — the shapes would simply vanish.
+  function applySquiggle() {
+    document.querySelectorAll(".mermaid svg").forEach((svg, index) => {
+      const id = "squiggle-" + index;
+
+      let defs = svg.querySelector("defs");
+      if (!defs) {
+        defs = createSvgEl("defs", {});
+        svg.insertBefore(defs, svg.firstChild);
+      }
+
+      if (!svg.querySelector("#" + id)) {
+        // `objectBoundingBox` units (the default) size the filter region
+        // relative to *each* element's own bounding box. Mermaid gives every
+        // node its own translated coordinate system (the box is centred on
+        // the local origin), so a single `userSpaceOnUse` region can't fit
+        // them all — it clips one side of every box. A generous bbox-relative
+        // margin leaves room for the displacement, and the per-element
+        // buffers stay small enough for Safari. `primitiveUnits` keeps the
+        // noise frequency in user space so the wobble looks the same on every
+        // box regardless of its size.
+        const filter = createSvgEl("filter", {
+          id,
+          x: "-50%",
+          y: "-50%",
+          width: "200%",
+          height: "200%",
+          primitiveUnits: "userSpaceOnUse",
+        });
+        filter.appendChild(
+          createSvgEl("feTurbulence", {
+            type: "fractalNoise",
+            baseFrequency: "0.03",
+            numOctaves: "2",
+            seed: "7",
+            result: "noise",
+          }),
+        );
+        filter.appendChild(
+          createSvgEl("feDisplacementMap", {
+            in: "SourceGraphic",
+            in2: "noise",
+            scale: "6",
+            xChannelSelector: "R",
+            yChannelSelector: "G",
+          }),
+        );
+        defs.appendChild(filter);
+      }
+
+      svg
+        .querySelectorAll(".nodes path, .edgePaths path")
+        .forEach((shape) => shape.setAttribute("filter", "url(#" + id + ")"));
+    });
+  }
+
+  async function render() {
+    if (!window.mermaid) return;
+
+    const nodes = document.querySelectorAll(".mermaid");
+    if (!nodes.length) return;
+
+    // Avoid overlapping runs if the theme is toggled rapidly.
+    if (rendering) {
+      rerenderQueued = true;
+      return;
+    }
+    rendering = true;
+
+    nodes.forEach((node) => {
+      if (node.getAttribute("data-source") === null) {
+        // First render: remember the original diagram source.
+        node.setAttribute("data-source", node.textContent);
+      } else {
+        // Re-render: restore source and let Mermaid process it again.
+        node.textContent = node.getAttribute("data-source");
+        node.removeAttribute("data-processed");
+      }
+    });
+
+    // Black-and-white, xkcd-style look: sketchy rough.js strokes, the xkcd
+    // handwriting font, no fills, and a single ink color that follows the
+    // light/dark theme.
+    const inkColor = ink();
+    const fontFamily =
+      '"xkcd", "Comic Sans MS", "Segoe Print", cursive, sans-serif';
+
+    // Initialize with `startOnLoad: false` *before* any `await` below, so
+    // Mermaid's own auto-render can't fire on DOMContentLoaded and paint the
+    // default theme while we're waiting on fonts.
+    window.mermaid.initialize({
+      startOnLoad: false,
+      theme: "base",
+      look: "handDrawn",
+      handDrawnSeed: 1,
+      fontFamily,
+      themeVariables: {
+        fontFamily,
+        // Transparent fills everywhere: no background boxes.
+        background: "transparent",
+        mainBkg: "transparent",
+        secondaryColor: "transparent",
+        tertiaryColor: "transparent",
+        clusterBkg: "transparent",
+        noteBkgColor: "transparent",
+        edgeLabelBackground: "transparent",
+        // Everything else is drawn in a single ink color.
+        primaryColor: "transparent",
+        primaryTextColor: inkColor,
+        primaryBorderColor: inkColor,
+        nodeBorder: inkColor,
+        arrowheadColor: inkColor,
+        secondaryBorderColor: inkColor,
+        tertiaryBorderColor: inkColor,
+        // No frame boxes around subgraphs.
+        clusterBorder: "transparent",
+        noteBorderColor: inkColor,
+        noteTextColor: inkColor,
+        lineColor: inkColor,
+        textColor: inkColor,
+        titleColor: inkColor,
+      },
+    });
+
+    // The hand-drawn look measures text in the "xkcd" font to size nodes, so
+    // wait for it to load to avoid clipped or overflowing labels.
+    if (document.fonts && document.fonts.load) {
+      try {
+        await document.fonts.load('1em "xkcd"');
+        await document.fonts.ready;
+      } catch (_) {
+        // Font loading is best-effort; fall back to whatever is available.
+      }
+    }
+
+    try {
+      await window.mermaid.run({
+        nodes: document.querySelectorAll(".mermaid"),
+      });
+      applySquiggle();
+    } finally {
+      rendering = false;
+      if (rerenderQueued) {
+        rerenderQueued = false;
+        render();
+      }
+    }
+  }
+
+  function init() {
+    render();
+
+    // Re-render whenever the theme changes.
+    new MutationObserver((mutations) => {
+      if (mutations.some((m) => m.attributeName === "data-theme")) {
+        render();
+      }
+    }).observe(document.documentElement, { attributes: true });
+  }
+
+  if (document.readyState === "loading") {
+    document.addEventListener("DOMContentLoaded", init);
+  } else {
+    init();
+  }
+})();