feat(collab-doc): Hocuspocus binary persistence + Next 16 seed/persist fixes (#6059)

* fix(collab-doc): make server-side seed conversion work under Next 16 / Turbopack

Opening a file left both collaborators read-only and stalled ~12s: the server-side
seed (markdown -> Yjs, run through the headless editor engine) was failing, so the
doc never seeded and the editor never left its readiness gate. Two root causes,
both latent until a real build/runtime (typecheck + unit tests don't exercise
either), surfaced by the Next 16 upgrade:

1. Build boundary: the server seed route imported the shared editor schema
   (`createMarkdownContentExtensions`), which pulled in the React node-view
   components (`useEffect`) -> 'client component in a Server Component'. Split each
   node's React-free schema into its own `*-schema.ts` (code-block, image,
   raw-markdown-snippet); the client editor still injects the React node views via
   the existing `nodeViews` param, unchanged.

2. Runtime DOM: the converter installs a jsdom `window` on `globalThis`, but
   Turbopack's server bundle gives bundled `@tiptap/core` a `window` that does NOT
   read `globalThis`, so `elementFromString` threw 'no window object available'.
   Externalize the `@tiptap/*` packages the converter uses (native Node require, so
   their `window` reads the real global) and fix the converter's DOM guard to gate
   on `window` (what TipTap checks) with no sticky flag.

Verified: seed route returns 200 with the Yjs update; 514 collab-doc + editor tests
pass; schema byte-identical after the split.

* feat(collab-doc): persist the Yjs binary and load it on cold-start (Hocuspocus pattern)

Adopt the industry-standard Hocuspocus store/load-document pattern so a cold room
open loads the file's last-persisted Yjs binary directly instead of re-converting
markdown -> Yjs on every open. Rebuilding the CRDT from markdown on each connect is
the exact anti-pattern Tiptap/Yjs warn against (fresh client ids -> duplicated
content); it also forced the fragile server-side headless-editor conversion on every
open. Now conversion runs only on a genuine first open or an external markdown edit.

- new table workspace_file_collab_state(file_id PK->workspace_files cascade,
  doc_state bytea, source_hash, updated_at): the Yjs binary + a hash of the markdown
  it was derived from (bounded <=~1MB by the 256KB round-trip gate). Mirrors
  Hocuspocus's extension-database (binary in a DB column). Migration 0275.
- persist upserts the binary (tagged with the exact markdown just written)
- cold-start seed returns the cached binary when its source_hash matches the file's
  current markdown; otherwise converts (and the next persist refreshes the cache)
- also externalize yjs / y-protocols / lib0 alongside @tiptap: bundling loaded a
  second yjs copy, so @tiptap/y-tiptap's 'instanceof Y.XmlElement' failed on
  app-created nodes ('Unexpected case') during Yjs -> markdown

Verified end-to-end: seed -> persist -> seed returns the exact persisted binary (a
cache hit, no re-conversion). 18 collab-doc + 51 realtime file-doc tests pass.

* fix(collab-doc): best-effort cache read + drop dead barrel

- seed: a cache-read failure (transient DB error, not-yet-migrated cache table)
  no longer aborts a cold room open — the durable markdown is already in hand, so
  fall through to conversion. Symmetric with persist's best-effort cache write.
  Addresses the Cursor Bugbot finding on the read/write asymmetry.
- remove the collab-doc index.ts barrel: nothing imported it (every consumer uses
  direct ./seed / ./merge / ./converter imports), so it was dead re-export surface.
  De-export COLLAB_DOC_FIELD accordingly — it is used only inside converter.ts.

Author: waleedlatif1

apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/code-block-schema.ts

@@ -0,0 +1,34 @@
+import type { JSONContent } from '@tiptap/core'
+import { CodeBlock } from '@tiptap/extension-code-block'
+
+/**
+ * React-free schema half of the code-block node. Lives apart from {@link ./code-block} (its React
+ * node view) so the shared editor schema — `createMarkdownContentExtensions` in `./extensions` — can
+ * be imported by server code (the collab-doc seed converter) without pulling a client component
+ * (`useEffect`) into a Server Component module. The client editor injects the node-view variant
+ * ({@link CodeBlockWithLanguage}) via `nodeViews`.
+ */
+
+function codeBlockText(node: JSONContent): string {
+  return (node.content ?? []).map((child) => child.text ?? '').join('')
+}
+
+/** Fence sized to one backtick longer than the longest run inside the code (CommonMark rule). */
+function fenceFor(text: string): string {
+  const longestRun = Math.max(0, ...[...text.matchAll(/`+/g)].map((match) => match[0].length))
+  return '`'.repeat(Math.max(3, longestRun + 1))
+}
+
+/**
+ * Code block whose markdown serializer sizes the fence to the interior backtick runs, so a code
+ * block that itself contains a ``` line round-trips instead of shattering. Shared by the test
+ * (plain) and live ({@link CodeBlockWithLanguage}) paths.
+ */
+export const MarkdownCodeBlock = CodeBlock.extend({
+  renderMarkdown: (node: JSONContent) => {
+    const language = typeof node.attrs?.language === 'string' ? node.attrs.language : ''
+    const text = codeBlockText(node)
+    const fence = fenceFor(text)
+    return `${fence}${language}\n${text}\n${fence}`
+  },
+})

apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/code-block.tsx

@@ -8,12 +8,11 @@ import {
   DropdownMenuTrigger,
   useCopyToClipboard,
 } from '@sim/emcn'
-import type { JSONContent } from '@tiptap/core'
-import { CodeBlock } from '@tiptap/extension-code-block'
 import type { ReactNodeViewProps } from '@tiptap/react'
 import { NodeViewContent, NodeViewWrapper, ReactNodeViewRenderer } from '@tiptap/react'
 import { Check, ChevronDown, Code, Copy, Eye, WrapText } from 'lucide-react'
 import { looksLikeMermaid, MermaidDiagram } from '../mermaid-diagram'
+import { MarkdownCodeBlock } from './code-block-schema'
 import { detectLanguage } from './detect-language'
 import { useEditorEditable } from './use-editor-editable'
 
@@ -228,30 +227,6 @@ function CodeBlockView({ node, updateAttributes, editor, getPos }: ReactNodeView
   )
 }
 
-function codeBlockText(node: JSONContent): string {
-  return (node.content ?? []).map((child) => child.text ?? '').join('')
-}
-
-/** Fence sized to one backtick longer than the longest run inside the code (CommonMark rule). */
-function fenceFor(text: string): string {
-  const longestRun = Math.max(0, ...[...text.matchAll(/`+/g)].map((match) => match[0].length))
-  return '`'.repeat(Math.max(3, longestRun + 1))
-}
-
-/**
- * Code block whose markdown serializer sizes the fence to the interior backtick runs, so a code
- * block that itself contains a ``` line round-trips instead of shattering. Shared by the test
- * (plain) and live ({@link CodeBlockWithLanguage}) paths.
- */
-export const MarkdownCodeBlock = CodeBlock.extend({
-  renderMarkdown: (node: JSONContent) => {
-    const language = typeof node.attrs?.language === 'string' ? node.attrs.language : ''
-    const text = codeBlockText(node)
-    const fence = fenceFor(text)
-    return `${fence}${language}\n${text}\n${fence}`
-  },
-})
-
 /**
  * Code block with hover-revealed controls (language picker, line-wrap toggle, copy). The
  * `language` attribute drives {@link CodeBlockHighlight}'s Prism highlighting and serializes to

apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/extensions.ts

@@ -11,13 +11,18 @@ import {
 } from '@tiptap/extension-table'
 import { Markdown } from '@tiptap/markdown'
 import StarterKit from '@tiptap/starter-kit'
-import { MarkdownCodeBlock } from './code-block'
+import { MarkdownCodeBlock } from './code-block-schema'
 import { Highlight } from './highlight'
-import { MarkdownImage } from './image'
+import { MarkdownImage } from './image-schema'
 import { MarkdownLinkInputRule } from './link-input-rule'
 import { MarkdownMention } from './mention/mention-node'
 import { SIM_LINK_SCHEME } from './mention/sim-link'
-import { FootnoteDef, FootnoteRef, RawHtmlBlock, RawInlineHtml } from './raw-markdown-snippet'
+import {
+  FootnoteDef,
+  FootnoteRef,
+  RawHtmlBlock,
+  RawInlineHtml,
+} from './raw-markdown-snippet-schema'
 
 /**
  * The `@`-mention link scheme, registered on the Link mark — without it the schema strips the

apps/sim/app/workspace/[workspaceId]/files/components/file-viewer/rich-markdown-editor/image-schema.ts

@@ -0,0 +1,161 @@
+import type { JSONContent } from '@tiptap/core'
+import { Image } from '@tiptap/extension-image'
+
+/**
+ * React-free schema half of the image node. Lives apart from {@link ./image} (its React resize node
+ * view) so the shared editor schema — `createMarkdownContentExtensions` in `./extensions` — can be
+ * imported by server code (the collab-doc seed converter) without pulling a client component
+ * (`useEffect`) into a Server Component module. The client editor injects the node-view variant
+ * ({@link ResizableImage}) via `nodeViews`.
+ */
+
+/**
+ * A markdown linked image `[![alt](src "t")](href "t2")` — an image wrapped in a link, the canonical
+ * form of a README badge. `@tiptap/markdown` parses this as a link mark over an image node, but an
+ * image node can't carry inline marks, so the wrapping link is silently dropped. We instead tokenize
+ * the whole construct ourselves and hang the link target on the image node's `href` attribute, so it
+ * round-trips losslessly (and the file stays editable rather than opening read-only).
+ */
+const LINKED_IMAGE_RE =
+  /^\[!\[([^\]]*)\]\(([^)\s]+)(?:\s+"([^"]*)")?\)\]\(([^)\s]+)(?:\s+"([^"]*)")?\)/
+
+/** Escape a value for safe interpolation into a double-quoted HTML attribute. */
+function escapeAttr(value: string): string {
+  return value
+    .replace(/&/g, '&')
+    .replace(/"/g, '"')
+    .replace(/</g, '&lt;')
+    .replace(/>/g, '&gt;')
+}
+
+/**
+ * Serialize an image to markdown when it has no explicit size, and to an HTML `<img>` tag when
+ * it does — standard markdown has no width syntax, so a resized image must round-trip as HTML to
+ * preserve its dimensions. Unsized images stay clean `![alt](src)`. An image with an `href` is
+ * wrapped in a markdown link so a linked badge round-trips as `[![alt](src)](href)`.
+ *
+ * A *sized **and** linked* image is the one case markdown can't represent: the linked-image tokenizer
+ * only recognizes `[![alt](src)](href)`, so emitting `[<img …>](href)` would silently drop the link on
+ * reparse (and the round-trip-safety probe wouldn't catch it). We keep the link and fall back to the
+ * unsized `[![alt](src)](href)` form — the link matters more than the exact dimensions for a badge.
+ */
+function imageMarkdown(node: JSONContent): string {
+  const attrs = node.attrs ?? {}
+  const src = typeof attrs.src === 'string' ? attrs.src : ''
+  const alt = typeof attrs.alt === 'string' ? attrs.alt : ''
+  const title = typeof attrs.title === 'string' ? attrs.title : ''
+  const href = typeof attrs.href === 'string' ? attrs.href : ''
+  const hrefTitle = typeof attrs.hrefTitle === 'string' ? attrs.hrefTitle : ''
+  const width = attrs.width
+  const height = attrs.height
+  let image: string
+  if ((width || height) && !href) {
+    const parts = [`src="${escapeAttr(src)}"`]
+    if (alt) parts.push(`alt="${escapeAttr(alt)}"`)
+    if (title) parts.push(`title="${escapeAttr(title)}"`)
+    if (width) parts.push(`width="${escapeAttr(String(width))}"`)
+    if (height) parts.push(`height="${escapeAttr(String(height))}"`)
+    image = `<img ${parts.join(' ')}>`
+  } else {
+    // Escape so an alt with `]`/`[` or a title with `"` can't break out of the `![…](… "…")` syntax
+    // and corrupt the round-trip; a src with spaces/parens goes in angle brackets (CommonMark).
+    const titlePart = title ? ` "${title.replace(/["\\]/g, '\\$&')}"` : ''
+    const safeSrc = /[\s()]/.test(src) ? `<${src}>` : src
+    image = `![${alt.replace(/[\\[\]]/g, '\\$&')}](${safeSrc}${titlePart})`
+  }
+  if (!href) return image
+  // Escape `"`/`\` so an href title can't break out of the `[…](href "title")` syntax (mirrors the
+  // image title escaping above).
+  const hrefTitlePart = hrefTitle ? ` "${hrefTitle.replace(/["\\]/g, '\\$&')}"` : ''
+  return `[${image}](${href}${hrefTitlePart})`
+}
+
+interface MarkdownImageToken {
+  /** Set only by our linked-image tokenizer; absent on the built-in `![](src)` token. */
+  src?: string
+  alt?: string
+  title?: string | null
+  /** Built-in image token holds the source URL here; our linked token holds the link target. */
+  href?: string
+  hrefTitle?: string | null
+  /** Built-in image token holds the alt text here. */
+  text?: string
+}
+
+/** Map both the built-in image token and our linked-image token onto the image node's attributes. */
+function parseImageToken(token: MarkdownImageToken): JSONContent {
+  const isLinked = typeof token.src === 'string'
+  return {
+    type: 'image',
+    attrs: isLinked
+      ? {
+          src: token.src,
+          alt: token.alt ?? '',
+          title: token.title ?? null,
+          href: token.href ?? null,
+          hrefTitle: token.hrefTitle ?? null,
+        }
+      : {
+          src: token.href ?? '',
+          alt: token.text ?? '',
+          title: token.title ?? null,
+          href: null,
+          hrefTitle: null,
+        },
+  }
+}
+
+const widthAttr = {
+  default: null,
+  parseHTML: (element: HTMLElement) => element.getAttribute('width'),
+  renderHTML: (attributes: Record<string, unknown>) =>
+    attributes.width ? { width: String(attributes.width) } : {},
+}
+
+const heightAttr = {
+  default: null,
+  parseHTML: (element: HTMLElement) => element.getAttribute('height'),
+  renderHTML: (attributes: Record<string, unknown>) =>
+    attributes.height ? { height: String(attributes.height) } : {},
+}
+
+/** Link target of a linked image — markdown-only state, never emitted as an HTML `<img>` attribute. */
+const hrefAttr = { default: null, rendered: false }
+const hrefTitleAttr = { default: null, rendered: false }
+
+/**
+ * Image node that carries optional `width`/`height` (serialized as an HTML `<img>` tag) and an
+ * optional `href`/`hrefTitle` (a wrapping markdown link, for badges). Shared by the headless
+ * round-trip path (no node view) and the live {@link ResizableImage}.
+ */
+export const MarkdownImage = Image.extend({
+  addAttributes() {
+    return {
+      ...this.parent?.(),
+      width: widthAttr,
+      height: heightAttr,
+      href: hrefAttr,
+      hrefTitle: hrefTitleAttr,
+    }
+  },
+  markdownTokenizer: {
+    name: 'image',
+    level: 'inline',
+    start: (src: string) => src.indexOf('[!['),
+    tokenize: (src: string): (MarkdownImageToken & { type: string; raw: string }) | undefined => {
+      const match = LINKED_IMAGE_RE.exec(src)
+      if (!match) return undefined
+      return {
+        type: 'image',
+        raw: match[0],
+        alt: match[1] ?? '',
+        src: match[2],
+        title: match[3] ?? null,
+        href: match[4],
+        hrefTitle: match[5] ?? null,
+      }
+    },
+  },
+  parseMarkdown: parseImageToken,
+  renderMarkdown: imageMarkdown,
+})