Skip to content

Fix CI hang: Don't try to render Graphviz diagrams with GFM#1230

Merged
nrichers merged 8 commits into
mainfrom
nrichers/sc-15275/ci-validate-llm-markdown-render-step-hangs
Mar 24, 2026
Merged

Fix CI hang: Don't try to render Graphviz diagrams with GFM#1230
nrichers merged 8 commits into
mainfrom
nrichers/sc-15275/ci-validate-llm-markdown-render-step-hangs

Conversation

@nrichers
Copy link
Copy Markdown
Collaborator

@nrichers nrichers commented Mar 20, 2026

Pull Request Description

What and why?

The "Validate LLM markdown render" CI step was consistently hanging on site/developer/how-to/testing-overview.qmd, the only file in the codebase that contains {dot} graphviz diagrams inside a .panel-tabset.

Root cause: When Quarto renders to GFM format, it attempts to render graphviz diagrams to PNG using Chrome. This was causing the CI to hang indefinitely.

Fix: After some experiments, I ended up replacing the diagrams with PNG files for now. For local testing, I added a make render-llm action for local troubleshooting but I cannot replicate the issue. Also noticed we were getting pandoc errors during the cleanup phase that removes some HTML tags that make it past the GFM render, now fixed.

Other stuff I tried: I tried a conditional exclude and installing chromium but the content render still hung on the diagrams on GitHub, likely because of the sequence in which Quarto renders the content. Because the issue occurs only on GitHub, troubleshooting is slow going.

Moral of the story: diagrams in tabs are problematic.

How to test

What needs special review?

N/A

Dependencies, breaking changes, and deployment notes

None

Release notes

N/A — internal CI fix

Checklist

  • What and why
  • How to test
  • Labels applied
  • PR linked to Shortcut

@nrichers nrichers added the internal Not to be externalized in the release notes label Mar 20, 2026
@nrichers nrichers force-pushed the nrichers/sc-15275/ci-validate-llm-markdown-render-step-hangs branch from 798220a to 05ea0a1 Compare March 21, 2026 00:29
@nrichers nrichers force-pushed the nrichers/sc-15275/ci-validate-llm-markdown-render-step-hangs branch from fe4e75a to 76638ee Compare March 23, 2026 17:05
@nrichers nrichers requested a review from validbeck March 23, 2026 20:32
@nrichers nrichers changed the title Fix CI hang: exclude graphviz diagrams from GFM rendering Fix CI hang: Don't try to render Graphviz diagrams with GFM Mar 23, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Lighthouse check results

⚠️ WARN: Average accessibility score is 0.87 (required: >0.9) — Check the workflow run

Show Lighthouse scores

Folder depth level checked: 0

Commit SHA: fd50d0d

Modify the workflow to check a different depth:

  • 0: Top-level navigation only — /index.html, /guide/guides.html, ...
  • 1: All first-level subdirectories — /guide/.html, /developer/.html, ...
  • 2: All second-level subdirectories — /guide/attestation/*.html, ...
Page Accessibility Performance Best Practices SEO
/developer/validmind-library.html 0.85 0.66 1.00 0.82
/get-started/get-started.html 0.85 0.71 1.00 0.73
/guide/guides.html 0.81 0.75 1.00 0.82
/index.html 0.93 0.69 1.00 0.82
/releases/all-releases.html 0.86 0.69 1.00 0.73
/support/support.html 0.91 0.68 1.00 0.82
/training/training.html 0.85 0.62 0.96 0.73

@github-actions
Copy link
Copy Markdown
Contributor

PR Summary

This PR introduces a new feature to render the site into GitHub Flavored Markdown (GFM) tailored for LLM ingestion. To support this, the PR makes several key changes:

  1. GitHub Workflows Updates:

    • In both the publish and validation workflows, additional steps have been added to install Pandoc using APT. This ensures that the environment has the necessary tools to convert documentation to markdown during CI runs.
  2. Makefile Enhancements:

    • A new phony target named render-llm is added along with an update to the PHONY list. This target echoes messages and runs the llm/render.sh and llm/clean.sh scripts to transform the site content for LLM ingestion. The output is directed to the llm/_llm-output/ directory.
  3. Documentation Update:

    • The testing overview documentation has been updated to replace inline dot diagram definitions with static images. This reduces clutter in the markdown file and ensures that users see pre-rendered diagrams, which likely improves load times and compatibility.

Overall, these modifications streamline the process of rendering documentation for LLM applications and ensure that CI workflows are correctly configured to support the necessary tools.

Test Suggestions

  • Verify that the GitHub workflows correctly install Pandoc and that the commands run without errors.
  • Test the new render-llm Makefile target locally to ensure it successfully executes the rendering and cleaning scripts and outputs the expected files.
  • Check that the updated documentation displays the images correctly and that the links to the generated images work as intended.
  • Run the LLM rendering process in a staging environment to ensure that the overall documentation output is as expected.

Copy link
Copy Markdown
Collaborator

@validbeck validbeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just made a quick adjustment in commit #60365ee to remove the white background from the images that wasn't there in the original version of the charts:

Image Image

@github-actions
Copy link
Copy Markdown
Contributor

Validate docs site

✓ INFO: A live preview of the docs site is available — Open the preview

@nrichers nrichers merged commit 2e9d212 into main Mar 24, 2026
7 of 8 checks passed
@nrichers nrichers deleted the nrichers/sc-15275/ci-validate-llm-markdown-render-step-hangs branch March 24, 2026 01:50
nrichers added a commit that referenced this pull request Mar 24, 2026
* Expand custom artifact type content

* Add margin footnote link

* Change to definition list

* Edits

* docs: Organization wide default tracking changes in content blocks  (#1202)

* Edit: Managing your organization

* WIP: Collaborate with others > Tracking changes

* Edit: Collaborate with others > Tracking changes

* Proofread

* docs: Insert inventory & artifact field values as variables into documents & library blocks (#1207)

* Save point

* Save point

* Work with content blocks > Reference field values

* Single-source prep

* Editing

* Save point

* Save point

* Save point

* Edit

* Editing...

* New content editing toolbar image

* Missing copyrights

* docs: Individual model inventory field activity (#1206)

* WIP: View model activity > Filter all model activity

* Save point

* View model activity: View individual model field activity

* Save point

* Save point

* Save point

* Proofreading

* Proofreading again

* Typo

* Update site/guide/model-inventory/edit-model-inventory-fields.qmd

Co-authored-by: Nik Richers <nik@validmind.ai>

* Fixing numbering

* Fixing indenting

---------

Co-authored-by: Nik Richers <nik@validmind.ai>

* feat: docs pipeline to render markdown

* feat: bump quarto to pre-release

* chore: run llm gen at the end so we don't delay pr previews

* LUA filter + Pandoc post-processing experiment

* docs: 'document' parameter in `vm.init()` for model document target (#1211)

* WIP

* Save point

* Save point

* Save point

* Edit: Install & initialize the ValidMind Library

* Cursor WIP

* Save point

* Save point

* Save point

* Save point

* Save point

* Store model credentials in env files

* What's next

* Updating notebook listing scripts for page-size: 4

* Save point

* Quick edit

* One more

* and another

* Hang on forgot training stuff

* Updated get started

* Updated enable monitoring

* Training updated

* copyright

* Missed a module

* feat: make llm publishing slimmer- no release no installation no library

* chore: put vm-library llm docs back

* Switch to sparse checkout for some repos

* fix: errors in publish llm gha

* chore: handle the case where branch name is not main or prod, for instance when testing

* Add "Quick reference" section to test overview page (#1205)

* Add key concepts section to test overview page

* Make section a callout, change to quick reference

* Save point for graphviz

* Some graphviz tweaks

* A few more tweaks

* Fill colour tweak

* Document vm_today and datetime support in calculated fields (#1167)

* Document vm_today and datetime support in calculated fields

Add collapsible example for date arithmetic in model inventory
calculation fields: vm_today, date, datetime, timedelta, relativedelta.
Includes next review date and days-remaining countdown examples.
Notes timestamp input and string output behavior.

[sc-7465]

* Fix whitespace

* Fix heading level and move inventory field types up one section

* Add screenshots, edits after testing

* Save point before moving content

* Move into its own file and relocate

* Better intro

* Restore better vm_today example

* Edits

* Minor text reschuffle

* Mention fields by name

* Move field types reference content below prequisites and bump to h2

* Fix example casing

* Consistency edit

* Rename and restructure Supported Models page (#1163)

* Rename and restructure Supported Models page

- Rename supported-models.qmd to supported-model-frameworks.qmd
- Add alias for old URL to preserve links
- Restructure content into clearer sections:
  - Framework support with wrapper classes and install commands
  - Test input requirements (predict vs predict_proba)
  - Custom model wrappers (FunctionModel, PipelineModel)
  - GenAI and LLM support
  - RAG evaluation with RAGAS
  - Python and dependency compatibility
- Add Mermaid diagrams for class hierarchy and test input flow
- Update references in faq-integrations.qmd and overview-model-documentation.qmd

* Fix missed sidebar update

* Fix broken link

* Add bridging sentence after callout

* Re-add a what does supported mean heading

* Minor edit

* Re-add some old goodness

* Much better detail

* Edits

* More edits

* Edits

* Remove old supported models page

* Minor edit

* Edits

* File rename again

* Remove duplicated embeddings content

* Uncomment and move info about tests for text-generating models

* Update site/developer/supported-models-and-frameworks.qmd

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Update site/developer/supported-models-and-frameworks.qmd

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Update site/developer/supported-models-and-frameworks.qmd

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Update site/developer/supported-models-and-frameworks.qmd

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Update site/developer/supported-models-and-frameworks.qmd

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Switch to definition list after removing lead-in text for bulleted list

* Two columns for a neater layout

* Remove pointless link

* Add links, text edits

* Update site/developer/supported-models-and-frameworks.qmd

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Whitespace

* Address review comment from Juan

* Remove links I'd added

---------

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* docs: Updated document type names (#1224)

* WIP

* WIP

* Editing validation report template def

* Edit...

* Glossary / about updated

* FAQ updated

* Quickstart -screenshots updated

* Work with test results done

* Enable monitoring done

* Review monitoring results done

* Work with metrics over time done

* save point

* save point

* Editing Export documents

* Export documents done

* Glossary proofreading

* Save point

* Save point

* Save point

* Save point

* Save point

* Save point

* Save point

* Save point

* Edit Manage document types

* Edit Manage model documents

* Edit Customize document templates

* Edit Manage document templates

* Edit Export documents

* Edit

* Edit

* Quickstart - Register a model

* Quickstart - Select a template

* Quickstart - Work with {document}

* Proofread

* Pulling in latest notebooks

* Removing unused page

* Removing backup screenshots

* Revert "Removing unused page"

This reverts commit 9a710fd.

* Revering a change that didn't make sense

* Updating notebooks again

* edit

* docs: Delete and archive artifact types (#1228)

* WIP

* WIP

* Edit: Manage artifact types

* Edit

* Edit after Slack convo

* Add R support documentation (#1227)

* Initial draft

* Add more install info

* Add quickstarts

* Grouping install & initalize in sidebar

* Silly sidebar variable quirk

---------

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Switch to RELEASE_NOTES_RO_PAT for installation repo (#1231)

* Fix CI hang: Don't try to render Graphviz diagrams with GFM (#1230)

* Add pandoc install step

* Try chromium install since stuff renders fine locally

* Switch to RELEASE_NOTES_RO_PAT for installation repo

* Try images instead of Graphviz

* Add render-llm make action for local testing

* Remove Chromium install steps

* BG transparency

---------

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Add auto-generated template schema documentation (#1166)

* Add auto-generated template schema documentation

- Create generate_template_schema_docs.py script to generate HTML schema
  docs from backend JSON Schema using json-schema-for-humans
- Update customize-document-templates.qmd to use generated include
- Add template schema and permissions documentation sections to README.md

The template schema is now regenerated from the v5 JSON schema in the
backend repo, replacing the outdated hardcoded HTML.

Closes sc-6164

* Switch to {html} codeblock, move styles into separate file, style tweaks

* Fixed whitespace issue and added rounded corners

* Fix heading level

* Delete generated template schema file

* Add make action and move template schema docs generation into workflows

* Minor edit

* Fix style for No Additional Items badges

* Kibosh wayward ellipsis

* Remove permissions documentation section from my other PR

* Strip out schema doc CSS that causes issues, style tweaks

* Move expand and collapse buttons above first accordion

* Smidgen of whitespace before embed

* Add README.md section of required repos, remove old R info

* Temporarily add Andres' branch ref

* Switch to new UI schema JSON

* Remove temporary branch ref

* docs: Export analytics (#1225)

* Setup: Export analytics

* WIP

* edit

* Proofreading

---------

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Spencer Krum <nibz@validmind.ai>
nrichers added a commit that referenced this pull request Mar 25, 2026
* Expand custom artifact type content

* Add margin footnote link

* Change to definition list

* Edits

* docs: Organization wide default tracking changes in content blocks  (#1202)

* Edit: Managing your organization

* WIP: Collaborate with others > Tracking changes

* Edit: Collaborate with others > Tracking changes

* Proofread

* docs: Insert inventory & artifact field values as variables into documents & library blocks (#1207)

* Save point

* Save point

* Work with content blocks > Reference field values

* Single-source prep

* Editing

* Save point

* Save point

* Save point

* Edit

* Editing...

* New content editing toolbar image

* Missing copyrights

* docs: Individual model inventory field activity (#1206)

* WIP: View model activity > Filter all model activity

* Save point

* View model activity: View individual model field activity

* Save point

* Save point

* Save point

* Proofreading

* Proofreading again

* Typo

* Update site/guide/model-inventory/edit-model-inventory-fields.qmd



* Fixing numbering

* Fixing indenting

---------



* feat: docs pipeline to render markdown

* feat: bump quarto to pre-release

* chore: run llm gen at the end so we don't delay pr previews

* LUA filter + Pandoc post-processing experiment

* docs: 'document' parameter in `vm.init()` for model document target (#1211)

* WIP

* Save point

* Save point

* Save point

* Edit: Install & initialize the ValidMind Library

* Cursor WIP

* Save point

* Save point

* Save point

* Save point

* Save point

* Store model credentials in env files

* What's next

* Updating notebook listing scripts for page-size: 4

* Save point

* Quick edit

* One more

* and another

* Hang on forgot training stuff

* Updated get started

* Updated enable monitoring

* Training updated

* copyright

* Missed a module

* feat: make llm publishing slimmer- no release no installation no library

* chore: put vm-library llm docs back

* Switch to sparse checkout for some repos

* fix: errors in publish llm gha

* chore: handle the case where branch name is not main or prod, for instance when testing

* Add "Quick reference" section to test overview page (#1205)

* Add key concepts section to test overview page

* Make section a callout, change to quick reference

* Save point for graphviz

* Some graphviz tweaks

* A few more tweaks

* Fill colour tweak

* Document vm_today and datetime support in calculated fields (#1167)

* Document vm_today and datetime support in calculated fields

Add collapsible example for date arithmetic in model inventory
calculation fields: vm_today, date, datetime, timedelta, relativedelta.
Includes next review date and days-remaining countdown examples.
Notes timestamp input and string output behavior.

[sc-7465]

* Fix whitespace

* Fix heading level and move inventory field types up one section

* Add screenshots, edits after testing

* Save point before moving content

* Move into its own file and relocate

* Better intro

* Restore better vm_today example

* Edits

* Minor text reschuffle

* Mention fields by name

* Move field types reference content below prequisites and bump to h2

* Fix example casing

* Consistency edit

* Rename and restructure Supported Models page (#1163)

* Rename and restructure Supported Models page

- Rename supported-models.qmd to supported-model-frameworks.qmd
- Add alias for old URL to preserve links
- Restructure content into clearer sections:
  - Framework support with wrapper classes and install commands
  - Test input requirements (predict vs predict_proba)
  - Custom model wrappers (FunctionModel, PipelineModel)
  - GenAI and LLM support
  - RAG evaluation with RAGAS
  - Python and dependency compatibility
- Add Mermaid diagrams for class hierarchy and test input flow
- Update references in faq-integrations.qmd and overview-model-documentation.qmd

* Fix missed sidebar update

* Fix broken link

* Add bridging sentence after callout

* Re-add a what does supported mean heading

* Minor edit

* Re-add some old goodness

* Much better detail

* Edits

* More edits

* Edits

* Remove old supported models page

* Minor edit

* Edits

* File rename again

* Remove duplicated embeddings content

* Uncomment and move info about tests for text-generating models

* Update site/developer/supported-models-and-frameworks.qmd



* Update site/developer/supported-models-and-frameworks.qmd



* Update site/developer/supported-models-and-frameworks.qmd



* Update site/developer/supported-models-and-frameworks.qmd



* Update site/developer/supported-models-and-frameworks.qmd



* Switch to definition list after removing lead-in text for bulleted list

* Two columns for a neater layout

* Remove pointless link

* Add links, text edits

* Update site/developer/supported-models-and-frameworks.qmd



* Whitespace

* Address review comment from Juan

* Remove links I'd added

---------



* docs: Updated document type names (#1224)

* WIP

* WIP

* Editing validation report template def

* Edit...

* Glossary / about updated

* FAQ updated

* Quickstart -screenshots updated

* Work with test results done

* Enable monitoring done

* Review monitoring results done

* Work with metrics over time done

* save point

* save point

* Editing Export documents

* Export documents done

* Glossary proofreading

* Save point

* Save point

* Save point

* Save point

* Save point

* Save point

* Save point

* Save point

* Edit Manage document types

* Edit Manage model documents

* Edit Customize document templates

* Edit Manage document templates

* Edit Export documents

* Edit

* Edit

* Quickstart - Register a model

* Quickstart - Select a template

* Quickstart - Work with {document}

* Proofread

* Pulling in latest notebooks

* Removing unused page

* Removing backup screenshots

* Revert "Removing unused page"

This reverts commit 9a710fd.

* Revering a change that didn't make sense

* Updating notebooks again

* edit

* docs: Delete and archive artifact types (#1228)

* WIP

* WIP

* Edit: Manage artifact types

* Edit

* Edit after Slack convo

* Add R support documentation (#1227)

* Initial draft

* Add more install info

* Add quickstarts

* Grouping install & initalize in sidebar

* Silly sidebar variable quirk

---------



* Switch to RELEASE_NOTES_RO_PAT for installation repo (#1231)

* Fix CI hang: Don't try to render Graphviz diagrams with GFM (#1230)

* Add pandoc install step

* Try chromium install since stuff renders fine locally

* Switch to RELEASE_NOTES_RO_PAT for installation repo

* Try images instead of Graphviz

* Add render-llm make action for local testing

* Remove Chromium install steps

* BG transparency

---------



* Add auto-generated template schema documentation (#1166)

* Add auto-generated template schema documentation

- Create generate_template_schema_docs.py script to generate HTML schema
  docs from backend JSON Schema using json-schema-for-humans
- Update customize-document-templates.qmd to use generated include
- Add template schema and permissions documentation sections to README.md

The template schema is now regenerated from the v5 JSON schema in the
backend repo, replacing the outdated hardcoded HTML.

Closes sc-6164

* Switch to {html} codeblock, move styles into separate file, style tweaks

* Fixed whitespace issue and added rounded corners

* Fix heading level

* Delete generated template schema file

* Add make action and move template schema docs generation into workflows

* Minor edit

* Fix style for No Additional Items badges

* Kibosh wayward ellipsis

* Remove permissions documentation section from my other PR

* Strip out schema doc CSS that causes issues, style tweaks

* Move expand and collapse buttons above first accordion

* Smidgen of whitespace before embed

* Add README.md section of required repos, remove old R info

* Temporarily add Andres' branch ref

* Switch to new UI schema JSON

* Remove temporary branch ref

* docs: Export analytics (#1225)

* Setup: Export analytics

* WIP

* edit

* Proofreading

---------

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Spencer Krum <nibz@validmind.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal Not to be externalized in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants