Feature 534495: Documentation Generation (GH Pages) by mokhld · Pull Request #38 · DEFRA/forms-engine-plugin

mokhld · 2025-04-03T16:54:40Z

Ticket Link:

https://dev.azure.com/defragovuk/DEFRA-CDP/_workitems/edit/534495

Proposed change

This PR introduces:

Markdown schema generation from the schema jsons in https://www.npmjs.com/package/@defra/forms-engine-plugin?activeTab=versions.
GH Workflow that build and publishes the GH Jekyll pages documentation site (here).
Tests for the schema docs generation script.

Azure DevOps work item:

Type of change

Bug fix
New feature
Breaking change
Misc. (documentation, build updates, etc)

Checklist

You have executed this code locally and it performs as expected.
You have added tests to verify your code works.
You have added code comments and JSDoc, where appropriate.
There is no commented-out code.
You have added developer docs in README.md and docs/* (where appropriate, e.g. new features).
The tests are passing (npm run test).
The linting checks are passing (npm run lint).
The code has been formatted (npm run format).

This reverts commit 477701d.

- Removed outdated navigation settings for schemas and features in the Jekyll configuration script. - Added explicit configuration for the features section, including navigation orders and parent-child relationships. - Cleaned up the script by removing unnecessary custom CSS creation steps, streamlining the documentation generation process.

- Modified schemas/index.md to include `nav_exclude: true` in the front matter, preventing it from appearing in the navigation structure.

- Replaced inline JavaScript for fixing links with a separate file for better maintainability. - Updated the create-jekyll-config.sh script to copy the link-fixer JavaScript instead of generating it inline. - Removed outdated link adjustments from fix-docs.sh to simplify the documentation processing. - Added `nav_exclude: true` to schemas/index.md and set `toc: false` in the schema reference for improved navigation control.

- Changed the title in both _config.yml and process-docs.sh from "DXT Documentation" to "DXT Docs" to maintain uniformity across documentation files.

mokhld · 2025-04-09T16:27:01Z

Well done on this, huge effort! 🎉

The .github/scripts/docs/fix-docs.sh and .github/scripts/docs/fix-schema-links.sh need a bit of a rethink though to be honest, the sed operations specifically are going to break easily when the .md drifts from the script.

Keen to get this merged so we have a docs website anyway, so maybe we pick that up separately?

Yeah agreed. A lot of the complexity for the schema links lie in how deeply nested they can be, but we can def look to simply later.

- Created a new SCSS file for custom styling of the site title, enhancing the visual presentation on larger screens. - Updated the create-jekyll-config.sh script to include the new SCSS file in the documentation generation process.

alexluckett · 2025-04-09T16:52:26Z

.github/scripts/docs/fix-docs.sh

+for location in "." "docs"; do
+  if [ -d "$location" ]; then
+    echo "  Checking $location directory"
+    for file in "$location"/*.md "$location"/GETTING_STARTED.md "$location"/PLUGIN_OPTIONS.md "$location"/CONTRIBUTING.md; do


"$location"/GETTING_STARTED.md "$location"/PLUGIN_OPTIONS.md "$location"/CONTRIBUTING.md are redundant, no? already covered by the first pattern "$location"/*.md?

Yh good spot, thats been removed

alexluckett · 2025-04-09T16:53:35Z

.github/scripts/docs/fix-schema-links.sh

+  # === Fix all .md links to match Jekyll's pretty permalinks AND add baseurl ===
+  sed "${SED_INPLACE[@]}" -E 's|\[([^]]+)\]\(([^)]+)\.md(#[^)]+)?\)|\[\1\]\(/forms-engine-plugin/\2\3\)|g' "$file"
+  sed "${SED_INPLACE[@]}" -E 's|\[([^]]+)\]\(([^)]+)\.md\)|\[\1\]\(/forms-engine-plugin/\2\)|g' "$file"
+  # Fix plain / roots to include baseurl
+  sed "${SED_INPLACE[@]}" -E 's|\[([^]]+)\]\(\/([^)]+)\)|\[\1\]\(/forms-engine-plugin/\2\)|g' "$file"
+  # Fix relative links to be absolute with baseurl
+  sed "${SED_INPLACE[@]}" -E 's|\[([^]]+)\]\(\./([^)]+)\)|\[\1\]\(/forms-engine-plugin/\2\)|g' "$file"


can you put an example of what these do? as with all regex, they're hard to read 🙂

alexluckett · 2025-04-09T16:54:05Z

.github/scripts/docs/fix-schema-links.sh

+    # Fix common schema reference patterns
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\(([a-zA-Z0-9_-]+-schema[a-zA-Z0-9_-]*)(\.md)?\)/[\1](\2)/g' "$file"
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\(([a-zA-Z0-9_-]+-schema-[a-zA-Z0-9_-]*)(\.md)?\)/[\1](\2)/g' "$file"
+
+    # Handle properties references
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\(([a-zA-Z0-9_-]+-properties-[a-zA-Z0-9_-]*)(\.md)?\)/[\1](\2)/g' "$file"
+
+    # Fix references to validation schemas
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\((min|max)(-length|-schema|-future|-past)?(\.md)?\)/[\1](\2\3)/g' "$file"
+
+    # Handle other schema patterns
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\((search|sorting|query|list)-options-schema(-[a-zA-Z0-9_-]*)?(\.md)?\)/[\1](\2-options-schema\3)/g' "$file"
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\((page|form|component)-([a-zA-Z0-9_-]+)(-[a-zA-Z0-9_-]*)?(\.md)?\)/[\1](\2-\3\4)/g' "$file"
+
+    # Extra pass for nested property references
+    sed "${SED_INPLACE[@]}" -E 's/\[([^\]]+)\]\(([a-zA-Z0-9_-]+)-schema-properties-([a-zA-Z0-9_-]+)(-[a-zA-Z0-9_-]*)?(\.md)?\)/[\1](\2-schema-properties-\3\4)/g' "$file"


ditto to the above comment

alexluckett · 2025-04-09T16:55:32Z

.github/scripts/docs/fix-schema-links.sh

+  sed "${SED_INPLACE[@]}" -E 's|\[see our guidance on page events\]([^)]*)/PAGE_EVENTS\.md|\[see our guidance on page events\]\(\.\.\/configuration-based\/PAGE_EVENTS\)|g' "./features/code-based/PAGE_VIEWS.md"
+fi
+
+if [ -f "./features/configuration-based/PAGE_TEMPLATES.md" ]; then
+  echo "  Fixing PAGE_TEMPLATES.md links"
+  sed "${SED_INPLACE[@]}" -E 's|\[see our guidance on page events\]([^)]*)/PAGE_EVENTS\.md|\[see our guidance on page events\]\(\.\.\/configuration-based\/PAGE_EVENTS\)|g' "./features/configuration-based/PAGE_TEMPLATES.md"


these will break fairly easily. Can you add a comment about what exactly is being fixed? we can come back to these as a follow-up,

This can be removed as was before we started absolute links in the md files themselves.

alexluckett · 2025-04-09T16:57:04Z

.github/scripts/docs/fix-schema-links.sh

+  sed "${SED_INPLACE[@]}" -E 's|\[([^]]+)\]\(\.\./configuration-based/PAGE_EVENTS\.?m?d?\)|\[\1\]\(\/features\/configuration-based\/PAGE_EVENTS\)|g' "./features/code-based/PAGE_VIEWS.md"
+  sed "${SED_INPLACE[@]}" -E 's|\[([^]]+)\]\(PAGE_EVENTS\.?m?d?\)|\[\1\]\(\/features\/configuration-based\/PAGE_EVENTS\)|g' "./features/code-based/PAGE_VIEWS.md"


these will break fairly easily. Can you add a comment about what exactly is being fixed? we can come back to these as a follow-up.

This can be removed as was before we started absolute links in the md files themselves.

alexluckett · 2025-04-09T16:58:38Z

.github/scripts/docs/process-docs.sh

@@ -0,0 +1,282 @@
+#!/bin/bash
+# process-docs.sh - Fixed to properly format front matter AND set correct navigation


Do we need this script if we just hardcode the front matter in the .md files directly?

Yes and no. We're already producing front matter in the schema docs script, but in this script is fixes the schema property via the regex and handles the schema specific navigation. A large chunk has been removed as we're handling the links in the non-schema markdown files directly though.

In a later ticket, we could prob look to simplify it.

…ove redundant file writing - Copied custom SCSS for site title styling from the script to a dedicated file, enhancing maintainability. - Updated the create-jekyll-config.sh script to reflect the new SCSS file structure. - Removed redundant file writing for README.md in the generate-schema-docs.js script to simplify the documentation generation process.

- Updated fix-docs.sh to process only markdown files in specified directories, simplifying the script. - Enhanced fix-schema-links.sh with detailed comments and additional regex patterns to standardize schema links and remove .md extensions. - Adjusted links in CUSTOM_SERVICES.md to use absolute paths for better navigation consistency.

- Updated regex patterns to exclude external links from base URL adjustments, ensuring that only internal links are modified. - Added comments for clarity on the changes made to link handling in the script.

- Modified links in CUSTOM_SERVICES.md, PAGE_EVENTS.md, and PAGE_TEMPLATES.md to use absolute paths for the forms-engine-plugin directory, improving navigation accuracy across the documentation.

- Updated fix-schema-links.sh to ensure schema links have the correct prefix, improving link accuracy. - Modified fix-links.js to handle schema links and other internal links more effectively, ensuring proper base URL application. - Adjusted PAGE_VIEWS.md to use the updated link format for consistency across documentation.

…orkflow triggers - Removed unnecessary blank line in fix-schema-links.sh for cleaner code. - Updated documentation-generation.yml to ensure it triggers on pushes to the main branch, enhancing workflow reliability.

- Changed the link to the GitHub workflows in CONTRIBUTING.md to an absolute URL, improving accessibility and clarity for contributors.

sonarqubecloud · 2025-04-10T16:05:50Z

Quality Gate failed

Failed conditions
1 Security Hotspot
86.5% Coverage on New Code (required ≥ 90%)

See analysis details on SonarQube Cloud

mokhld temporarily deployed to github-pages-test April 4, 2025 11:22 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 4, 2025 11:42 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 4, 2025 11:49 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 4, 2025 12:06 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 4, 2025 12:12 — with GitHub Actions Inactive

mokhld had a problem deploying to github-pages-test April 4, 2025 14:01 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 14:02 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 14:05 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 14:08 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 14:40 — with GitHub Actions Failure

mokhld temporarily deployed to github-pages-test April 4, 2025 14:43 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 4, 2025 15:07 — with GitHub Actions Inactive

mokhld had a problem deploying to github-pages-test April 4, 2025 15:13 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 15:40 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 16:23 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 17:18 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 17:32 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 4, 2025 17:38 — with GitHub Actions Failure

mokhld temporarily deployed to github-pages-test April 4, 2025 18:00 — with GitHub Actions Inactive

mokhld had a problem deploying to github-pages-test April 7, 2025 18:28 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 7, 2025 18:32 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 7, 2025 18:47 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 7, 2025 18:54 — with GitHub Actions Failure

mokhld temporarily deployed to github-pages-test April 7, 2025 19:12 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 7, 2025 20:31 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 7, 2025 20:46 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 7, 2025 21:12 — with GitHub Actions Inactive

mokhld had a problem deploying to github-pages-test April 7, 2025 21:27 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 7, 2025 21:31 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 7, 2025 21:38 — with GitHub Actions Failure

Revert "docs: enhance Jekyll configuration and documentation structure"

d795dbe

This reverts commit 477701d.

mokhld temporarily deployed to github-pages-test April 9, 2025 13:21 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 9, 2025 13:22 — with GitHub Actions Inactive

mokhld had a problem deploying to github-pages-test April 9, 2025 13:49 — with GitHub Actions Failure

mokhld had a problem deploying to github-pages-test April 9, 2025 14:06 — with GitHub Actions Failure

fix: missing EOF

03a64c2

mokhld temporarily deployed to github-pages-test April 9, 2025 14:11 — with GitHub Actions Inactive

docs: update schemas/index.md to exclude from navigation

0ff0e14

- Modified schemas/index.md to include `nav_exclude: true` in the front matter, preventing it from appearing in the navigation structure.

mokhld temporarily deployed to github-pages-test April 9, 2025 14:58 — with GitHub Actions Inactive

mokhld temporarily deployed to github-pages-test April 9, 2025 16:09 — with GitHub Actions Inactive

docs: update title in Jekyll configuration for consistency

aa29663

- Changed the title in both _config.yml and process-docs.sh from "DXT Documentation" to "DXT Docs" to maintain uniformity across documentation files.

mokhld temporarily deployed to github-pages-test April 9, 2025 16:17 — with GitHub Actions Inactive

mokhld marked this pull request as ready for review April 9, 2025 16:39

mokhld requested a review from alexluckett April 9, 2025 16:40

mokhld temporarily deployed to github-pages-test April 9, 2025 16:49 — with GitHub Actions Inactive

mokhld changed the title ~~Add schema documentation generation script and update project configu…~~ Feature 534495: Documentation Generation (GH Pages) Apr 9, 2025

alexluckett reviewed Apr 9, 2025

View reviewed changes

mokhld added 6 commits April 9, 2025 18:04

docs: enhance link processing in fix-schema-links.sh

9ee1a50

- Updated regex patterns to exclude external links from base URL adjustments, ensuring that only internal links are modified. - Added comments for clarity on the changes made to link handling in the script.

docs: update links in documentation for consistency

22ddd9c

- Modified links in CUSTOM_SERVICES.md, PAGE_EVENTS.md, and PAGE_TEMPLATES.md to use absolute paths for the forms-engine-plugin directory, improving navigation accuracy across the documentation.

alexluckett approved these changes Apr 10, 2025

View reviewed changes

docs: update link in CONTRIBUTING.md for clarity

87bef5d

- Changed the link to the GitHub workflows in CONTRIBUTING.md to an absolute URL, improving accessibility and clarity for contributors.

		sed "${SED_INPLACE[@]}" -E 's\|\[([^]]+)\]\(\.\./configuration-based/PAGE_EVENTS\.?m?d?\)\|\[\1\]\(\/features\/configuration-based\/PAGE_EVENTS\)\|g' "./features/code-based/PAGE_VIEWS.md"
		sed "${SED_INPLACE[@]}" -E 's\|\[([^]]+)\]\(PAGE_EVENTS\.?m?d?\)\|\[\1\]\(\/features\/configuration-based\/PAGE_EVENTS\)\|g' "./features/code-based/PAGE_VIEWS.md"

		@@ -0,0 +1,282 @@
		#!/bin/bash
		# process-docs.sh - Fixed to properly format front matter AND set correct navigation

Conversation

mokhld commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ticket Link:

Proposed change

Type of change

Checklist

Uh oh!

mokhld commented Apr 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 10, 2025

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mokhld commented Apr 3, 2025 •

edited

Loading