Add evals and improve SKILL.md with version routing and review workflow#4
Add evals and improve SKILL.md with version routing and review workflow#4
Conversation
…ow, and reference index Eval diagnostics revealed gaps in the original SKILL.md: - No version-specific guidance (v12/v13/v14 routing) - No structured review workflow for code reviews - Missing reference index for security, performance, migration docs - No explicit deprecation cross-referencing instructions SKILL.md improvements (216 -> 386 words, under 500 limit): - Added Version-Specific Guidance section - Added Review Workflow with 6 concrete steps - Added Reference Index table mapping needs to files - Expanded rules to include deprecation checking
There was a problem hiding this comment.
Code Review
This pull request enhances the TYPO3 TypoScript reference documentation by expanding usage examples, adding version-specific guidance for TYPO3 v12 through v14, and defining a structured review workflow. Additionally, a new evaluation suite is introduced to verify the assistant's proficiency in TypoScript and Fluid tasks. Review feedback correctly identifies a factual inaccuracy regarding the deprecation of FLUIDTEMPLATE in v14 and recommends correcting a regex pattern in the evaluation assertions to align with the defined CLI flags.
| scripts/lookup.sh --checklist typoscript | ||
| - **v12**: Use FLUIDTEMPLATE, sys_template static includes, constants.typoscript | ||
| - **v13**: Prefer PAGEVIEW for new page templates, introduce Site Sets, use settings.definitions.yaml | ||
| - **v14**: Site Sets mandatory, FLUIDTEMPLATE deprecated, @import replaces INCLUDE_TYPOSCRIPT |
There was a problem hiding this comment.
The statement that FLUIDTEMPLATE is deprecated in v14 contradicts the provided reference files. According to references/review/migration-v12-to-v13.md (line 13) and references/review/deprecations.md (line 76), FLUIDTEMPLATE is NOT deprecated and continues to work in v13 and v14, although PAGEVIEW is recommended for new page templates.
| - **v14**: Site Sets mandatory, FLUIDTEMPLATE deprecated, @import replaces INCLUDE_TYPOSCRIPT | |
| - **v14**: Site Sets mandatory, @import replaces INCLUDE_TYPOSCRIPT, PAGEVIEW recommended |
| "name": "lint_rules_lookup", | ||
| "prompt": "What lint rules should I follow when writing TypoScript?", | ||
| "assertions": [ | ||
| { "type": "tool_use", "tool": "Bash", "pattern": "lookup\\.sh.*--lint" }, |
There was a problem hiding this comment.
The assertion pattern lookup\.sh.*--lint is imprecise. The actual flag defined in SKILL.md and implemented in lookup.sh is --lint-rules. Updating the pattern ensures the evaluation specifically verifies the correct flag usage.
| { "type": "tool_use", "tool": "Bash", "pattern": "lookup\\.sh.*--lint" }, | |
| { "type": "tool_use", "tool": "Bash", "pattern": "lookup\\.sh.*--lint-rules" }, |
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Snapshot WarningsEnsure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice. Scanned FilesNone |
Summary
SKILL.md Improvements
Eval analysis revealed these gaps in the original SKILL.md, now addressed:
A/B Analysis (20 evals)
Summary: 7 evals improved from WEAK to PASS, 13 unchanged (already passing). Zero regressions.
Test plan