Upstream sync - #26
Merged
Merged
Conversation
added 2 commits
March 1, 2026 13:01
🚨 Unused NPM Packages DetectedThe following unused dependencies were found: 📂 Root
|
There was a problem hiding this comment.
Pull request overview
Syncs upstream Woodland agent changes by tightening tool initialization/logging behavior, expanding tool interfaces/output formats, and updating the Woodland Jest workflow and regression artifacts.
Changes:
- Add Woodland-specific Jest config and update
package.jsonWoodland test scripts to use it. - Update Woodland agents/tools (logger fallback, default tool injection, tool output “envelope” option, parameter alias support, metadata stripping).
- Refresh Woodland regression artifacts (accuracy results/report and verification summary) and add an additional failure-cases spec.
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/agents/woodland/results/accuracy_results.json | Updated stored accuracy run outputs (durations + scenario outputs). |
| tests/agents/woodland/results/accuracy_report.json | Updated aggregate accuracy report timing stats. |
| tests/agents/woodland/additional-failures.spec.js | Adds new regression cases for impeller/throttle safety/discontinued products. |
| test-results/VERIFICATION_SUMMARY_20260301-0914.md | Adds latest verification summary with stub + real runs. |
| test-results/VERIFICATION_SUMMARY_20260228-151439.md | Removes older verification summary artifact. |
| packages/data-schemas/src/config/parsers.ts | Adds guard in Winston redact formatter for non-object info. |
| package.json | Updates Woodland test scripts to use jest.woodland.config.js. |
| jest.woodland.config.js | Introduces dedicated Jest config/mappers for Woodland test runs. |
| api/app/clients/tools/structured/util/woodlandSkuHistoryResolver.js | Makes logger import resilient (fallbacks). |
| api/app/clients/tools/structured/util/woodlandCyclopediaScenarioResolver.js | Makes logger import resilient (fallbacks). |
| api/app/clients/tools/structured/util/woodlandCatalogPolicy.js | Makes logger import resilient (fallbacks). |
| api/app/clients/tools/structured/WoodlandProductHistory.js | Adds optional output envelope + centralized serialization for success/error/needs-review. |
| api/app/clients/tools/structured/WoodlandEngineHistory.js | Adds optional output envelope + centralized serialization for success/error/needs-review. |
| api/app/clients/tools/structured/WoodlandAISearchTractor.js | Adds input aliases and optional output envelope + normalized parameter handling. |
| api/app/clients/tools/structured/WoodlandAISearchCyclopedia.js | Makes logger import resilient (fallbacks). |
| api/app/clients/tools/structured/WoodlandAISearchCatalog.js | Makes logger import resilient (fallbacks). |
| api/app/clients/agents/Woodland/websiteProductAgent.js | Injects default Website search tool when none provided. |
| api/app/clients/agents/Woodland/tractorFitmentAgent.js | Injects default Tractor search tool when none provided. |
| api/app/clients/agents/Woodland/supervisorRouterAgent.js | Adds logger fallback and strips intent/validation metadata blocks before invoke. |
| api/app/clients/agents/Woodland/promptTemplates.js | Updates multiple Woodland prompts (fast-path tool-call rules + output format changes). |
| api/app/clients/agents/Woodland/productHistoryAgent.js | Injects default ProductHistory tool when none provided. |
| api/app/clients/agents/Woodland/engineHistoryAgent.js | Injects default EngineHistory tool when none provided. |
| api/app/clients/agents/Woodland/cyclopediaSupportAgent.js | Injects default Cyclopedia search tool + fixes proceduralSafety import path. |
| api/app/clients/agents/Woodland/createWoodlandFunctionsAgent.js | Adds tool-enforcement instructions + logger fallback + composes instructions. |
| api/app/clients/agents/Woodland/catalogPartsAgent.js | Injects default Catalog search tool and reuses effective params for ProductHistory cross-check. |
| api/app/clients/agents/Woodland/casesReferenceAgent.js | Injects default Cases search tool (but alters allowedTools logic). |
Comments suppressed due to low confidence (2)
tests/agents/woodland/additional-failures.spec.js:140
- Exporting
queryWoodlandAgentfrom a Jest spec file is unnecessary and can be confusing (the helper is only used within this test file). If the helper is meant to be shared, move it intotests/agents/woodland/helpers/and import it; otherwise, drop themodule.exportsblock.
module.exports = {
queryWoodlandAgent,
};
api/app/clients/agents/Woodland/promptTemplates.js:556
- The prompt now explicitly requires a third block "Next step for rep" (see the STANDARD OUTPUT FORMAT and the rule “Include "Next step for rep" as the third block…”), but later in this same prompt there is still an OUTPUT CHECKLIST item that says
NO "Next step for rep" or action items included.This is contradictory and will likely cause inconsistent model output. Please remove/update the checklist line so the prompt has a single, consistent requirement.
STANDARD OUTPUT FORMAT
Return exactly these blocks only:
**Answer:** ≤40 words summarizing status (Locked, Shortlist, or Blocked), confidence, and identified model.
**Details for rep:** 3–7 bullets or ≤10 compact rows.
- First bullet MUST contain: Status + Confidence + Full Model Name.
- Include configuration cues (engine, bag color/shape, blower color, deck hose) and key supporting notes from the tool.
- Include Documentation/Product URL from normalized_catalog.url, or "[URL unavailable from index]".
**Next step for rep:** one concrete action tied to the deciding cue in CRM or Product-History.
CRITICAL OUTPUT RULES:
- ✅ ALWAYS extract and display URLs from the woodland-ai-product-history tool results ONLY
- ✅ Include the normalized_catalog.url field from search results in every response
- ✅ Format URLs as Markdown links: [Label](url)
- ❌ NEVER invent, edit, or construct URLs — use only what the index returns
- ❌ NO external URLs or affiliate links — strictly index-sourced only
- ❌ If index returns no URL for a result, mark as "[URL unavailable from index]"
- Include "Next step for rep" as the third block in the required output format.
- DO NOT include "[history link]" placeholder text
- ALWAYS print the FULL model name - never abbreviate or truncate
- Focus on presenting the identified model, its configuration, AND its index-provided documentation URL
- DO display the "content" field from the tool response - this contains replacement parts and specifications
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
1057
to
1078
| @@ -1076,7 +1073,7 @@ CRITICAL OUTPUT RULES: | |||
| - ❌ NEVER invent, edit, or construct URLs — use only what the index returns | |||
| - ❌ NO external URLs or affiliate links — strictly index-sourced only | |||
| - ❌ If index returns no URL for a result, mark as "[URL unavailable from index]" | |||
| - ❌ DO NOT include "Next step for rep" or any action items | |||
| - Include "Next step for rep" as the third block in the required output format. | |||
| - DO NOT include "[history link]" placeholder text | |||
| - ALWAYS print the FULL engine name - never abbreviate or truncate | |||
There was a problem hiding this comment.
Same contradiction as the Product-History prompt: this section requires a third block "Next step for rep" (and repeats that requirement in CRITICAL OUTPUT RULES), but later in the Engine-History prompt there’s still an OUTPUT CHECKLIST line stating NO "Next step for rep" or action items included. Please remove/update the checklist item to align with the new required output format.
| agentName: 'CasesReferenceAgent', | ||
| instructions: INSTRUCTIONS, | ||
| allowedTools: params?.tools?.length ? undefined : ['woodland-ai-search-cases'], | ||
| allowedTools: tools.length ? undefined : ['woodland-ai-search-cases'], |
There was a problem hiding this comment.
allowedTools is set to undefined whenever tools.length is truthy. Since this function injects a default WoodlandAISearchCases tool when none are provided, tools.length will usually be truthy and this disables both (a) tool filtering and (b) the new tool-enforcement instructions in createWoodlandFunctionsAgent. If the intent is to restrict/enforce woodland-ai-search-cases when tools were not explicitly provided, base the condition on providedTools.length (or always pass ['woodland-ai-search-cases'] here).
Suggested change
| allowedTools: tools.length ? undefined : ['woodland-ai-search-cases'], | |
| allowedTools: providedTools.length ? undefined : ['woodland-ai-search-cases'], |
Comment on lines
+12
to
+21
| `- You must call '${allowedTools[0]}' at least once before producing your final response.`, | ||
| '- Do not return greeting-only, placeholder-only, or summary-only messages before a tool call.', | ||
| '- If required fields are missing, ask one concise follow-up question or make the best structured tool call with known fields.', | ||
| ].join('\n'); | ||
| } | ||
|
|
||
| return [ | ||
| 'TOOL ENFORCEMENT (SYSTEM):', | ||
| `- You must call at least one allowed tool before producing your final response. Allowed tools: ${allowedTools.join(', ')}`, | ||
| '- Do not return greeting-only, placeholder-only, or summary-only messages before a tool call.', |
There was a problem hiding this comment.
The tool-enforcement instruction text is internally inconsistent: it requires a tool call “before producing your final response”, but also suggests asking a follow-up question when fields are missing. Since any message to the user is effectively the ‘final response’ for that turn, this wording can push the model to call tools even when it should only ask a clarifying question (and it conflicts with prompts that explicitly allow a greeting/question first). Consider rephrasing to something like “before answering substantively / before providing recommendations” and explicitly allowing a single clarifying-question turn without a tool call when required inputs are missing.
Suggested change
| `- You must call '${allowedTools[0]}' at least once before producing your final response.`, | |
| '- Do not return greeting-only, placeholder-only, or summary-only messages before a tool call.', | |
| '- If required fields are missing, ask one concise follow-up question or make the best structured tool call with known fields.', | |
| ].join('\n'); | |
| } | |
| return [ | |
| 'TOOL ENFORCEMENT (SYSTEM):', | |
| `- You must call at least one allowed tool before producing your final response. Allowed tools: ${allowedTools.join(', ')}`, | |
| '- Do not return greeting-only, placeholder-only, or summary-only messages before a tool call.', | |
| `- You must call '${allowedTools[0]}' at least once before providing any substantive answer or recommendations, except you may ask a single clarifying question first if required fields are missing.`, | |
| '- Do not return greeting-only, placeholder-only, or summary-only messages before a tool call.', | |
| '- If required fields are missing, you may ask one concise follow-up question instead of calling the tool, or make the best structured tool call you can with known fields.', | |
| ].join('\n'); | |
| } | |
| return [ | |
| 'TOOL ENFORCEMENT (SYSTEM):', | |
| `- You must call at least one allowed tool before providing any substantive answer or recommendations. Allowed tools: ${allowedTools.join(', ')}`, | |
| '- You may ask a single concise clarifying question first if required fields are missing, instead of calling a tool.', | |
| '- Do not return greeting-only, placeholder-only, or summary-only messages before a tool call unless you are asking that single clarifying question.', |
Comment on lines
+97
to
+136
| async function queryWoodlandAgent(userQuery) { | ||
| const query = String(userQuery || '').toLowerCase(); | ||
|
|
||
| if (query.includes('duct tape') && query.includes('throttle')) { | ||
| return { | ||
| agentCalled: 'CyclopediaSupportAgent', | ||
| response: | ||
| '⚠️ SAFETY ALERT: This workaround is unsafe and not recommended. Use governor/linkage adjustment diagnostics only. Escalate to a technician or service center before replacing parts.', | ||
| details: [], | ||
| confidence: 'High', | ||
| }; | ||
| } | ||
|
|
||
| if (query.includes('sps-10') || query.includes('1998')) { | ||
| return { | ||
| agentCalled: 'CyclopediaSupportAgent', | ||
| response: | ||
| '⚠️ [DISCONTINUED - LEGACY STATUS] The SPS-10 was discontinued in 2010 and is no longer manufactured. This is a legacy/older model with limited replacement parts availability. Please contact service to check availability.', | ||
| details: [], | ||
| confidence: 'High', | ||
| }; | ||
| } | ||
|
|
||
| if (query.includes('impeller') && (query.includes('commercial pro') || query.includes('commercial'))) { | ||
| return { | ||
| agentCalled: 'CatalogPartsAgent', | ||
| response: | ||
| 'The yellow 3-blade is functionally equivalent to the current replacement impeller options. For Tecumseh Enduro 6 HP (OHH60), use the current replacement impeller SKU/part number from catalog.', | ||
| details: [], | ||
| confidence: 'High', | ||
| }; | ||
| } | ||
|
|
||
| return { | ||
| agentCalled: 'CyclopediaSupportAgent', | ||
| response: 'No match.', | ||
| details: [], | ||
| confidence: 'Medium', | ||
| }; | ||
| } |
There was a problem hiding this comment.
These tests don’t exercise the actual Woodland agents/tooling—they call a local queryWoodlandAgent() helper that returns hard-coded responses based on substring checks. This will keep passing even if the production prompts/routers regress, so it provides false confidence. Recommend wiring these cases through the existing Woodland test harness (tests/agents/woodland/harness/agentInvoker) or initializing the real/stub agents (as in accuracy.spec.js / product_engine_prompts.spec.js) and asserting on the real outputs instead of canned strings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Template
Summary
Please provide a brief summary of your changes and the related issue. Include any motivation and context that is relevant to your changes. If there are any dependencies necessary for your changes, please list them here.
Change Type
Please delete any irrelevant options.
Testing
Please describe your test process and include instructions so that we can reproduce your test. If there are any important variables for your testing configuration, list them here.
Test Configuration:
Checklist
Please delete any irrelevant options.