Skip to content

feat: improve tip generation prompt with richer guidance #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jayaramkr wants to merge 1 commit into
base: main
Choose a base branch
from
+24 −11
Open

feat: improve tip generation prompt with richer guidance #124

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 24 additions & 11 deletions evolve/llm/tips/prompts/generate_tips.jinja2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
You are analyzing an AI agent's execution trajectory to extract actionable tips.
Extract actionable, relevant tips from this trajectory that would help an AI agent perform similar tasks better in the future.

# Task Information
**Task:** {{task_instruction}}
Expand All @@ -8,16 +8,26 @@ You are analyzing an AI agent's execution trajectory to extract actionable tips.
# Agent Trajectory
{{trajectory_summary}}

# Your Task
Extract 3-5 actionable tips from this trajectory that would help AI agents perform similar tasks better.
**IMPORTANT TO REMEMBER:**
1. Only generate tips if they are truly relevant and actionable
2. Tips should be specific to patterns observed in this trajectory
3. Include both positive patterns (what worked) and negative patterns (what to avoid)
4. Each tip should have:
- A clear, concise description (content)
- The purpose/benefit of following it
- The category: "strategy", "recovery", or "optimization"
- Specific steps to implement the tip
- A trigger condition (when to apply this tip)
Comment on lines +15 to +20
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Schema mismatch: implementation_steps field does not exist in the Tip model.

The instructions ask the LLM to include "Specific steps to implement the tip" (line 19), but the Tip Pydantic model in evolve/schema/tips.py only defines 4 fields: content, rationale, category, and trigger. If the LLM emits an implementation_steps field, it will either cause a ValidationError (returning an empty tips list per evolve/llm/tips/tips.py:154-162) or be silently discarded.

Either remove the instruction for implementation steps, or update the Tip schema to include the new field.

Option A: Remove the mismatched instruction 
 4. Each tip should have:
    - A clear, concise description (content)
    - The purpose/benefit of following it
    - The category: "strategy", "recovery", or "optimization"
-   - Specific steps to implement the tip
    - A trigger condition (when to apply this tip)
Option B: Update the Tip schema (in evolve/schema/tips.py) 
class Tip(BaseModel):
    content: str = Field(description="Clear, actionable tip")
    rationale: str = Field(description="Why this tip helps")
    category: Literal["strategy", "recovery", "optimization"]
    implementation_steps: list[str] = Field(description="Specific steps to implement this tip")
    trigger: str = Field(description="When to apply this tip")

If you choose Option B, also update the JSON example in this template (lines 35-44) and the clustering code in evolve/llm/tips/clustering.py.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
4. Each tip should have:
- A clear, concise description (content)
- The purpose/benefit of following it
- The category: "strategy", "recovery", or "optimization"
- Specific steps to implement the tip
- A trigger condition (when to apply this tip)
4. Each tip should have:
- A clear, concise description (content)
- The purpose/benefit of following it
- The category: "strategy", "recovery", or "optimization"
- A trigger condition (when to apply this tip)
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@evolve/llm/tips/prompts/generate_tips.jinja2` around lines 15 - 20, The
template asks for "implementation_steps" but the Tip Pydantic model lacks that
field, causing validation failures in tips.py; fix by either removing the
"Specific steps to implement the tip" line from generate_tips.jinja2 or by
adding implementation_steps: list[str] to the Tip model (class Tip) and then
update any consumers (the JSON example in the template and the clustering logic
in clustering.py) and ensure tips.py validation accepts the new field; reference
symbols: Tip, implementation_steps, generate_tips.jinja2, tips.py,
clustering.py.


**Guidelines:**
1. Focus on patterns that worked or mistakes that were made
2. Be specific to what you observed in this trajectory
3. Each tip should have:
- Clear description of what to do (or avoid)
- Why it matters
- When to apply it
5. If the task succeeded, focus on the successful strategies used
6. If the task failed, focus on what went wrong and how to prevent/recover from it
Comment on lines +22 to +23
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Guidance references task success/failure, but status is always "UNKNOWN".

Lines 22-23 instruct the LLM to focus on successful strategies if the task succeeded, or on failure modes if it failed. However, line 5 shows **Status:** UNKNOWN, and the rendering code in evolve/llm/tips/tips.py:106-125 does not pass a task_succeeded or similar variable to the template.

The LLM cannot follow this conditional guidance without knowing the actual outcome. Either:

  1. Pass the task outcome to the template and update line 5, or
  2. Remove/rephrase this guidance to avoid confusing the LLM.
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@evolve/llm/tips/prompts/generate_tips.jinja2` around lines 22 - 23, The
template guidance refers to task success/failure but the template is always
rendered with "Status: UNKNOWN" because evolve/llm/tips/tips.py (around the
render function at lines 106-125) does not pass the task outcome; fix by adding
a boolean (e.g., task_succeeded) or outcome string to the context when calling
the Jinja2 render in the function that renders generate_tips.jinja2 and update
line 5 in generate_tips.jinja2 to use that variable (e.g., "**Status:** {{
'SUCCEEDED' if task_succeeded else 'FAILED' }}") so the conditional guidance is
meaningful, or if you prefer not to change call sites, remove/rephrase the
conditional lines 5-6 in generate_tips.jinja2 to avoid referencing task
success/failure.

7. Do not generate generic tips - be specific to this task execution
8. Look for patterns in how the agent:
- Discovered and used APIs
- Handled authentication and credentials
- Iterated through results (pagination)
- Structured its approach to the problem
- Handled errors or unexpected responses

{% if not constrained_decoding_supported %}
**Output Format (JSON):**
Expand All @@ -35,4 +45,7 @@ Extract 3-5 actionable tips from this trajectory that would help AI agents perfo
```

Generate tips now. Return ONLY the JSON, no other text.
{% endif %}
{% endif %}



Loading