Skip to content

fix: improve Cypher query generation accuracy #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hupeky wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: improve Cypher query generation accuracy #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions codebase_rag/cypher_queries.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@
CYPHER_EXAMPLE_LIMIT_ONE = """MATCH (f:File) RETURN f.path as path, f.name as name, labels(f) as type LIMIT 1"""

CYPHER_EXAMPLE_CLASS_METHODS = f"""MATCH (c:Class)-[:DEFINES_METHOD]->(m:Method)
WHERE c.qualified_name ENDS WITH '.UserService'
RETURN m.name AS name, m.qualified_name AS qualified_name, labels(m) AS type
WHERE c.name = 'UserService'
RETURN c.name AS className, m.name AS methodName, m.qualified_name AS qualified_name, labels(m) AS type
LIMIT {CYPHER_DEFAULT_LIMIT}"""

CYPHER_EXPORT_NODES = """
Expand Down
10 changes: 9 additions & 1 deletion codebase_rag/prompts.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,14 @@ def build_rag_orchestrator_prompt(tools: list["Tool"]) -> str:
- CORRECT: `MATCH (c:Class) RETURN count(c) AS total`
- WRONG: `MATCH (c:Class) RETURN c.name, count(c) AS total` (returns all items!)

**VALUE PATTERN RULES (CRITICAL FOR NAME MATCHING):**
- The `qualified_name` property contains FULL paths like: `'Project.folder.subfolder.ClassName'`
- When users mention a class or function by SHORT NAME (e.g., "VatManager", "UserService"), you MUST match using the `name` property, NOT `qualified_name`.
- CORRECT: `WHERE c.name = 'VatManager'`
- WRONG: `WHERE c.qualified_name = 'VatManager'` (will never match!)
- Use `DEFINES_METHOD` relationship to find methods of a class.
- Use `DEFINES` relationship to find functions/classes defined in a module.

**Examples:**

* **Natural Language:** "How many classes are there?"
Expand Down Expand Up @@ -235,7 +243,7 @@ def build_rag_orchestrator_prompt(tools: list["Tool"]) -> str:
```

* **Natural Language:** "What methods does UserService have?" or "Show me methods in UserService" or "List UserService methods"
* **Cypher Query (Use ENDS WITH to match class by short name):**
* **Cypher Query (Note: match by `name` property, use `DEFINES_METHOD` relationship):**
```cypher
{CYPHER_EXAMPLE_CLASS_METHODS}
```
Expand Down
27 changes: 24 additions & 3 deletions codebase_rag/services/llm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,30 @@ def _create_provider_model(config: ModelConfig) -> Model:


def _clean_cypher_response(response_text: str) -> str:
query = response_text.strip().replace(cs.CYPHER_BACKTICK, "")
if query.startswith(cs.CYPHER_PREFIX):
query = query[len(cs.CYPHER_PREFIX) :].strip()
"""Clean LLM response to extract pure Cypher query.

Handles markdown formatting that models sometimes output:
- Triple backticks (```cypher ... ```)
- Bold text (**Cypher Query:**)
- Headers and other markdown
"""
Comment on lines +29 to +35
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

According to the project's general rules, docstrings are not allowed. Please remove this docstring to adhere to the project's coding standards.

References
  1. Docstrings are not allowed in this project, as enforced by a pre-commit hook.

import re
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Per PEP 8, imports should be at the top of the file. Please remove this import from here and add import re to the top-level imports section of the file.

References
  1. PEP 8: Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants. (link)

Copy link
Contributor

@greptile-apps greptile-apps bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move import re to top-level imports (after line 1). Module-level imports belong with stdlib imports at the file top, not inside functions.

Suggested change
import re
"""Clean LLM response to extract pure Cypher query.
Handles markdown formatting that models sometimes output:
- Triple backticks (```cypher ... ```)
- Bold text (**Cypher Query:**)
- Headers and other markdown
"""

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Prompt To Fix With AI 
This is a comment left during a code review.
Path: codebase_rag/services/llm.py
Line: 36:36

Comment:
Move `import re` to top-level imports (after line 1). Module-level imports belong with stdlib imports at the file top, not inside functions.

```suggestion
    """Clean LLM response to extract pure Cypher query.

    Handles markdown formatting that models sometimes output:
    - Triple backticks (```cypher ... ```)
    - Bold text (**Cypher Query:**)
    - Headers and other markdown
    """
```

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.


query = response_text.strip()

# Extract content from code blocks if present (```cypher ... ``` or ``` ... ```)
code_block_match = re.search(r"```(?:cypher)?\s*(.*?)```", query, re.DOTALL | re.IGNORECASE)
if code_block_match:
query = code_block_match.group(1).strip()
else:
# Remove markdown bold/headers (e.g., **Cypher Query:**)
query = re.sub(r"\*\*[^*]+\*\*:?\s*", "", query)
# Remove single backticks
query = query.replace(cs.CYPHER_BACKTICK, "")
# Remove "cypher" prefix if present
if query.lower().startswith(cs.CYPHER_PREFIX):
query = query[len(cs.CYPHER_PREFIX):].strip()
Comment on lines +44 to +51
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The current logic in the else block may not correctly handle all cases with leading whitespace, and the cleaning steps could be ordered for better robustness. For example, a response like **Cypher Query:** MATCH (n) would result in a query with a leading space: MATCH (n);, which could cause execution to fail. This refactoring handles whitespace more consistently and correctly identifies and removes the cypher prefix even if it has leading spaces.

Suggested change
else:
# Remove markdown bold/headers (e.g., **Cypher Query:**)
query = re.sub(r"\*\*[^*]+\*\*:?\s*", "", query)
# Remove single backticks
query = query.replace(cs.CYPHER_BACKTICK, "")
# Remove "cypher" prefix if present
if query.lower().startswith(cs.CYPHER_PREFIX):
query = query[len(cs.CYPHER_PREFIX):].strip()
else:
# Remove markdown bold/headers (e.g., **Cypher Query:**)
query = re.sub(r"\*\*[^*]+\*\*:?\s*", "", query)
# Remove "cypher" prefix if present
if query.lower().strip().startswith(cs.CYPHER_PREFIX):
query = query[query.lower().find(cs.CYPHER_PREFIX) + len(cs.CYPHER_PREFIX):]
# Remove single backticks and strip any remaining whitespace
query = query.replace(cs.CYPHER_BACKTICK, "").strip()

Comment on lines +50 to +51
Copy link
Contributor

@greptile-apps greptile-apps bot Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Case mismatch: query.lower().startswith() but using original cs.CYPHER_PREFIX length. If cs.CYPHER_PREFIX = "cypher" and query is "CYPHER MATCH...", slicing by len("cypher") (6 chars) from "CYPHER MATCH..." works correctly. However, for safety and clarity, use consistent casing.

Suggested change
if query.lower().startswith(cs.CYPHER_PREFIX):
query = query[len(cs.CYPHER_PREFIX):].strip()
if query.lower().startswith(cs.CYPHER_PREFIX.lower()):
query = query[len(cs.CYPHER_PREFIX):].strip()
Prompt To Fix With AI 
This is a comment left during a code review.
Path: codebase_rag/services/llm.py
Line: 50:51

Comment:
Case mismatch: `query.lower().startswith()` but using original `cs.CYPHER_PREFIX` length. If `cs.CYPHER_PREFIX = "cypher"` and query is `"CYPHER MATCH..."`, slicing by `len("cypher")` (6 chars) from `"CYPHER MATCH..."` works correctly. However, for safety and clarity, use consistent casing.

```suggestion
        if query.lower().startswith(cs.CYPHER_PREFIX.lower()):
            query = query[len(cs.CYPHER_PREFIX):].strip()
```

How can I resolve this? If you propose a fix, please make it concise.


if not query.endswith(cs.CYPHER_SEMICOLON):
query += cs.CYPHER_SEMICOLON
return query
Expand Down