🤖 Multi-Domain Support Triage Agent

HackerRank Orchestrate Hackathon 2026

A terminal-based AI support triage agent that automatically classifies, routes, and responds to support tickets across three domains — HackerRank, Claude (Anthropic), and Visa — using RAG (Retrieval-Augmented Generation).

🏆 Hackathon Context

Built for the HackerRank Orchestrate Hackathon, where the challenge was to build a support triage agent that:

• Handles tickets across three company ecosystems
• Grounds every response strictly in the provided support corpus
• Makes smart escalation decisions for high-risk or sensitive tickets
• Produces a structured output CSV with five prediction columns

---

🧠 How It Works

CSV Row (issue, subject, company)
        ↓
  classifier.py  →  resolve company   (CSV → keyword → LLM fallback)
        ↓
  classifier.py  →  escalation risk check  (keyword scan, no LLM)
        ↓
  retriever.py   →  embed query, cosine search, +0.20 company boost
        ↓
  agent.py       →  build prompt with top-10 chunks, call LLM
        ↓
  agent.py       →  validate JSON output + safety override
        ↓
  output.csv     →  issue, subject, company, response, product_area,
                    status, request_type, justification

---

✨ Key Features

• RAG Pipeline — retrieves top-10 relevant chunks from a local corpus before every LLM call; responses are grounded in documentation, not model memory
• Multi-backend LLM support — works with Groq API (cloud) or Ollama (fully local, no API key needed)
• Three-layer company resolver — CSV value → keyword scan → LLM fallback; LLM only fires when keywords fail
• Smart escalation logic — keyword-based risk detection (fraud, jailbreak, account lockout) + confidence-based safety override
• Embedding cache — corpus is indexed once and cached; rebuilds automatically only when files change
• Fully deterministic — temperature=0 on all LLM calls for reproducible outputs

---

🏗️ Architecture

File	Responsibility
retriever.py	Chunks corpus (1200 words / 300 overlap), embeds with all-MiniLM-L6-v2, caches index, exposes retrieve()
classifier.py	Keyword company detector + escalation risk keywords + LLM fallback
prompts.py	System prompts and user prompt builders
agent.py	Full pipeline: resolve → risk check → retrieve → LLM → validate → safety override
main.py	CLI entry point with argparse, summary stats, debug logging

---

⚙️ Design Decisions

Decision	Why
RAG over fine-tuning	Corpus changes frequently; RAG updates by rebuilding index. No labelled data needed.
all-MiniLM-L6-v2	Runs locally, free, cached after first run. No second API dependency.
1200-word chunks	Captures one complete support article per chunk. Less fragmentation.
300-word overlap	Prevents answers from being split across chunk boundaries.
+0.20 company boost	Soft nudge toward correct domain without hard-filtering cross-domain results.
Top-10 retrieval	Broader coverage for multi-document answers across three domains.
Threshold = 0.26	Real matches score 0.28–0.55 with 1200-word chunks. 0.26 sits just below that floor.
temperature=0	Same input → same output. Makes debugging and reproducibility possible.
Three-layer resolver	Keywords handle ~90% of cases for free. LLM only fires on genuinely ambiguous tickets.

---

🚀 Getting Started

Prerequisites

pip install groq sentence-transformers numpy python-dotenv ollama

Setup

Create a .env file in the project root:

GROQ_API_KEY=gsk_your_key_here

# Optional — for Ollama backend
TRIAGE_BACKEND=ollama
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=llama3

Project Structure

project-root/
├── code/
│   ├── agent.py
│   ├── classifier.py
│   ├── retriever.py
│   ├── prompts.py
│   └── main.py
├── data/
│   ├── hackerrank/
│   ├── claude/
│   └── visa/
├── support_tickets/
│   ├── support_tickets.csv
│   ├── sample_support_tickets.csv
│   └── output.csv
└── .env

---

🖥️ Usage

# Run on sample tickets (Groq)
python code/main.py --sample

# Run on full dataset
python code/main.py

# Smoke test — first 5 rows only
python code/main.py --sample --limit 5

# Use Ollama (local, no API key)
python code/main.py --backend ollama --sample

# Use a specific model
python code/main.py --backend ollama --model mistral --sample

# Debug mode — shows retrieval scores + raw LLM output
python code/main.py --sample --debug

All CLI Flags

Flag	Default	Description
--backend	groq	groq or ollama
--model	model default	Override model name
--sample	—	Use sample CSV
--input	support_tickets/support_tickets.csv	Custom input path
--output	support_tickets/output.csv	Custom output path
--limit N	—	Process first N rows only
--data-dir	data/	Corpus directory
--debug	—	Enable debug logging

---

📊 Output Format

output.csv columns:

Column	Description
issue	Original ticket body
subject	Original subject line
company	Resolved company
response	User-facing answer grounded in corpus
product_area	Support category (e.g. Account Access, Billing)
status	replied or escalated
request_type	product_issue / feature_request / bug / invalid
justification	Internal routing rationale

---

🛡️ Escalation Logic

The agent escalates when any of these are true:

• Keyword triggers (deterministic, no LLM): fraud, unauthorized charges, account locked/banned, jailbreak/prompt injection attempts
• LLM decision: topic out of scope, no corpus support, legal/safety risk
• Safety override: escalation keyword present + LLM said "replied" + top retrieval score < 0.26

---

🔧 Tech Stack

• Python 3.11
• Groq API — llama-3.3-70b-versatile (cloud)
• Ollama — local LLM support
• sentence-transformers — all-MiniLM-L6-v2 for embeddings
• numpy — cosine similarity via matrix multiply
• python-dotenv — secrets management

---

📈 Retrieval Details

• Corpus walked recursively from data/ — supports .txt, .md, .html, .json
• HTML: <script> and <style> blocks stripped before chunking
• JSON: string leaf values extracted (no structural noise)
• Embeddings cached to .retriever_cache.pkl, keyed on corpus content hash
• Company boost: +0.20 added to chunks from the resolved company

---

Built for HackerRank Orchestrate Hackathon — May 2025