LANGFUSE DATA PULL - README
============================

This folder pulls data out of Langfuse (the LLM observability tool used to
track the OpenFN assistant's conversations and quality evaluators) using
Langfuse's JavaScript SDK, and turns it into CSV files for analysis.

SETUP
-----
1. You need Node.js installed (check with `node -v` in a terminal).
2. Get API keys from Langfuse: log into cloud.langfuse.com, open your
   project, go to Settings > API Keys, and create a new key pair.
3. Copy .env.example to .env and paste in your real Public Key and Secret
   Key.
4. Run `npm install` in this folder (only needed once, or again after
   deleting node_modules).

Then run any script with:
    node --env-file=.env <script-name>.mjs

FILES
-----

build-scores-export.mjs
    The main script. Pulls every real conversation trace (job_chat,
    workflow_chat, global_chat, anthropic.chat - explicitly excluding
    "Execute evaluator" and "dataset-run-item" traces, which are automated
    judge/testing runs, not real user conversations), then pulls every
    evaluator score in the project and matches each one back to the real
    conversation it belongs to - including scores generated by batched or
    manual dataset experiments, which link back to the original
    conversation through a chain of dataset items rather than directly.
    Takes a few minutes to run because some of Langfuse's endpoints are
    slow/flaky when pulling full conversation text; it retries
    automatically and logs progress as it goes.

    Outputs:
    - langfuse_scores_by_observation.csv
        One row per scored step (observation) within a conversation. If a
        conversation has multiple separately-scored steps, it appears as
        multiple rows. Conversations with zero scores still get one row
        (blank observationId, blank score columns).
    - langfuse_scores_by_trace.csv
        One row per conversation, with all of that conversation's scores
        collapsed onto a single row (one column per evaluator).

    Both files share the same base columns (traceId, timestamp, sessionId,
    userId, name, latencySeconds, totalCostUsd, input, output) plus two
    columns per evaluator found in the data: <evaluator name> (the score
    or category) and <evaluator name>_comment (the judge's reasoning).

these scripts make graphs (not very well)/observation-counts-per-trace.mjs
    A separate, lighter-weight script, kept in its own subfolder along with
    its outputs since it exists to feed charts. Answers "how many actual
    steps (observations) does each conversation have?" - regardless of
    whether any of those steps were ever scored by an evaluator. Also
    computes some weekly rollups from that data. Much faster than
    build-scores-export.mjs because it only asks Langfuse for
    structural/count data, never the actual conversation text.

    Outputs (in the same subfolder):
    - observation-counts-per-trace.json
        Raw list: one entry per real conversation trace, with its date and
        its true observation count.
    - observation-counts-weekly.json
        For each week, how many traces had exactly N observations (N = 0
        upwards).
    - observation-counts-weekly-stats.json
        For each week: min / 25th percentile / median / 75th percentile /
        max / mean observation count.
    - traces-per-week.json
        Simple count of how many real conversation traces started each
        week.
    - observations-per-week-by-name.json
        Total observation count per week, broken down by conversation type
        (job_chat / workflow_chat / global_chat / anthropic.chat).
    Note: the actual chart-drawing code (Chart.js/HTML) that turned these
    JSON files into graphs was never saved as a file - it only exists as
    visualisations shown inline in the chat that produced this project.



human-review-queue-old.mjs
    An annotation-queue builder built for reviewing newly backfilled/updated evaluator prompts (e.g.
    "Code quality judge v5", "Workflow quality judge v3", "General openfn
    quality judge v3") rather than general triage. Looks at the last 7
    days and selects 20 observations total:
      - 10 job_chat: the 5 lowest- and 5 highest-scoring on
        "Code quality judge v5"
      - 5 workflow_chat and 5 global_chat (selected independently, same
        rule for each): lowest and highest "Workflow quality judge v3",
        lowest and highest "General openfn quality judge v3", plus a
        second-lowest "Workflow quality judge v3" pick
    If a pick collides with one already selected (e.g. the same
    observation is both the lowest and second-lowest on a metric), it
    backfills with the next-most-extreme candidate so you still get the
    full count. Creates a new queue named "Human review of all three
    evaluator judges" with nine specific pre-existing score configs
    attached, in a specific order (the order is sent correctly via the
    API - if the Langfuse UI displays them in a different order, that's
    the UI re-sorting for display, not a bug in this script; confirmed by
    reading the queue back via the API after creating it).

code-judge-human-review-queue.mjs
    A narrower successor to human-review-queue-old.mjs, scoped to just one
    evaluator: every observation that has a "Code quality judge v5" score
    in the last 7 days (no restriction by trace type needed - that score
    only ever attaches to job_chat observations anyway). Selection is 5
    lowest-scoring + 5 highest-scoring + 10 more picked at random from
    whatever's left in the pool (up to 20 total; fewer if the pool is
    smaller than that). Creates a new queue named "Code judge human
    review queue (<date range>)" - e.g. "...(2026-07-06 to 2026-07-13)" -
    with the date range baked into the name so re-running this weekly
    never collides with a previous run's queue (unlike weekly-review-
    queue.mjs and human-review-queue-old.mjs, which reuse a fixed name and
    will error with "A queue with this name already exists" if the
    previous one hasn't been deleted first). Five score configs are
    attached, in this order: "Was agent response reasonable", "Code
    quality judge review", "Code style judge review", "Escalate for
    deeper review", "Comment". Note "Comment" (singular) is a real,
    distinct score config from "Comments" (plural, used by the other
    scripts) - confirmed both exist separately, this isn't a typo.

summarize-evaluator-comments.mjs
    Uses the Anthropic API (not just Langfuse) to summarise what
    evaluators are actually saying in their comments, per evaluator, for
    the last 7 days. Only looks at source=EVAL scores (i.e. automated
    judge comments - human-review annotations from the queues above are
    explicitly excluded) that are non-perfect and have a comment. Exact-
    duplicate comments are collapsed into one entry with an occurrence
    count before being sent to Claude, so recurring boilerplate isn't
    repeated in the prompt. For each evaluator, asks Claude to cluster
    the comments into named issue themes and write one summary paragraph.

    Outputs go in the "Evaluator summaries" subfolder (created
    automatically if missing). Filenames include the 7-day date range
    covered, e.g. 2026-07-02_to_2026-07-09, so different weeks' runs
    don't overwrite each other:
    - Evaluator summaries/evaluator-comment-summaries-<date range>.txt
        Plain text, one paragraph per evaluator - the actual deliverable.
    - Evaluator summaries/chart-<evaluator-name-slugified>-<date range>.svg
        (one per evaluator) - a horizontal bar chart of how often each
        tagged issue occurred, rendered as a hand-built SVG (no canvas/
        image-rendering dependency - just a plain text file you can open
        in any browser).

    Costs real money to run (Anthropic API calls, roughly one per
    evaluator with qualifying comments) - unlike every other script in
    this folder, which only reads from Langfuse for free.

.env / .env.example
    .env holds your real API keys and is never committed to git (see
    .gitignore): LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY for Langfuse,
    and ANTHROPIC_API_KEY (sk-ant-api03-...) for summarize-evaluator-
    comments.mjs. Note Anthropic only has ONE key (no public/secret pair
    like Langfuse) - don't add an "Anthropic public key" line, there's no
    such thing. .env.example is the template with placeholder values.

package.json / package-lock.json
    Standard Node project files. Dependencies: @langfuse/client (the
    official Langfuse JS/TS SDK, v5) and @anthropic-ai/sdk (official
    Anthropic SDK, used only by summarize-evaluator-comments.mjs).

WHAT WE LEARNED ALONG THE WAY (worth knowing before changing anything)
-----------------------------------------------------------------------
- Langfuse's "list traces with full content" endpoint times out fairly
  often when pulling many records with input/output text at once - both
  scripts use small page sizes (10) with retries for anything that needs
  full content, and much larger page sizes (100) for anything that only
  needs lightweight/structural fields.
- "Execute evaluator: ..." traces and "dataset-run-item-..." traces are NOT
  real user conversations - they're the judge model being invoked (live,
  and via manual/batched dataset experiments respectively). Their outputs
  and scores get attributed back to the real conversation instead of
  being kept as their own rows.
- A conversation's evaluator scores are attached directly to the real
  trace/observation (trace.scores), whether the scoring happened live or
  via a batched dataset experiment - you don't need to go hunting through
  the judge traces themselves to find a conversation's scores.
- Langfuse's live "evaluation rules" (the thing that auto-scores new
  traces) are explicitly documented as live-ingestion only - they cannot
  backfill/re-run against existing historical traces. There's no
  supported API for that; it has to be done by hand in the UI (or,
  cautiously, by calling the LLM yourself and writing scores back with
  langfuse.score.create - see the conversation history if this needs
  automating later).
- The Anthropic API can return an unrequested "thinking" content block as
  content[0], pushing your actual text response to content[1] or later -
  don't assume content[0].text; find the block by type instead. Pass
  thinking: { type: "disabled" } to avoid this (and the token cost of
  thinking eating into your max_tokens budget, which caused silent
  truncation the first time this happened here).
- Even with an explicit "respond with ONLY valid JSON" instruction,
  Claude sometimes wraps its answer in a ```json ... ``` code fence, and
  occasionally produces a small JSON syntax slip (e.g. one stray
  character). summarize-evaluator-comments.mjs strips code fences and
  retries once on a parse failure to handle this.