Skip to content

fix(analytics): stop counting bot User rows as humans - #761

Merged
samxu01 merged 1 commit into
mainfrom
fix/analytics-human-filter-isbot
Jul 27, 2026
Merged

fix(analytics): stop counting bot User rows as humans#761
samxu01 merged 1 commit into
mainfrom
fix/analytics-human-filter-isbot

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

The bug

HUMAN_FILTER in backend/routes/admin/analytics.ts keyed only on botMetadata.agentName absence, through an $or that also passed anything with no botMetadata object at all. It never checked isBot:

$or: [
  { botMetadata: { $exists: false } },
  { 'botMetadata.agentName': { $exists: false } },   // ← true for bots without an agentName key
]

Not every bot User row carries agentName. Rows written by the gateway bridge, the summarizer, and several openclaw install paths set botMetadata without it, so they satisfied the second clause and were counted as humans.

On the dev instance that is 8 rows out of 85clawdbot-bridge, commonly-summarizer, four openclaw-inst-*, socialpulse-adasdsad. Effects:

  • totalUsers, DAU, WAU and daily signups inflated by ~10%
  • every funnel rate deflated, because a bot sitting in the denominator can never attach an agent or send a message

Corrected human count is 77, and the activation rate moves up accordingly.

Three sites, one definition

The flaw appeared three times, because the shared constant was declared below the funnel route that needed it:

Site Was
/funnel inlined its own copy of the filter
HUMAN_FILTER used by /usage signups, DAU, WAU, totalUsers
botIds the same criterion inverted, so it missed the same 8 rows and let them count as human posters too

HUMAN_FILTER and BOT_FILTER are now exact logical complements, hoisted above first use, with a test pinning that property — if they ever drift, a row is either counted twice or dropped from both.

What was NOT wrong

Checked while verifying, so nobody re-audits it: messages read PostgreSQL (the Mongo path is correctly flagged in-file as an undercount), the JS dayKey and the SQL AT TIME ZONE 'UTC' day-cast agree so no days are silently dropped by the if (row) merge, and lastActive is populated on all 77 humans.

Tests

New admin.analytics.human-filter.test.js runs against a real in-memory Mongo, so filter semantics are under test rather than a mock's recorded call args. It seeds a human, a classic agent row, a bot with botMetadata but no agentName (the leak), and a bot marked only by isBot — then asserts one human out of four across totalUsers, DAU/WAU, and signup cohorts, plus the complement property.

Test Suites: 2 passed, 2 total
Tests:       12 passed, 12 total

One fixture change in the existing suite: it decided "is this the bot query?" from the mere presence of the botMetadata.agentName field. HUMAN_FILTER now names that field too ($exists: false), so the signup query took the bot branch and returned docs with no createdAtnew Date(undefined).toISOString() threw and the endpoint 500'd. Retargeted to the actual discriminator ($or).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RWkf1SuYRHosxnPqsXwaVa

HUMAN_FILTER keyed only on `botMetadata.agentName` absence, via an $or that
also passed anything with no botMetadata object at all. It never checked
`isBot`. Bot rows written by the gateway bridge, the summarizer and several
openclaw install paths carry botMetadata WITHOUT an agentName key, so they
satisfied the second clause and were counted as humans.

On the dev instance that is 8 rows of 85 (clawdbot-bridge, commonly-summarizer,
four openclaw-inst-*, socialpulse-adasdsad): totalUsers / DAU / WAU / signups
inflated ~10%, and every funnel rate deflated, because a bot in the
denominator can never convert. Corrected human count is 77.

Three sites shared the flaw and are now one definition:
  - the /funnel route inlined a duplicate of the filter
  - HUMAN_FILTER itself (used by /usage signups, DAU, WAU, totalUsers)
  - the botIds query feeding "distinct human posters", which is the same
    criterion inverted and so missed the same 8 rows

HUMAN_FILTER and BOT_FILTER are now exact logical complements, hoisted above
first use, with a test pinning that property — if they drift, a row is either
counted twice or dropped from both.

Nothing else in the dashboard was wrong: messages read PostgreSQL, and the JS
dayKey and the SQL UTC day-cast agree, so no days are silently dropped.

The funnel fixture keyed "is this the bot query?" on mere presence of the
botMetadata.agentName field. HUMAN_FILTER now names that field too, so the
signup query took the bot branch and returned docs with no createdAt, throwing
on new Date(undefined).toISOString(). Retargeted to the real discriminator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWkf1SuYRHosxnPqsXwaVa
@samxu01
samxu01 merged commit 49f0cd8 into main Jul 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants