fix(analytics): stop counting bot User rows as humans - #761
Merged
Conversation
HUMAN_FILTER keyed only on `botMetadata.agentName` absence, via an $or that
also passed anything with no botMetadata object at all. It never checked
`isBot`. Bot rows written by the gateway bridge, the summarizer and several
openclaw install paths carry botMetadata WITHOUT an agentName key, so they
satisfied the second clause and were counted as humans.
On the dev instance that is 8 rows of 85 (clawdbot-bridge, commonly-summarizer,
four openclaw-inst-*, socialpulse-adasdsad): totalUsers / DAU / WAU / signups
inflated ~10%, and every funnel rate deflated, because a bot in the
denominator can never convert. Corrected human count is 77.
Three sites shared the flaw and are now one definition:
- the /funnel route inlined a duplicate of the filter
- HUMAN_FILTER itself (used by /usage signups, DAU, WAU, totalUsers)
- the botIds query feeding "distinct human posters", which is the same
criterion inverted and so missed the same 8 rows
HUMAN_FILTER and BOT_FILTER are now exact logical complements, hoisted above
first use, with a test pinning that property — if they drift, a row is either
counted twice or dropped from both.
Nothing else in the dashboard was wrong: messages read PostgreSQL, and the JS
dayKey and the SQL UTC day-cast agree, so no days are silently dropped.
The funnel fixture keyed "is this the bot query?" on mere presence of the
botMetadata.agentName field. HUMAN_FILTER now names that field too, so the
signup query took the bot branch and returned docs with no createdAt, throwing
on new Date(undefined).toISOString(). Retargeted to the real discriminator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RWkf1SuYRHosxnPqsXwaVa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
HUMAN_FILTERinbackend/routes/admin/analytics.tskeyed only onbotMetadata.agentNameabsence, through an$orthat also passed anything with nobotMetadataobject at all. It never checkedisBot:Not every bot User row carries
agentName. Rows written by the gateway bridge, the summarizer, and several openclaw install paths setbotMetadatawithout it, so they satisfied the second clause and were counted as humans.On the dev instance that is 8 rows out of 85 —
clawdbot-bridge,commonly-summarizer, fouropenclaw-inst-*,socialpulse-adasdsad. Effects:totalUsers,DAU,WAUand daily signups inflated by ~10%Corrected human count is 77, and the activation rate moves up accordingly.
Three sites, one definition
The flaw appeared three times, because the shared constant was declared below the funnel route that needed it:
/funnelHUMAN_FILTER/usagesignups, DAU, WAU, totalUsersbotIdsHUMAN_FILTERandBOT_FILTERare now exact logical complements, hoisted above first use, with a test pinning that property — if they ever drift, a row is either counted twice or dropped from both.What was NOT wrong
Checked while verifying, so nobody re-audits it: messages read PostgreSQL (the Mongo path is correctly flagged in-file as an undercount), the JS
dayKeyand the SQLAT TIME ZONE 'UTC'day-cast agree so no days are silently dropped by theif (row)merge, andlastActiveis populated on all 77 humans.Tests
New
admin.analytics.human-filter.test.jsruns against a real in-memory Mongo, so filter semantics are under test rather than a mock's recorded call args. It seeds a human, a classic agent row, a bot withbotMetadatabut noagentName(the leak), and a bot marked only byisBot— then asserts one human out of four acrosstotalUsers, DAU/WAU, and signup cohorts, plus the complement property.One fixture change in the existing suite: it decided "is this the bot query?" from the mere presence of the
botMetadata.agentNamefield.HUMAN_FILTERnow names that field too ($exists: false), so the signup query took the bot branch and returned docs with nocreatedAt—new Date(undefined).toISOString()threw and the endpoint 500'd. Retargeted to the actual discriminator ($or).🤖 Generated with Claude Code
https://claude.ai/code/session_01RWkf1SuYRHosxnPqsXwaVa