Usage estimator performance

[jdolle]

Background

I've noticed several spikes in IOWait duration in Clickhouse. This could be caused by a number of things, but one of them is if the read amount is excessive.

Usage estimator polls for usage data using our operations_hourly table. This table has an unoptimized order for querying a target's operation count without any filters.

Description

This adjusts the operation to use the new table that is ideal for this query.

This results in significantly fewer rows being read and a much faster response.

Before:

After:

[gemini-code-assist]

Code Review

This change introduces conditional querying of the optimized operations_by_target_hourly table based on a configured creation date. To ensure correctness, the Zod schema default for OPERATIONS_BY_TARGET_TABLE_CREATED_AT should be changed from new Date() to a deterministic fixed date like new Date(0). Additionally, the year in the configured date 2027-06-11T00:00:00Z should be corrected to prevent queries from falling back to the unoptimized table.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[github-actions]

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package	Version	Info
hive	11.5.0-alpha-20260709202147-674f3306da5351a67a6d7e3cb53d3ce2264401c7	npm ↗︎ unpkg ↗︎

[github-actions]

🐋 This PR was built and pushed to the following Docker images:

Targets: build

Platforms: linux/amd64

Image Tags: 674f3306da5351a67a6d7e3cb53d3ce2264401c7, 674f330

[kamilkisiela]

I think we could use monthly_overview instead.

The query you're optimizing is from estimateOperationsForAllTargets and it's called only by tRPC's estimateOperationsForAllTargets procedure and finally by:

console/packages/services/commerce/src/rate-limit/limiter.ts

Line 80 in 27a58e5

usageEstimator.estimateOperationsForAllTargets(timeWindow),

The time window there looks like this:

console/packages/services/commerce/src/rate-limit/limiter.ts

Lines 70 to 73 in 27a58e5

	const timeWindow = {
	startTime: startOfMonth(now),
	endTime: endOfMonth(now),
	};

And we sum the targets by organization:

console/packages/services/commerce/src/rate-limit/limiter.ts

Line 118 in 27a58e5

orgRecord.operations.current += operations[target] || 0;

so yeah, we could do:

SELECT
    sum(total) as total,
    organization
FROM
    monthly_overview PREWHERE date = '2026-07-01'
GROUP BY
    organization
# Elapsed: 0.053s Read: 2,650 rows (38.85 KB)

[jdolle]

Nice. Even better.

[jdolle]

We should always use UTC to avoid timezone issues.