POR Record Timeseries Query Triggers 500 Error

[msweier]

Describe the bug
When querying a large data set like:

curl -X 'GET' \  'https://CWBITEST/cwms-data/timeseries?name=Baldhill_Dam.Flow-Out.Inst.~15Minutes.0.best&office=MVP&begin=1945-09-03T16%3A11%3A00.307Z&end=2025-09-03T16%3A11%3A00.307Z' \  -H 'accept: application/json;version=2'

a 500 error is typically returned. Sometimes it will complete though. Similar behavior with local production CDA instance. @MikeNeilson suggested it related to the volume of data and the initial indexing query may be timing out.

CDA Version (please complete the following information):
2025.09.03-test

[rma-rripken]

Part of the ts query gets a total so that you know where you are in the full set of results as you page thru the data. I wonder if that part of the query is just too slow to run across the full POR.

[msweier]

I'm still getting an error on test w/ this query (some other POR timeseries will work though).

Failed to fetch.
Possible Reasons:

CORS
Network Failure
URL scheme must be "http" or "https" for CORS request.

On the local server it worked fine.

[rma-psmorris]

From 2026-01-27 CDA Meeting: Still an issue, cache or job queue. Ryan R suggests

1. adding a timer or metric around item count part of query to see if that is the culprit. If count is > 100k or some limit, then short circuit the count. Run in a thread and if ready by time needed, great otherwise don't return a count.
2. Mike N adds: Adding in a time in the API handler to avoid returning 500 and issue a better message on the error occurring (taking too long).
3. Another alternative is to drop total from the return, this would be an API change, so limited to v2.
4. Total could also be a representative value from extents.

Pursue steps 1 and 2.

[krowvin]

Possibly resolved with #1692

Will want to confirm 500 does not exist for versioned/other TS that do not short circuit

Followup to see if anything has been discussed since Peter's comment above in January

[rma-rripken]

My pr #1700 added metrics to the ts retrieve so we should get some idea of whether it times out on the count or on the data retrieve. I also added a timer in the Controller that should give a better error message.

When the changes make it into test perhaps someone could try and duplicate Mitch's original request and see how the changes are working.

Its very possible #1692 will resolve the issue as well.