Grafana not pulling all entries compared to Google Logs

[randmark]

This is a Grafana SE putting in the issue after talking with the Google team for the Chicago Merchantile Exchange Group (CMEG).

CMEG is running queries and the count that has come back from larger data sets (for example 100k lines) doesn't match the count that we're getting when we click on the "run query in Google Logs" box. This has been repeatable in CME's environment. We can provide you with more data as needed.

[xiangshen-dk]

Thanks for raising the issue. I was able to reproduce the result for some of my queries. I have reached out to the Cloud Logging engineering team and received the following response:

Generally, the counts are reliable for small counts, but it's not 100% accurate when there's a lot of data (>100M).

If you are querying the latest logs, logs may be ingested out-of-order or late, and the count may be increasing over time. When you download the logs, it issues a new request which may fetch more logs that may have arrived late.
For logs further away in the past, you should get stable counts for small number of logs. Let me know if you can reproduce this consistently on small counts.

We do have an issue with counting very large number of logs (>100M). We do mention this in our docs: