feat(access-logs): make GCP log buffer size configurable and count drops#271
Merged
Conversation
added 2 commits
July 1, 2026 17:06
The Google Cloud Logging client buffers pending entries in memory using a non-blocking bundler whose default BufferedByteLimit is 1GiB. With full payload/variable access logging enabled, a slow Cloud Logging flush lets the buffer grow until the container crosses its memory limit and is OOMKilled before the bundler's own overflow-drop can kick in. Make the buffer size configurable via `log_buffer_max_size_mb` (default 1000, preserving prior behaviour) and add a `gcp_dropped_total` counter plus error logging on the async OnError path so overflow drops are observable instead of silent.
rnikkels
reviewed
Jul 1, 2026
ldebruijn
approved these changes
Jul 1, 2026
wmuizelaar
reviewed
Jul 2, 2026
| } | ||
|
|
||
| logger := client.Logger(logName) | ||
| logger := client.Logger(logName, logging.BufferedByteLimit(cfg.BufferedByteLimit<<20)) |
Contributor
There was a problem hiding this comment.
I think it's rather confusing that logging.BufferedByteLimit is in byes, and cfg.BufferedByteLimit is apparently in megabytes. So I would suggest to either implement it in bytes in cfg as well, or rename the variable to BufferedMegaByteLimit or something to make sure people don't make expensive/risky mistakes here.
Collaborator
Author
There was a problem hiding this comment.
the yaml property does explicitly state: log_buffer_max_size_mb so i think we should be good here, its the only place where the property can be defined
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Google Cloud Logging client buffers pending log entries in memory . Its default
BufferedByteLimitis 1GiB. When the Cloud Logging flush slows down, the buffer grows unbounded. Drops on the GCP path were also completely silent (no metric, no log).Changes
log_buffer_max_size_mb(default1000, i.e. ~1GiB — preserves current behaviour). Wired intologging.BufferedByteLimit. Operators can now bound it below the container memory limit so the bundler drops entries instead of OOMing.graphql_protect_access_logging_gcp_dropped_totalcounter — increments when the bundler returnsErrOverflow, so drops are observable.OnError— non-overflow async errors now incrementgcp_errors_totaland emit a warning, instead of being swallowed.Tests
TestDefaultConfig— asserts the buffer default stays1000(behaviour-neutral upgrade).TestGoogleCloudConfig_YAMLBinding— guards the YAML tag name against typos.TestGoogleCloudConfig_YAMLOmittedKeepsDefault— omitted key preserves the default when unmarshalling overDefaultConfig().go vetclean, full package test suite passes.