Context
I recently discovered that it seems that the Datadog Agent does not actually enforce a bin limit any longer on its sketch structures used for distributions coming over DogStatsD. This shows up in terms of how we now handle distributions in ADP which involves clamping the incoming sample rate to avoid trying to add more samples to a DDSketch than would be allowed by the maximum theoretical count based on the configured bin limit and integral types.
While the Agent will happily track more samples than the theoretical limit, we do not in ADP. This is only visible in some internal deployments where distributions with very high sample rates are present.
The main question is: why do we not seem to exercise this code path in correctness tests given that we're using lading, which often generates payload in a very fuzzer-like fashion. We should investigate if something on the lading side is preventing sufficiently high sample rates from being generated in these payloads, or if something else is at play.
Context
I recently discovered that it seems that the Datadog Agent does not actually enforce a bin limit any longer on its sketch structures used for distributions coming over DogStatsD. This shows up in terms of how we now handle distributions in ADP which involves clamping the incoming sample rate to avoid trying to add more samples to a
DDSketchthan would be allowed by the maximum theoretical count based on the configured bin limit and integral types.While the Agent will happily track more samples than the theoretical limit, we do not in ADP. This is only visible in some internal deployments where distributions with very high sample rates are present.
The main question is: why do we not seem to exercise this code path in correctness tests given that we're using
lading, which often generates payload in a very fuzzer-like fashion. We should investigate if something on theladingside is preventing sufficiently high sample rates from being generated in these payloads, or if something else is at play.