Skip to content

[all] make otel sync gauge emit last recorded metrics in every export interval#2127

Merged
m-nagarajan merged 1 commit into
linkedin:mainfrom
m-nagarajan:fixSyncGaugeExport
Sep 12, 2025
Merged

[all] make otel sync gauge emit last recorded metrics in every export interval#2127
m-nagarajan merged 1 commit into
linkedin:mainfrom
m-nagarajan:fixSyncGaugeExport

Conversation

@m-nagarajan

@m-nagarajan m-nagarajan commented Sep 11, 2025

Copy link
Copy Markdown
Contributor

Problem Statement

In AggregationTemporalitySelector#deltaPreferred() : GAUGE falls back to default i,e DELTA, which results in no data getting exported if a synchronousGAUGE instrument is not recorded in that export interval, resulting in patches of data only whenever the metric is recorded.

Solution

Raised an issue with OSS OTel open-telemetry/opentelemetry-java#7634 and it was concluded that its intentionally kept that way by design as an instrument should not emit old recorded data unless its recorded in that export interval.
This PR makes GAUGE to be CUMULATIVE based on the below config which is enabled by default.
otel.venice.export.last.recorded.value.for.synchronous.gauge

Code changes

  • Added new code behind a config. If so list the config names and their default values in the PR description.
  • Introduced new log lines.
    • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • Code has no race conditions or thread safety issues.
  • Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • New unit tests added.
  • New integration tests added.
  • Modified or extended existing tests.
  • Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • No. You can skip the rest of this section.
  • Yes. Clearly explain the behavior change and its impact.
    Otel Sync Gauge metrics will always export the last recorded value unless this config is disabled.

@m-nagarajan m-nagarajan merged commit 4cb1da2 into linkedin:main Sep 12, 2025
50 checks passed
arjun4084346 pushed a commit to arjun4084346/venice that referenced this pull request Dec 9, 2025
…rt interval (linkedin#2127)

Make GAUGE to be CUMULATIVE based on the `otel.venice.export.last.recorded.value.for.synchronous.gauge` config which is enabled by default to make sure the last recorded data in a synchronous gauge is always exported regardless of whether it was recorded in that interval or not.
sushantmane pushed a commit to sushantmane/venice that referenced this pull request Jan 30, 2026
…rt interval (linkedin#2127)

Make GAUGE to be CUMULATIVE based on the `otel.venice.export.last.recorded.value.for.synchronous.gauge` config which is enabled by default to make sure the last recorded data in a synchronous gauge is always exported regardless of whether it was recorded in that interval or not.
misyel pushed a commit to misyel/venice that referenced this pull request Feb 2, 2026
…rt interval (linkedin#2127)

Make GAUGE to be CUMULATIVE based on the `otel.venice.export.last.recorded.value.for.synchronous.gauge` config which is enabled by default to make sure the last recorded data in a synchronous gauge is always exported regardless of whether it was recorded in that interval or not.
sushantmane pushed a commit to sushantmane/venice that referenced this pull request Feb 8, 2026
…rt interval (linkedin#2127)

Make GAUGE to be CUMULATIVE based on the `otel.venice.export.last.recorded.value.for.synchronous.gauge` config which is enabled by default to make sure the last recorded data in a synchronous gauge is always exported regardless of whether it was recorded in that interval or not.
misyel pushed a commit to misyel/venice that referenced this pull request Feb 17, 2026
…rt interval (linkedin#2127)

Make GAUGE to be CUMULATIVE based on the `otel.venice.export.last.recorded.value.for.synchronous.gauge` config which is enabled by default to make sure the last recorded data in a synchronous gauge is always exported regardless of whether it was recorded in that interval or not.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants