[9.3.0] Re-sample the execution graph clock converter per command - #30679
Open
fmeum wants to merge 1 commit into
Open
[9.3.0] Re-sample the execution graph clock converter per command#30679fmeum wants to merge 1 commit into
fmeum wants to merge 1 commit into
Conversation
…#30594) ### Description `ExecutionGraphModule` built its `NanosToMillisSinceEpochConverter` in a *field initializer*: ```java private NanosToMillisSinceEpochConverter nanosToMillis = BlazeClock.createNanosToMillisSinceEpochConverter(); ``` Modules are instantiated once per server, so that converter captures the offset between the monotonic and the wall clock as it stood at server startup, and is then reused for every command that server ever serves. Build it in `beforeCommand` instead, so each command converts through an offset sampled during that command. ### Motivation `System.nanoTime()` and `System.currentTimeMillis()` are backed by different operating system clocks, and the offset between them is not constant. NTP steps the wall clock when it corrects it. A virtual machine does so more dramatically: shortly after boot, and again whenever a snapshot is resumed — while the monotonic clock never steps and does not advance across the pause. A long-lived server that survives a snapshot resume keeps converting with an offset captured before the snapshot, so every `start_timestamp_millis` it writes into the execution graph log afterwards is shifted by the entire time the snapshot spent on disk. Anything that reconstructs a timeline from the log — critical path, action concurrency, correlation against other traces on the machine — is wrong by that amount. On bare metal an NTP step produces the same corruption at a smaller magnitude. ### Build API Changes No ### Checklist - [ ] I have added tests for the new use cases (if any). - [ ] I have updated the documentation (if applicable). ### Release Notes RELNOTES: None Closes bazelbuild#30594. PiperOrigin-RevId: 960312424 Change-Id: I3295137f08c41bc6d76593766e4a8386dd548fb5 (cherry picked from commit 13e656c)
fmeum
marked this pull request as ready for review
August 12, 2026 09:41
iancha1992
enabled auto-merge
August 12, 2026 18:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ExecutionGraphModulebuilt itsNanosToMillisSinceEpochConverterin a field initializer:Modules are instantiated once per server, so that converter captures the offset between the monotonic and the wall clock as it stood at server startup, and is then reused for every command that server ever serves. Build it in
beforeCommandinstead, so each command converts through an offset sampled during that command.Motivation
System.nanoTime()andSystem.currentTimeMillis()are backed by different operating system clocks, and the offset between them is not constant. NTP steps the wall clock when it corrects it. A virtual machine does so more dramatically: shortly after boot, and again whenever a snapshot is resumed — while the monotonic clock never steps and does not advance across the pause.A long-lived server that survives a snapshot resume keeps converting with an offset captured before the snapshot, so every
start_timestamp_millisit writes into the execution graph log afterwards is shifted by the entire time the snapshot spent on disk. Anything that reconstructs a timeline from the log — critical path, action concurrency, correlation against other traces on the machine — is wrong by that amount. On bare metal an NTP step produces the same corruption at a smaller magnitude.Build API Changes
No
Checklist
Release Notes
RELNOTES: None
Closes #30594.
PiperOrigin-RevId: 960312424
Change-Id: I3295137f08c41bc6d76593766e4a8386dd548fb5
(cherry picked from commit 13e656c)
9.3.0 adaptation: the release branch has not had master's test refactor that merged
createModule()/initializeRoots()into a singlesetUp()and dropped the UUID argument fromstartLogging, so themodule.resetNanosToMillis()call was added tocreateModule()(wheremoduleis guaranteed to be constructed) rather than tosetUp(), and the existingstartLogging(eventBus, UUID.randomUUID(), ...)call sites were left as they are. The production change is unmodified.Closes #30601