Skip to content

[9.3.0] Re-sample the execution graph clock converter per command - #30679

Open
fmeum wants to merge 1 commit into
bazelbuild:release-9.3.0from
fmeum:cp-9.3.0-30594
Open

[9.3.0] Re-sample the execution graph clock converter per command#30679
fmeum wants to merge 1 commit into
bazelbuild:release-9.3.0from
fmeum:cp-9.3.0-30594

Conversation

@fmeum

@fmeum fmeum commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Description

ExecutionGraphModule built its NanosToMillisSinceEpochConverter in a field initializer:

private NanosToMillisSinceEpochConverter nanosToMillis =
    BlazeClock.createNanosToMillisSinceEpochConverter();

Modules are instantiated once per server, so that converter captures the offset between the monotonic and the wall clock as it stood at server startup, and is then reused for every command that server ever serves. Build it in beforeCommand instead, so each command converts through an offset sampled during that command.

Motivation

System.nanoTime() and System.currentTimeMillis() are backed by different operating system clocks, and the offset between them is not constant. NTP steps the wall clock when it corrects it. A virtual machine does so more dramatically: shortly after boot, and again whenever a snapshot is resumed — while the monotonic clock never steps and does not advance across the pause.

A long-lived server that survives a snapshot resume keeps converting with an offset captured before the snapshot, so every start_timestamp_millis it writes into the execution graph log afterwards is shifted by the entire time the snapshot spent on disk. Anything that reconstructs a timeline from the log — critical path, action concurrency, correlation against other traces on the machine — is wrong by that amount. On bare metal an NTP step produces the same corruption at a smaller magnitude.

Build API Changes

No

Checklist

  • I have added tests for the new use cases (if any).
  • I have updated the documentation (if applicable).

Release Notes

RELNOTES: None

Closes #30594.

PiperOrigin-RevId: 960312424
Change-Id: I3295137f08c41bc6d76593766e4a8386dd548fb5

(cherry picked from commit 13e656c)

9.3.0 adaptation: the release branch has not had master's test refactor that merged createModule()/initializeRoots() into a single setUp() and dropped the UUID argument from startLogging, so the module.resetNanosToMillis() call was added to createModule() (where module is guaranteed to be constructed) rather than to setUp(), and the existing startLogging(eventBus, UUID.randomUUID(), ...) call sites were left as they are. The production change is unmodified.

Closes #30601

…#30594)

### Description

`ExecutionGraphModule` built its `NanosToMillisSinceEpochConverter` in a *field initializer*:

```java
private NanosToMillisSinceEpochConverter nanosToMillis =
    BlazeClock.createNanosToMillisSinceEpochConverter();
```

Modules are instantiated once per server, so that converter captures the offset between the monotonic and the wall clock as it stood at server startup, and is then reused for every command that server ever serves. Build it in `beforeCommand` instead, so each command converts through an offset sampled during that command.

### Motivation

`System.nanoTime()` and `System.currentTimeMillis()` are backed by different operating system clocks, and the offset between them is not constant. NTP steps the wall clock when it corrects it. A virtual machine does so more dramatically: shortly after boot, and again whenever a snapshot is resumed — while the monotonic clock never steps and does not advance across the pause.

A long-lived server that survives a snapshot resume keeps converting with an offset captured before the snapshot, so every `start_timestamp_millis` it writes into the execution graph log afterwards is shifted by the entire time the snapshot spent on disk. Anything that reconstructs a timeline from the log — critical path, action concurrency, correlation against other traces on the machine — is wrong by that amount. On bare metal an NTP step produces the same corruption at a smaller magnitude.

### Build API Changes

No

### Checklist

- [ ] I have added tests for the new use cases (if any).
- [ ] I have updated the documentation (if applicable).

### Release Notes

RELNOTES: None

Closes bazelbuild#30594.

PiperOrigin-RevId: 960312424
Change-Id: I3295137f08c41bc6d76593766e4a8386dd548fb5

(cherry picked from commit 13e656c)
@fmeum
fmeum marked this pull request as ready for review August 12, 2026 09:41
@fmeum
fmeum requested a review from a team as a code owner August 12, 2026 09:41
@github-actions github-actions Bot added team-Rules-Server Issues for serverside rules included with Bazel awaiting-review PR is awaiting review from an assigned reviewer labels Aug 12, 2026
@iancha1992
iancha1992 requested a review from meisterT August 12, 2026 18:16
@iancha1992
iancha1992 enabled auto-merge August 12, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting-review PR is awaiting review from an assigned reviewer team-Rules-Server Issues for serverside rules included with Bazel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant