Don't mutate an action's input map after it has been executed - #30693
Draft
fmeum wants to merge 1 commit into
Draft
Don't mutate an action's input map after it has been executed#30693fmeum wants to merge 1 commit into
fmeum wants to merge 1 commit into
Conversation
`ActionInputMap` is thread-compatible, but the action filesystem reads the one
belonging to the action being executed, and it is read from other threads after
the action has finished: build events referencing its `Path`s are uploaded
asynchronously, and with `--remote_cache_async` (on by default) the action
result is uploaded on a background thread.
`ActionExecutionFunction#addDiscoveredInputs` mutated that map from the Skyframe
thread after execution. Since `putIfAbsent` links an entry into its hash bucket
before filling in the corresponding slot, a concurrent reader can observe a
half-initialized entry:
java.lang.NullPointerException: Cannot invoke "String.hashCode()" because "this.paths[index]" is null
at com.google.devtools.build.lib.actions.ActionInputMap.getIndex(ActionInputMap.java:196)
at com.google.devtools.build.lib.actions.ActionInputMap.getInput(ActionInputMap.java:343)
...
at com.google.devtools.build.lib.remote.RemoteActionFileSystem.statInternal(RemoteActionFileSystem.java:686)
at com.google.devtools.build.lib.remote.RemoteActionFileSystem.getFastDigest(RemoteActionFileSystem.java:467)
...
at com.google.devtools.build.lib.remote.ByteStreamBuildEventArtifactUploader.readPathMetadata(ByteStreamBuildEventArtifactUploader.java:203)
Keep inputs that are only discovered after execution in a separate map, so that
the map the action filesystem reads is immutable from the moment the action
starts executing. The action cache update, the only consumer of those inputs,
gets a metadata provider that covers both maps.
Fixes bazelbuild#30683.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ActionInputMapis thread-compatible, but the action filesystem reads the map belonging to the action being executed, and that map is read from other threads after the action has finished:Paths are uploaded asynchronously byByteStreamBuildEventArtifactUploader;--remote_cache_async(on by default),RemoteExecutionService#doUploadOutputsruns on a background thread and digests the action's stdout/stderr, which live under the exec root and are resolved through the action filesystem.ActionExecutionFunction#addDiscoveredInputsmutated that map from the Skyframe thread after execution, for actions that discover new inputs while running (CppCompileActionvia its.dfile,LtoBackendAction,StarlarkActionwith an unused inputs list). SinceputIfAbsentlinks a new entry into its hash bucket before filling in the corresponding slot, a concurrent reader can observe a half-initialized entry:Inputs that are only discovered after execution now go into a separate map, so the map the action filesystem reads is immutable from the moment the action starts executing. The pre-execution call is unchanged — nothing else can observe the filesystem yet at that point. The action cache update, the only consumer of the late-discovered metadata, receives a provider that covers both maps.
Motivation
Fixes a flaky internal-error crash. The new test in
TimestampBuilderTestfails without the change (the input metadata provider handed to the action reports the late-discovered input afterwards) and additionally asserts that the input is still tracked by the action cache.Fixes #30683.
Build API Changes
No
Checklist
Release Notes
RELNOTES: Fixed a rare crash with
NullPointerExceptioninActionInputMap.getIndexwhen an action discovers additional inputs while executing.