Support dynamic query limit configuration#3532
Open
lbschanno wants to merge 15 commits into
Open
Conversation
Support the ability to dynamically update concurrent query limits while
the webservers are running. Add the class `QueryLimitReloader`. A
`QueryLimitReloader` will operate within the Zookeeper namespace
`QueryLimitConfig` to listen for events that should trigger a reload. A
reload will be triggered by the following:
- The creation or modification of the node `/path` with non-empty data.
- The creation, modification, or deletion of the node `/trigger`.
Upon a triggering event, the reloader will attempt to load a valid
`QueryLimitConfiguration` and supply it to all listeners, such as the
one registered by `QueryLimiter`. The reloader will attempt to
deserialize a `QueryLimiterConfiguration` from a file. It will look for
the filepath in the node `/path`. The target file may be JSON, XML, or
YAML, and may have the URI schemes `http:`, `https:`, `hdfs:`, `file:`,
or none.
After a reload attempt, the following nodes will be updated with the
status of the attempt:
- `/attempts/<serverIpAddress>/cause`: The triggering event for the
reload. This may be `PATH_NODE_CREATED`, `PATH_NODE_MODIFIED`,
`TRIGGER_NODE_CREATED`, `TRIGGER_NODE_MODIFIED`, or
`TRIGGER_NODE_DELETED`.
- `/attempts/<serverIpAddress>/status`: The final status of the reload
attempt. It may be one of the following:
- `SUCCESS`: A valid `QueryLimitConfiguration` was loaded and supplied
to all listeners.
- `LISTENER_ERROR`: A valid `QueryLimitConfiguration` was loaded, but
an exception was thrown when supplying it to one or more listeners.
- `RELOAD_ERROR`: A valid `QueryLimitConfiguration` could not be
loaded.
- `/attempts/<serverIpAddress>/time`: The time of the reload attempt in
ISO-8601 format.
- `/attempts/<serverIpAddress>/errors`: A node whose children will
contain brief descriptions of the errors that occurred. Exists only if
any errors occurred. The children's paths will have the format
`error_X` where X is a value from `0...N` where N is one less than the
total errors that were captured. The full stacktrace for the errors
will be captured in the logs.
When the `QueryLimiter` is supplied with a new configuration, it will
update its internal configuration, and rebuild its internal limit
providers.
Closes #3511
ivakegg
reviewed
May 14, 2026
| @Override | ||
| public void validate(Object object) { | ||
| Preconditions.checkArgument((object instanceof QueryLimitConfiguration), "Object must be an instance of " + QueryLimitConfiguration.class.getName()); | ||
| QueryLimitConfigurationValidationUtils.validate((QueryLimitConfiguration) object); |
Collaborator
There was a problem hiding this comment.
Why have a utils class if this is the only class to use it ?
Collaborator
Author
There was a problem hiding this comment.
QueryLimitConfigurationValidationUtils is also used in QueryLimiter.updateConfiguration() when the argument validationRequired is true. This will get used if the initial QueryLimiter created by Spring has a configured QueryLimitConfiguration that needs to be validated.
The class QueryLimitConfigurationValidator allows us to hook in validation of any updated configurations within the configured ZkObjectPublisher and capture any violations in the feedback mechanism.
I was debating between having specific configurable object validators for ZkObjectPublisher instances, or perhaps having the validation occur when the configuration is supplied to the QueryLimiter's subscriber. I landed on the former to allow for extra flexibility, but I am open to other approaches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support the ability to dynamically update concurrent query limits while the webservers are running.
Add the class
ZkObjectPublisherthat provides the ability to trigger and publish updates of a configured class to all subscribers. See this README for full details.Integrate using a configured
ZkObjectPublisherwithin theQueryLimiterclass to update its internal configured limits whenever an update has been triggered.Closes #3511