-
Notifications
You must be signed in to change notification settings - Fork 55
fix: provider state race #380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
fa0b999
fe60ffc
8f1228f
630ef55
6d70ac9
885c67f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| --- | ||
| id: appendix-e | ||
| title: "Appendix E: Migrations" | ||
| description: Migration guidance for breaking spec changes | ||
| sidebar_position: 6 | ||
| --- | ||
|
|
||
| # Appendix E: Migrations | ||
|
|
||
| This appendix provides non-normative guidance for provider authors and SDK authors on migrating to new or changed specification requirements. | ||
|
|
||
| ## Provider Status Ownership | ||
|
|
||
| ### Background | ||
|
|
||
| Prior to `v0.9.0`, provider status (e.g. `NOT_READY`, `READY`, `ERROR`) was managed by the SDK on behalf of the provider. | ||
| The SDK would set status and emit events after lifecycle methods (`initialize`, `shutdown`, `on context change`) returned. | ||
| This created a race condition in multi-threaded SDKs: the provider could change its own state (e.g. emit an error event from a background thread) in the window between the lifecycle method returning and the SDK writing its post-lifecycle status and emitting the corresponding event. | ||
| The result was incorrect event ordering and inconsistent status. | ||
|
|
||
| The spec now requires providers to own their status and emit events atomically with status transitions (see [provider status](./sections/02-providers.md#28-provider-status)). | ||
|
|
||
| ### For provider authors | ||
|
|
||
| Providers are now responsible for maintaining their own `status` and emitting events atomically with status transitions. | ||
|
|
||
| #### What to implement | ||
|
|
||
| - A `status` accessor returning the provider's current readiness: `NOT_READY`, `READY`, `STALE`, `ERROR`, or `FATAL` (plus `RECONCILING` in the static-context paradigm) | ||
| - `status` must be `NOT_READY` before `initialize` is called and after `shutdown` terminates | ||
| - `status` must be safe for concurrent access | ||
| - Status transitions and associated event emissions must be atomic from the perspective of external observers; set the status before emitting the corresponding event | ||
|
|
||
| #### The `StateManagingProvider` interface | ||
|
|
||
| To signal to the SDK that your provider manages its own status, implement an opt-in interface (or equivalent mechanism) defined by the SDK. | ||
| This interface should expose: | ||
|
|
||
| - A `status` accessor that returns the provider's current status | ||
| - A discriminant or marker (e.g. an additional interface, a boolean property, or a type-level tag) that allows the SDK to detect at registration time that the provider manages its own state | ||
|
|
||
| Providers that do not implement this interface will continue to have their status managed by the SDK. | ||
| This legacy behavior is deprecated and will be removed in the next major version. | ||
|
|
||
| ### For SDK authors | ||
|
|
||
| SDKs must detect whether a registered provider manages its own state and branch behavior accordingly. | ||
|
|
||
| #### Detecting state-managing providers | ||
|
|
||
| At registration time, check whether the provider implements the `StateManagingProvider` interface (or equivalent). | ||
| Store this as a flag on the internal provider wrapper for use during lifecycle calls and event handling. | ||
|
|
||
| #### SDK wrapper behavior | ||
|
|
||
| SDKs typically wrap registered providers in an internal adapter (e.g. a "provider wrapper" or "state manager") that mediates lifecycle calls and event forwarding. | ||
| The wrapper should branch based on whether the registered provider implements the state-managing interface. | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| A[Provider registered with SDK] --> B{Implements state-managing interface?} | ||
|
|
||
| B -- Yes --> C[SDK wrapper delegates status to provider] | ||
| C --> C1[initialize / shutdown / onContextChange: SDK skips state writes AND event emissions] | ||
| C --> C2[Provider events: SDK skips state writes] | ||
| C --> C3[status: reads from provider directly] | ||
|
|
||
| B -- No --> D[SDK wrapper manages state internally - legacy, deprecated] | ||
| D --> D1[initialize / shutdown / onContextChange: SDK sets state AND emits events after return] | ||
| D --> D2[Provider events: SDK updates state on emit] | ||
| D --> D3[status: reads from SDK wrapper] | ||
|
|
||
| C1 --> E[Provider-emitted events still propagate to registered handlers] | ||
| C2 --> E | ||
| C3 --> E | ||
| D1 --> E | ||
| D2 --> E | ||
| D3 --> E | ||
| ``` | ||
|
|
||
| #### What the SDK skips for state-managing providers | ||
|
|
||
| For providers that implement the state-managing interface, the SDK must not perform any of the following actions that it would normally perform for legacy providers: | ||
|
|
||
| - Setting status to `READY` after `initialize()` succeeds | ||
| - Setting status to `ERROR` or `FATAL` after `initialize()` fails | ||
| - Setting status to `NOT_READY` after `shutdown()` completes | ||
| - Emitting `PROVIDER_READY` or `PROVIDER_ERROR` events after `initialize()` | ||
| - Updating status when the provider emits events at runtime (the provider already set its own status atomically with the event) | ||
| - (Static-context paradigm only) Setting `RECONCILING` status, emitting `PROVIDER_RECONCILING`, setting `READY`/`ERROR` status, or emitting `PROVIDER_CONTEXT_CHANGED`/`PROVIDER_ERROR` during `on context change` handling | ||
|
|
||
| #### What the SDK still does for all providers | ||
|
|
||
| Regardless of whether the provider manages its own state, the SDK continues to: | ||
|
|
||
| - Call `initialize()`, `shutdown()`, and `on context change` lifecycle methods on the provider | ||
| - Forward provider-emitted events to registered domain and API-level event handlers | ||
| - Run late-attached handlers immediately if the provider is already in the associated state | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. major: I believe SDKs can't do this safely as this requires that subscription and emission of the current status is atomic (there should be no new events emitted between reading current status and running the handler) — and you can't guarantee that when another thread is emitting events (unless SDK implements event buffering which is cumbersome).
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree this is a real concern. With provider-owned state, I don't see how the SDK can safely guarantee atomic late-attached handler execution or short-circuit evaluation without adding significant complexity (e.g. shared locks between the SDK and provider, or subscription-awareness in the provider). That said, this is existing behavior that's important for usability; users need to be able to attach Your suggestion about an event queue is interesting; if the SDK serializes event dispatch through a queue, late-attached handler checks could go through the same queue, which might also address concerns about event handler interleaving. Maybe we need this even though it will be heavy?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've taken a different approach: #380 (comment)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Swift SDK is actually doing this events/subscription serialization with a lock in
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thinking about this more, I believe we're confusing "event observing" and "state observing" and this adds a lot of accidental complexity. Many languages have separate tools for a regular channel and a channel that remembers the last value:
The issue is that we want a weird mix of the two: we want it to remember the last state... except "not ready" status because it doesn't have a corresponding event to re-emit)... and don't remember "configuration changed" event (because it doesn't change status)... and "context changed" should be remapped to "ready" (but only if it's a replay and not an online observation). This leads to a couple of issues. The obvious one is that the behavior is convoluted and is complicated to implement. In MultiProvider, we get a mismatch between emitted events and actual provider state (the spec requires MultiProvider to re-emit all events from the underlying providers, but underlying provider going into fatal state does not mean that the whole MultiProvider does). And the last one is that waiting for I think a much cleaner API is to add "status observers" that remember the last status and are guaranteed to fire when the provider switches to the corresponding status (i.e. a successful reconciliation should invoke ready handler). And keep event observers without this last-value memory We can even keep this backward-compatible for SDKs that have subscription topics (JS, Go) by saying that An alternative take is that this issue only exists for languages that have these native tools for broadcast and last-state observers (swift, kotlin), so SDKs decided to expose native observers instead of named event handlers as the spec describes. So now these SDKs are fighting with synchronizing these subjects/flows. As they are already deviating from the spec, perhaps they can deviate further to implement status subscription in addition to events (or alternatively align with the spec and implement named event handlers instead of subject/flow — though this is less idiomatic and will be more cumbersome for users) cc @nicklasl @fabriziodemaria @typotter for your thoughts as this is a pain point in kotlin and swift
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I haven't had time to fully consider this, but I think it's close to what I've proposed in my other pr: #385 It doesn't mention the "BehaviorSubject" construct exactly, but describes something basically similar. I agree that we could use those abstractions though. |
||
| - Enforce short-circuit behavior for `NOT_READY` and `FATAL` statuses during flag evaluation | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. major: multi-threaded SDKs can't enforce this because of TOCTOU
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See #380 (comment) |
||
|
|
||
| #### Deprecation | ||
|
|
||
| The legacy path (SDK-managed status) should be deprecated in the release that introduces the state-managing interface, with removal targeted for the next major version. | ||
| SDK authors should update any first-party providers and provider base classes to implement the new interface. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -305,3 +305,60 @@ The track function performs side effects required to record the `tracking event` | |
| Providers should be careful to complete any communication or flush any relevant uncommitted tracking data before they shut down. | ||
|
|
||
| See [shutdown](#25-shutdown). | ||
|
|
||
| ### 2.8. Provider status | ||
|
|
||
| [](https://github.com/open-feature/spec/tree/main/specification#hardening) | ||
|
|
||
| Providers own their current status and any events associated with status transitions. | ||
| This allows providers to atomically update their status and emit the corresponding event, avoiding races between lifecycle methods terminating and events or status updates produced by concurrent work (such as background threads or pollers) maintained by the provider. | ||
|
|
||
| SDKs may provide a base class, wrapper, or other mechanism that maintains status on behalf of provider implementations which do not define it natively, for migration purposes. | ||
|
|
||
| see: [provider lifecycle management](./01-flag-evaluation.md#17-provider-lifecycle-management), [provider events](./05-events.md#51-provider-events) | ||
|
|
||
| #### Requirement 2.8.1 | ||
|
|
||
| > The provider **MUST** define a `status` accessor which indicates the provider's current readiness, with possible values `NOT_READY`, `READY`, `STALE`, `ERROR`, or `FATAL`. | ||
|
|
||
| The `status` accessor reflects the provider's readiness to evaluate flags. | ||
| The client's `provider status` accessor (see [requirement 1.7.1](./01-flag-evaluation.md#requirement-171)) delegates to this accessor. | ||
|
|
||
| see: [provider status](../types.md#provider-status) | ||
|
|
||
| #### Condition 2.8.2 | ||
|
|
||
| > The implementation uses the static-context paradigm. | ||
|
|
||
| see: [static-context paradigm](../glossary.md#static-context-paradigm) | ||
|
|
||
| ##### Conditional Requirement 2.8.2.1 | ||
|
|
||
| > In addition to `NOT_READY`, `READY`, `STALE`, `ERROR`, or `FATAL`, the provider's `status` accessor **MUST** support possible value `RECONCILING`. | ||
|
|
||
| In the static-context paradigm, the provider must define a `status` value indicating that it is reconciling its internal state due to a context change. | ||
|
|
||
| see: [provider context reconciliation](#26-provider-context-reconciliation) | ||
|
|
||
| #### Requirement 2.8.3 | ||
|
|
||
| > The provider's `status` **MUST** be `NOT_READY` before `initialize` is called and after `shutdown` terminates. | ||
|
|
||
| Providers which do not define an `initialize` function are assumed to be ready at all times, and their `status` may be `READY` from construction. | ||
|
|
||
| see: [initialization](#24-initialization), [shutdown](#25-shutdown) | ||
|
|
||
| #### Requirement 2.8.4 | ||
|
|
||
| > The provider's `status` accessor **MUST** be safe for concurrent access. | ||
|
|
||
| In languages supporting multi-threaded execution, the provider must ensure that concurrent reads of the `status` accessor do not observe torn or inconsistent values. | ||
|
|
||
| #### Requirement 2.8.5 | ||
|
|
||
| > Status changes and any associated event emissions **MUST** be atomic from the perspective of external observers. | ||
|
|
||
| When a provider transitions between statuses and emits an event associated with that transition, external observers (such as SDK event handlers) must observe a consistent view: the updated `status` value and the emitted event are visible together. | ||
| This prevents ordering anomalies where, for example, a `PROVIDER_READY` handler runs while `status` still indicates `NOT_READY` or `ERROR`, or where the provider transitions out of a status before the associated event is dispatched. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Dispatch" may be a big ambiguous here:
Is the intent here that event handlers see the exact status that triggered the event? If so, this may be problematic as it requires holding the status lock while all handlers run, preventing the provider from changing its own status. I think what we're after here is establishing an observable "happens-before" relationship:
So an event handler may observe next status changes, the important bit is that it must observe the status change that triggered the event.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I will remove this line, I think. I have some bigger concerns though.
I think we would also need some sort of lock to :
As far as I can tell, none of the SDKs currently do this. I'm a bit hesitant to add this though, since it means a misbehaving or blocked handler could block subsequent event emissions.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Possible relates to: #380 (comment)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Given that all event handlers are synchronous and most SDKs use either a queue or direct event firing, I think this is actually the default behavior we have today? Perhaps relaxing this a bit we can have: for each event handler, it should be fired in the order of events happening and each invocation should wait for the previous invocation to finish (this corresponds to a per-subscriber queue). If we relax it more, that would allow the publisher to either publish events out-of-order or concurrently, and subscriber would have no way to establish the real order — not very great. fwiw, I think we can drop this from the spec — SDKs should be able to figure out what's appropriate for each language
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Moot in the alternative approach: #380 (comment) |
||
|
|
||
| see: [provider events](./05-events.md#51-provider-events) | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor/aside: I'd argue that the better design is implementing an adapter that takes a non-
StateManagingProviderand implements theStateManagingProviderinterface. This way, all special-casing/mapping is concentrated in one place, and it is about the only thing that needs to be deleted in the next major release.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIUC this is essentially what the PoCs already do; FeatureProviderStateManager in Java and providerReference in Go are effectively that adapter/wrapper. They wrap legacy providers and handle state management on their behalf, so downstream SDK code doesn't need to special-case. At the next major version, the wrapper is the only thing that gets deleted.
Or am I missing something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was referring to using adapter pattern to adapt legacy interface to the new one. Strictly speaking, none of the PoCs do that.
Kotlin PR is an example.:
LegacyFeatureProviderAdapterwraps a non-state-managing provider and implement state-managing interface (manage status, emits events, etc.). It nicely contains/abstracts the knowledge of non-managing providers, and the rest of openfeature sdk handles all providers equally as if they can manage their state.This also reduces the cyclomatic complexity overall: the adapter knows the wrapped provider doesn't manage status, so it doesn't need any ifs. The SDK can treat all providers as state-managing, so it doesn't need any ifs either. The only
ifis insetProviderto determine whether wrapping is needed.In other PoCs, the provider wrapper now has a double-duty of whatever-it-was-doing-before + adapting legacy providers, so it's riddled with
ifs. And it still doesn't contain the adapting completely and this part doesn't hold:In JS,
OpenFeatureCommonAPIreaches inside withwrappedProvider.delegateManagesStateto determine whether to emit events or not. In Java,ProviderRepositoryreaches in withnewManager.delegateManagesState. Go is doing the checks in both event executor and openfeature API. So we'd need to modify these components as well.It's not a huge deal though, especially for a temporary migration. I find the proper adapter slightly cleaner but it is more work than throwing a bunch of if's in
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, OK, you were just talking about encapsulating this logic more elegantly then? If that's the case I agree. The PoCs were just there to prove the flow was possible, not necessarily in the cleanest way.