Cluster sync adj in p&a flavour#1814
Cluster sync adj in p&a flavour#1814M4KIF wants to merge 14 commits intofeature/database-controllersfrom
Conversation
|
CLA Assistant Lite bot: I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request |
| // return ctrl.Result{}, nil | ||
| } | ||
|
|
||
| type clusterReadynessCheck interface { |
There was a problem hiding this comment.
sound like a great thing to add to the specific port capabilities :-)
There was a problem hiding this comment.
Then the interface here would be implemented by the ports. Clean and nice idea
| pgcConstants "github.com/splunk/splunk-operator/pkg/postgresql/cluster/business/core/types/constants" | ||
| ) | ||
|
|
||
| type Provisioner interface { |
There was a problem hiding this comment.
I think our secondary ports should reflect that we create cluster and database and we should map our interfaces around it.
| */ | ||
|
|
||
| // basically a sync logic | ||
| state := pgcConstants.EmptyState |
There was a problem hiding this comment.
I think the idea here was to decouple status check from cnpg status, At the same time we also check health after every stage and we move forward only if we are ok, if it is still in progress we requeue or raise error. Here we have a code we can use to check where we are with status iteratively, but I dont see yet how it solve our core problem
There was a problem hiding this comment.
we build our state here, after checking all ports for readyness/not dying there is the moment for us to decide what happened and how It happened.
We don't set our state as state == cnpgVariable mapped to ours, we decide what do we want to do with the fact that cm's, secrets, provisioner etc. are ready.
| cnpgv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" | ||
| enterprisev4 "github.com/splunk/splunk-operator/api/v4" | ||
| clustercore "github.com/splunk/splunk-operator/pkg/postgresql/cluster/core" | ||
| clustercore "github.com/splunk/splunk-operator/pkg/postgresql/cluster/business/core" |
There was a problem hiding this comment.
tbh business string is redundat here. Core itself is already a domain
There was a problem hiding this comment.
I agree It's redundant, here It's a tradeoff for verbosity and segregation of components. And a service pattern at once, ie. the service/ is the primary port (reconciler that we provide) implementation. core/ is core, and ports/ are the contracts that we need for the core to work. They can grow large, hence the whole separate dir for ports.
| cnpgCluster *cnpgv1.Cluster | ||
| } | ||
|
|
||
| func (c *provisionerHealthCheck) Condition() (pgcConstants.State, error) { |
There was a problem hiding this comment.
I think all of this conditions check should be part of our Ports, also how you want to map condition to phase?
There was a problem hiding this comment.
I agree, they were placed here as what I need. Solving It like you say is the thing I'm hoping for. For the provisioner/cluster etc. ports to include an interface for checking It's state.
Then the adapter would be essentially mapping the dependency state to our abstraction of It's state. Ie. we have cluster ready, provisioning, failover, cupcake, coffee etc.
Mapping condition to phase would be the job of the facade, ie. the cluster.go. That would be the whole operational brain behind. Ie. lot's of individual pieces funneled into our business decisions.
There was a problem hiding this comment.
Sth. like a stateMapper, or another objects that specialises in deciding on what phase/condition we're in could also be born. The bit mask could be used for covering the phase 1, ie. state = FinaliserNotAdded && !ClusterProvisioning etc.
Phase 2 -> state == Finaliser && ClusterReady etc.
| } | ||
|
|
||
| func (p *poolerHealthCheck) Condition() (pgcConstants.State, error) { | ||
| return pgcConstants.PoolerReady, nil |
There was a problem hiding this comment.
how do we want to check the actual condition component has in status?
There was a problem hiding this comment.
wdym? It's kind of the job of the adapter to test and provide that the state is actual. Ie. If we place this as a method of a port, and implement It via adapters. We actually won't work on the real state of the component in our core. Only on our understanding of It.
There was a problem hiding this comment.
Currently It would be just to copy paste the thing that we do inside cluster.go, ie. the resource obj. of the Pooler, k8s.Get(obj, ...) and all similar. As there is no abstraction currently.
| ) | ||
|
|
||
| const ( | ||
| ComponentsReady = PoolerReady | ProvisionerReady | SecretReady | ConfigMapReady |
There was a problem hiding this comment.
what we try to achieve here? Is it a bitmask? Since you use IOTA we endup in having just random integer?
That would be probably simpler to just use struct with keeping the state like that:
type ClusterState struct {
Provisioner ComponentPhase
Pooler ComponentPhase
ConfigMap ComponentPhase
Secret ComponentPhase
}
There was a problem hiding this comment.
It's a bitmask. And It does the same job of keeping a struct with aditional field.
And It kinda solves the case of having to create new types just for each component.
Just adding additional states to the state machine, ie. the values in the "enum". The iota usage is an enum in go: https://yourbasic.org/golang/iota/
It's the first usage in this file, hence It's basically an enum from 0
There was a problem hiding this comment.
And It kinda solves the case of having to create new types just for each component.
We already create types for a new component for many reasons, so what problem it really solve? I agree it is smart way of doing this, but neverthless if new component arise you need to add it to the const and extend types. I feel like we trade go readability for a really small c-like optimisation especially with this model of bitwise comparision later:
state |= componentHealth.State
if state&pgcConstants.ComponentsReady == pgcConstants.ComponentsReady
Also, if we build state incrementally it means that the LAST successful state in state machine is an final success. With this in mind we dont need to check every other component state afterwards.
Can we do this simpler so when Im wake up at 5am in the morning I easily understand the code?
There was a problem hiding this comment.
I agree that this state check, after the iteration health check passes is redundant here.
And taking into consideration the potential future work. Which could include a file division.
I could expand the *healthCheck types with them returning the *(component)StateDto instead of relying on generic state bits.
There was a problem hiding this comment.
Because later, in the very near future It seems that the project could follow this footstep of having some separation in phases and It's crucial elements. Like we've discussed on the p&a ideas brainstorm.
| rc.emitPoolerReadyTransition(postgresCluster, poolerOldConditions) | ||
| } | ||
|
|
||
| if state&pgcConstants.ComponentsReady != 0 { |
There was a problem hiding this comment.
Not sure if this logic is not broken. What happens if we set ProvisionerReady but later in stage we set failed for something. Sue to way we set state and components this would pass. I think it is because iota incrementing by 1 not by power of 2? I think we should rely not on bitmasking here, but simple struct with state for every stage and if all is good we are good
There was a problem hiding this comment.
with unsetting the bit's at any space, this condition starts failing. As well as with not setting the bits, the values don't AND, hence if we mark ProvisionerReady and then set masks for ConfigMapFailed, it won't fire.
And to prove how this logic would work, there would be tests that would make sure any misfires aren't possible.
b0b2f12 to
ad7aaec
Compare
| oldPhase = *postgresCluster.Status.Phase | ||
| // Aggregate component readiness from iterative health checks. | ||
| state := pgcConstants.EmptyState | ||
| conditions := []clusterReadynessCheck{ |
There was a problem hiding this comment.
so after every phase that is not immediate like cluster creation we should also incorporate state check rediness. I think we discussed that we dont really need to check at the end assuming we check intermediary statuses per phase?
There was a problem hiding this comment.
I agree with that, but Isn't then the scope == refactor the reconciler?
I've tried to stick with changing the sync logic and doing the ground prep for more changes in coming tickets and potential p&a rework.
| for _, check := range conditions { | ||
| componentHealth, err := check.Condition(ctx) | ||
| if err != nil { | ||
| if statusErr := updateStatus(componentHealth.Condition, metav1.ConditionFalse, componentHealth.Reason, componentHealth.Message, componentHealth.Phase); statusErr != nil { |
There was a problem hiding this comment.
If we run this at the end of reconcillation it seems that some of the code is dead i.e if we are here, we cannot have a configmap or secret orphaned. If we do it should be discovered during this phase and requeue/err
| } | ||
| return ctrl.Result{}, statusErr | ||
| } | ||
| logger.Error(err, "Component health check reported issues", |
There was a problem hiding this comment.
Please follow our logging strategy: https://splunk.atlassian.net/wiki/spaces/CCP/pages/1079831167399/PostgreSQL+Controllers+Logging+Strategy
| return componentHealth.Result, err | ||
| } | ||
|
|
||
| if isPendingState(componentHealth.State) { |
There was a problem hiding this comment.
If we run this code on every phase separately, here we should requeue
| if postgresCluster.Status.Phase != nil { | ||
| newPhase = *postgresCluster.Status.Phase | ||
|
|
||
| if state&pgcConstants.ComponentsReady == pgcConstants.ComponentsReady { |
There was a problem hiding this comment.
thois could be potentially method on a state so it reads natually at 4 am i.e
func (s State) HasAll(required State) bool {
return s&required == required
}
if state.HasAll(pgcConstants.ComponentsReady) {}
There was a problem hiding this comment.
after changing the state idea, It currently isn't used that much.
| return &provisionerHealthCheck{cluster: cluster, cnpgCluster: cnpgCluster} | ||
| } | ||
|
|
||
| func (c *provisionerHealthCheck) Condition(_ context.Context) (StateInformationDto, error) { |
There was a problem hiding this comment.
I like providing interface like that! One doubt I have is about pureness and responsiblity of those condition methods. They check conditions but also fetch k8s objects. IMO k8s objects should be evaluated and the reconciller kickoff and propagated
There was a problem hiding this comment.
I've proposed another approach. Which is quite similiar to the current tbh
|
I like the direction we are heading two!
|
|
vivekr-splunk
left a comment
There was a problem hiding this comment.
I found one blocking regression in the status/event flow.
| logger.Error(err, "Failed to sync status") | ||
|
|
||
| logger.Info("Reconciliation complete") | ||
| if err := updateStatus(clusterReady, metav1.ConditionTrue, reasonCNPGClusterHealthy, msgAllComponentsReady, readyClusterPhase); err != nil { |
There was a problem hiding this comment.
This refactor appears to drop cluster phase transition events entirely. Before the change, the controller captured the old/new phase around the final status sync and called rc.emitClusterPhaseTransition(...); that helper still exists in events.go, but I no longer see it invoked on the happy path or on degraded transitions. From the product point of view, that means users lose the ClusterReady / ClusterDegraded event stream even though status still changes, which looks like a behavior regression rather than just an internal refactor.
There was a problem hiding this comment.
I agree, I've missed this when attempting the rewrite. Thank you for your input and I agree, I've split the logic and focused only on the buzzwords/keywords and totally went dark on observability/event part.
| PostgresClusterFinalizerName string = "postgresclusters.enterprise.splunk.com/finalizer" | ||
|
|
||
| // cluster phases | ||
| readyClusterPhase reconcileClusterPhases = "Ready" |
There was a problem hiding this comment.
I see that the condition types SecretReady, ManagedRolesReady, and ConfigMapReady have not been added.
At the moment, we only define clusterReady and poolerReady. The functions secretModel.Converge, configMapModel.Converge, and the provisioner all update clusterReady, which means the status does not reflect the more granular state described in the ticket.
According to the ticket, the expected status snapshot should be:
SecretReady = True (SuperUserSecretReady)
ManagedRolesReady = True (ManagedRolesReconciled)
ConfigMapReady = True (ConfigMapReconciled)
The reason values also need adjustment. Currently, secretModel.Converge uses:
reasonClusterBuildSucceeded for the success case, and
reasonUserSecretFailed for the "ref not yet written" pending case.
Both are inherited from the old clusterReady vocabulary. However, the ticket specifies that the reasons should be SuperUserSecretReady and SuperUserSecretFailed, so these should be updated accordingly.
| if statusErr := updateStatus(poolerReady, metav1.ConditionFalse, reasonPoolerCreating, | ||
| "Connection poolers are being provisioned", provisioningClusterPhase); statusErr != nil { | ||
| logger.Error(statusErr, "Failed to update status") | ||
| if rolesErr := reconcileManagedRoles(ctx, p.client, p.cluster, p.cnpgCluster); rolesErr != nil { |
There was a problem hiding this comment.
ManagedRoles (Step 3) is currently handled inside provisionerModel.Actuate, but it would be clearer if it were implemented as a separate pipeline step.
Right now:
There is no ManagedRolesReady condition.
There is no distinct Step 3 in the pipeline.
Observers cannot distinguish between a failure in provisioning CNPG and a failure during role reconciliation.
Separating this into its own step would make the pipeline state clearer and improve observability of failures and it is also one of the acceptance criteria :-)
| "requeueAfter", result.RequeueAfter) | ||
| return result, nil | ||
| } | ||
| return result, nil |
There was a problem hiding this comment.
Thing to fix once we split phases per step.
manageComponentHealth never writes Condition=True on the success path.
The function has three branches:
error → writes False
pending state → writes False
ready → writes nothing
As a result, the pipeline only ever stamps conditions as False. The only place where Condition=True is written during reconciliation is the final call.
This happens only after all steps succeed, and it updates clusterReady only.
This breaks the audit-trail invariant described in the ticket. After a fully healthy reconcile, conditions like SecretReady, ManagedRolesReady, and ConfigMapReady never appear in etcd.
If CNPG enters a switchover during the next reconcile, the provisioner returns early with ClusterReady=False, and those provisioning conditions are still absent, not True. As a result, an operator running kubectl describe cannot distinguish between:
“the secret was provisioned successfully and is still healthy”, and
“the secret provisioning step has never run.”
| p.health.Message = fmt.Sprintf("Failed to check RW pooler existence: %v", err) | ||
| p.health.Phase = failedClusterPhase | ||
| p.health.Result = ctrl.Result{} | ||
| return p.health, err |
There was a problem hiding this comment.
previously we were setting:
cluster.Status.ConnectionPoolerStatus = &ConnectionPoolerStatus{Enabled: true}
when everything was fine, currently we miss it. We use it in database.go to check if we should use connection pooler. Without it we will be always using direct cnpg endpoint
There was a problem hiding this comment.
we should guard this logic in unit tests/integration tests against future regressions in contract
|
|
||
| // Build desired CNPG Cluster spec. | ||
| desiredSpec := buildCNPGClusterSpec(mergedConfig, postgresSecretName) | ||
| provisionerComponent := components[1].(*provisionerModel) |
There was a problem hiding this comment.
this seem to be fragile if we change order.
| return s.health, nil | ||
| } | ||
|
|
||
| func isPendingState(state pgcConstants.State) bool { |
There was a problem hiding this comment.
The function name is bit misleading as it returns true for Pending, Provisioning, and Configuring. The name implies only the first.
| func (p *provisionerModel) Name() string { return pgcConstants.ComponentProvisioner } | ||
|
|
||
| func (p *provisionerModel) EvaluatePrerequisites(_ context.Context) (prerequisiteDecision, error) { | ||
| if p.cluster.Status.Resources == nil || p.cluster.Status.Resources.SuperUserSecretRef == nil { |
There was a problem hiding this comment.
we might still be in provision state where secrets is being created and thats why we dont have it in the status. this reason can be misleading
| Allowed: false, | ||
| Health: componentHealth{ | ||
| State: pgcConstants.Pending, | ||
| Condition: clusterReady, |
|
|
||
| func (c *configMapModel) Name() string { return pgcConstants.ComponentConfigMap } | ||
|
|
||
| func (c *configMapModel) EvaluatePrerequisites(_ context.Context) (prerequisiteDecision, error) { |
There was a problem hiding this comment.
I think this gate can never actually fire, If CNPG is not healthy, provisionerModel.Converge returns a pending/failed state → phase() returns a non-zero result → the first loop returns early.
Also this should set its own condition not clusterReady
There was a problem hiding this comment.
Yes, that's the case under usual condition. It's a double caution safety. So It can be deleted if the system is to be trusted enough. My bet would be to leave It for now.
Condition altered.
There was a problem hiding this comment.
ok can you share what would be the case it fires? My understanding is that this is dead code, what do I miss? If this is dead code, we simply dont need it
There was a problem hiding this comment.
It mostly dead code, shouldn't be fired. So yeah, deleted.
| * PC-09 ignores no-op updates | ||
| */ | ||
|
|
||
| func containsEvent(events []string, recorder *record.FakeRecorder, eventType string, event string) bool { |
There was a problem hiding this comment.
Not sure if I read it right but The helper reads exactly one event from the channel per call and returns on the first read regardless of whether it matches. If the target event is the
second event in the channel, the Eventually loop never finds it.
There was a problem hiding this comment.
the eventually loop calls It repeatedly until it finds or times out
There was a problem hiding this comment.
why not to do this in one run? I.e pick all events and iterate over them. In current situation received can be also not accurate i.e recorder.Events != received if you return true early. Plus if we take one event why for :-)
There was a problem hiding this comment.
Ah I just realised that the received will not be accurate as we do copy inside of the function by appending. we should pass * if you want to update this value
There was a problem hiding this comment.
i agree with the slice pointer. I didn't understand, but yeah, the fake recorder is just a channel to which we write some strings.
There was a problem hiding this comment.
eventually + for single exit seems like an overkill, as It's an await for a real async scenario.
But, we write to the channel, w/o async. So a for w/ contains is enough
| newConfigMapModel(c, rc.Scheme, rc, updateComponentHealthStatus, runtimeView, postgresCluster, postgresSecretName), | ||
| } | ||
|
|
||
| for _, component := range runtimeComponents { |
There was a problem hiding this comment.
why we are not iterating over full list of components in line 180?
There was a problem hiding this comment.
because managedRoles, Pooler, ConfigMap models need some information from previous stages and injecting It needs to happen in-between.
The later phases should utilise ports in later iterations of the source to use the information they need to function.
|
|
||
| components := []componentModel{ | ||
| secretComponent, | ||
| provisionerComponent, |
There was a problem hiding this comment.
Id rather name it clusterComponent not provisionerComponent to provide better code readability. Provisioner is notion of external postgres provider that can do many things including cluster,roles,database, pooler etc
| Result ctrl.Result | ||
| } | ||
|
|
||
| type componentModel interface { |
There was a problem hiding this comment.
I wonder if we cannot merge this interfaces into one i.e
type component interface {
Actuate(ctx context.Context) error
Converge(ctx context.Context) (componentHealth, error)
EvaluatePrerequisites(ctx context.Context) (prerequisiteDecision, error)
Namer() string
}
they seem to be logically bounded and I dont see a use-case to use this interfaces in separation i.e we cannot have component that doesnt have function to actuate, converge etc.We can always split it later if it grows too much, or we have different usecase
There was a problem hiding this comment.
I agree, I tend to go over the board with verbosity. I agree component containing the above methods is bound, logical and consistent.
| } | ||
|
|
||
| if len(status.Pending) > 0 { | ||
| m.health.State = pgcConstants.Failed |
There was a problem hiding this comment.
this should be Provisioning state, and it is also not an error
| } | ||
|
|
||
| s.health.State = pgcConstants.Ready | ||
| s.health.Reason = reasonClusterBuildSucceeded |
There was a problem hiding this comment.
this should be probably reasonSuperUserSecretReady not reasonClusterBuildSucceeded
There was a problem hiding this comment.
yeah, as we saw in the kubectl describe. Thanks! adjusted
| if postgresCluster.Status.Phase != nil { | ||
| oldPhase = *postgresCluster.Status.Phase | ||
| c.health.State = pgcConstants.Ready | ||
| c.health.Reason = reasonClusterBuildSucceeded |
There was a problem hiding this comment.
this should be probably reasonConfigMapReady reason
| } | ||
|
|
||
| cluster.Status.ConnectionPoolerStatus = &enterprisev4.ConnectionPoolerStatus{Enabled: true} | ||
| rwDesired, rwScheduled := poolerInstanceCount(rwPooler) |
There was a problem hiding this comment.
i see you removed poolerInstanceCount usage, but did we delete the function?
There was a problem hiding this comment.
It was left hanging in the units. Deleted It
|
@M4KIF did a review + additionally passed pr through LLM. Few points brought: A. managedRolesModel.EvaluatePrerequisites uses State=Failed for a blocked gate func (m *managedRolesModel) EvaluatePrerequisites(_ context.Context) (prerequisiteDecision, error) { The gate blocks because CNPG isn't healthy yet — that's a transient wait, not a failure. But because the phase() driver calls component.Converge(ctx) when a B. managedRolesModel.Converge duplicates the prerequisite check and returns an error for a wait state func (m *managedRolesModel) Converge(ctx context.Context) (health componentHealth, err error) { This is the same condition as the prerequisite gate — structurally unreachable when called normally (gate would have blocked), but reached in the blocked-gate Compare how the provisioner handles this: its blocked gate path sets State=Pending, err=nil so the driver returns the requeue result cleanly. C. configMapModel.EvaluatePrerequisites is structurally unreachable but still has a blocking problem The gate is now dead code (structurally can't fire since the provisioner gates the first loop). However it uses configMapsReady (correct condition type). The dead code concern remains — if the pipeline is reorganised, the gate will incorrectly return early with State=Provisioning and then the blocked gate path in phase() calls Converge, which also checks !IsHealthy() and returns State=Provisioning, err=nil, safely causing a requeue. |
|
⏳ CLA signed — now checking Code of Conduct status... |
Description
Rewritten the sync logic
Key Changes
Testing and Verification
local integ suite passes (make test) and units (pkg/postgres/cluster/core go test) passes
Related Issues
Jira tickets, GitHub issues, Support tickets...
PR Checklist