MaxRetries: counter is reset on every resource submission

[eytan-avisror]

instance-manager/controllers/providers/kubernetes/crd.go

Lines 148 to 156 in c0eb25d

	_, err = kube.Resource(GVR).Namespace(customResourceNamespace).Create(context.Background(), customResource, metav1.CreateOptions{})
	if err != nil {
	if !kerr.IsAlreadyExists(err) {
	return false, errors.Wrap(err, "failed to submit custom resource")
	}
	} else {
	log.Info("submitted custom resource", "instancegroup", instanceGroupNamespacedName)
	status.SetStrategyRetryCount(0)
	}

The above logic states - if there is an error in creating a resource, and it already exists - do nothing, or fail if otherwise.
If creation is successful - reset the maxRetry counter, this is problematic in the cases where the creation is due to replacement of a failed CR, causing an infinite retry scenario, which is not the intention.

We should evaluate not resetting the counter there and test out various concurrency policies with it.