Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
[BUG] ssl-internal secret never gets created if the operator crashes and restarts at some particular point
General
Escalation
General
Escalation
Description
Description We find that the controller might never be able to create the `ssl-internal` secret if it crashes in the middle of a reconcile and restarts.Specifically, the controllers uses `reconcileSSL` to create secrets. The method works as follows:
func (r *ReconcilePerconaServerMongoDB)reconsileSSL(cr *api.PerconaServerMongoDB) error { secretObj :=corev1.Secret{} err :=r.client.Get(context.TODO(),...)// Get ssl secretif err == nil {return nil
}elseif!k8serrors.IsNotFound(err){returnfmt.Errorf("get secret: %v", err)}... err = r.createSSLManualy(cr)// Create secrete is ssl secret is not found...}
and inside `createSSLManualy` it tries to create two secrets: the `ssl` one and the `ssl-internal` one:
If the operator crashes between 2 and 3 and restarts, it will face a dirty state: the `ssl` secret is created, while the `ssl-internal` secret is not. More importantly, the operator cannot recover from the dirty state because it only checks whether `ssl` exists to decide whether to create the two secrets. Since `ssl` already exists, the operator will never create `ssl-internal`.
Possible Solution A potential solution is to check the existence of `ssl` secret and `ssl-internal` secret and create them separately. We are willing to send a PR to help fix this issue.
Description
We find that the controller might never be able to create the `ssl-internal` secret if it crashes in the middle of a reconcile and restarts.Specifically, the controllers uses `reconcileSSL` to create secrets. The method works as follows:
func (r *ReconcilePerconaServerMongoDB) reconsileSSL(cr *api.PerconaServerMongoDB) error { secretObj := corev1.Secret{} err := r.client.Get(context.TODO(), ...) // Get ssl secret if err == nil { return nil } else if !k8serrors.IsNotFound(err) { return fmt.Errorf("get secret: %v", err) } ... err = r.createSSLManualy(cr) // Create secrete is ssl secret is not found ... }and inside `createSSLManualy` it tries to create two secrets: the `ssl` one and the `ssl-internal` one:
err = r.client.Create(context.TODO(), &secretObj) // create ssl secret ... err = r.client.Create(context.TODO(), &secretObjInternal) // create ssl-internal secretTo put it in a simple way, the controller:
check whether `ssl` secret exists, if not exist:
create `ssl` secret
create `ssl-internal` secret
If the operator crashes between 2 and 3 and restarts, it will face a dirty state: the `ssl` secret is created, while the `ssl-internal` secret is not. More importantly, the operator cannot recover from the dirty state because it only checks whether `ssl` exists to decide whether to create the two secrets. Since `ssl` already exists, the operator will never create `ssl-internal`.
Possible Solution
A potential solution is to check the existence of `ssl` secret and `ssl-internal` secret and create them separately. We are willing to send a PR to help fix this issue.