Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
Operator makes the MongoDB cluster to crash after changing spec.secrets.encryptionKey
General
Escalation
General
Escalation
Description
Describe the bug After change the encryptionKey from empty to some other string value, the operator is supposed to generated a new secret using the value as the name and using this new secret as encryptionKey. The operator does generated the secret and use the updated secret when starting the cluster; however, the MongoDB pods start crashing afterward with error messages:
Add the spec.secrets.encryptionKey field with value "non-exist-mongodb-encryption-key"
The replset and config-server restart but are crashing
Expected behavior The MongoDB cluster should be able to use the updated secret generated by the operator and run without an issue.
Current behavior The operator does generate the secret but the pods start crashing.
Note Changing the order of step 1 and step 2 will also cause the pods be in CrashLoopBackOff state
Root Cause We suspect the reason of crashing is that the operator encrypt the wiredTiger with old secret and store it on the PVCs. After the operator regenerates a new secret, the mongod starts by using the new secret which is inconsistent with the old storage.
Desktop (please complete the following information):
OS: Ubuntu
Browser [e.g. chrome, safari]
Reproduced on latest release v0.8.3
Environment
None
Activity
Aaditya Dubey September 6, 2024 at 2:42 PM
Hi @KOS
Thank you for the report. Verified as described. Please note our latest PSMDB operator is 1.16.2 at the moment and 1.17.0 is expected to be released soon.
Describe the bug
After change the
encryptionKeyfrom empty to some other string value, the operator is supposed to generated a new secret using the value as the name and using this new secret asencryptionKey. The operator does generated the secret and use the updated secret when starting the cluster; however, the MongoDB pods start crashing afterward with error messages:[1724614382:810307][1:0x7fe65a69bc00], file:WiredTiger.wt, connection: [WT_VERB_EXTENSION][ERROR]: libcrypto: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:To Reproduce
Steps to reproduce the behavior:
Deploy an CR, for example:
apiVersion: psmdb.percona.com/v1 kind: PerconaServerMongoDB metadata: name: minimal-cluster spec: crVersion: 1.17.0 image: perconalab/percona-server-mongodb-operator:main-mongod7.0 unsafeFlags: replsetSize: true mongosSize: true upgradeOptions: apply: disabled schedule: "0 2 * * *" secrets: users: minimal-cluster replsets: - name: rs0 size: 1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi sharding: enabled: true configsvrReplSet: size: 1 volumeSpec: persistentVolumeClaim: resources: requests: storage: 3Gi mongos: size: 1Add the
spec.secrets.encryptionKeyfield with value "non-exist-mongodb-encryption-key"The replset and config-server restart but are crashing
Expected behavior
The MongoDB cluster should be able to use the updated secret generated by the operator and run without an issue.
Current behavior
The operator does generate the secret but the pods start crashing.
Note
Changing the order of step 1 and step 2 will also cause the pods be in
CrashLoopBackOffstateRoot Cause
We suspect the reason of crashing is that the operator encrypt the wiredTiger with old secret and store it on the PVCs. After the operator regenerates a new secret, the mongod starts by using the new secret which is inconsistent with the old storage.
Desktop (please complete the following information):
OS: Ubuntu
Browser [e.g. chrome, safari]
Reproduced on latest release v0.8.3