Support ClusterIssuer for MongoDB ReplicaSet Cluster with TLS and Let's Encrypt

Description

k8s Cluster: GKE 1.24.11-gke.1000

Percona Operator: - v1.14

We are attempting to create a MongoDB ReplicaSet Cluster that supports TLS with a Certificate from a ClusterIssuer of Let's Encrypt.

We have a cert-manager set up and running in the cluster.

As far as I can understand, the operator does not support ClusterIssuer only his own issuer, is that true?

I tried a few workarounds, one of them was to create my own Certificate for the replica set nodes as DNS and ClusterIssuer of Letsencrypt using cert-manager, then I took the tls.crt and tls.key and combine them with the downloaded ca.pam from the Let's Encrypt website and crate secret with ssl and ssl-internal. 

psmdb is stuck on the initializing state.

Activity

Aaditya Dubey June 2, 2023 at 8:26 AM

Hi ,

Thank you for the report and feedback.

marc May 30, 2023 at 8:59 AM

I am sorry, my answer was not differential enough.

As there is a Bug https://forums.percona.com/t/splithorizon-dns/20205/11 with exposing the ReplicaSet, I am connecting the cluster through `mongos`, like it is recommended in the linked topic.

I do not like to post a complete guide with step-ca, step-issuer ... within this issue, as this requires a lot of time. If percona is interested in that, I could do this for a blog post. I haven't got an answer from percona yet.

I do not know how you want to get certificates with TDL `svc.cluster.local` signed by letsencrypt, for this will not be possible.

Here is my `cr.yaml` (without out-commented lines)

```

apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDB
metadata:
 name: percona
 finalizers:
   - delete-psmdb-pods-in-order
   - delete-psmdb-pvc
spec:
 image: percona/percona-server-mongodb:5.0.5
 imagePullPolicy: Always
 allowUnsafeConfigurations: false
 updateStrategy: SmartUpdate
 upgradeOptions:
   versionServiceEndpoint: https://check.percona.com
   apply: disabled
   schedule: "0 2 * * *"
   setFCV: false
 secrets:
   users: my-cluster-name-secrets
   encryptionKey: my-cluster-name-mongodb-encryption-key
   ssl: rdx-tls-mongodb-external
   sslInternal: rdx-tls-mongodb-internal

 pmm:
   enabled: false
   image: percona/pmm-client:2.35.0
   serverHost: monitoring-service
 replsets:

 - name: rs0
   size: 3
   configuration: |
     net:
       tls:
         mode: requireTLS
     security:
       clusterAuthMode: keyFile
   affinity:
     antiAffinityTopologyKey: "kubernetes.io/hostname"
   podDisruptionBudget:
     maxUnavailable: 1
   expose:
     enabled: false
     exposeType: LoadBalancer
   resources:
     limits:
       cpu: "300m"
       memory: "0.5G"
     requests:
       cpu: "300m"
       memory: "0.5G"
   volumeSpec:
     persistentVolumeClaim:
       resources:
         requests:
           storage: 3Gi

   nonvoting:
     enabled: false
     size: 3
     configuration: |
       net:
         tls:
           mode: requireTLS
     affinity:
       antiAffinityTopologyKey: "kubernetes.io/hostname"
     podDisruptionBudget:
       maxUnavailable: 1
     resources:
       limits:
         cpu: "300m"
         memory: "0.5G"
       requests:
         cpu: "300m"
         memory: "0.5G"
     volumeSpec:
       persistentVolumeClaim:
         resources:
           requests:
             storage: 3Gi
   arbiter:
     enabled: false
     size: 1
     affinity:
       antiAffinityTopologyKey: "kubernetes.io/hostname"

 sharding:
   enabled: true

   configsvrReplSet:
     size: 3
     configuration: |
       net:
         tls:
           mode: requireTLS
     affinity:
       antiAffinityTopologyKey: "kubernetes.io/hostname"
     podDisruptionBudget:
       maxUnavailable: 1
     expose:
       enabled: false
       exposeType: LoadBalancer
     resources:
       limits:
         cpu: "300m"
         memory: "0.5G"
       requests:
         cpu: "300m"
         memory: "0.5G"
     volumeSpec:
       persistentVolumeClaim:
         resources:
           requests:
             storage: 3Gi

   mongos:
     size: 3
     configuration: |
       net:
         tls:
           mode: requireTLS
           allowConnectionsWithoutCertificates: true
     affinity:
       antiAffinityTopologyKey: "kubernetes.io/hostname"
     podDisruptionBudget:
       maxUnavailable: 1
     resources:
       limits:
         cpu: "300m"
         memory: "0.5G"
       requests:
         cpu: "300m"
         memory: "0.5G"
     expose:
       exposeType: ClusterIP
       servicePerPod: true

 backup:
   enabled: true
   image: perconalab/percona-server-mongodb-operator:main-backup
   serviceAccountName: percona-server-mongodb-operator
   pitr:
     enabled: false
     compressionType: gzip
     compressionLevel: 6
```

 

My Certificate looks like this

 

```

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: rdx-tls-mongodb-external
spec:
 secretName: rdx-tls-mongodb-external
 issuerRef:
   name: reddoxx
   kind: StepClusterIssuer
   group: certmanager.step.sm
 commonName:  "percona"
 dnsNames:
   - "localhost"
   - "percona-rs0"
   - "percona-rs0.staging-percona"
   - "percona-rs0.staging-percona.svc.cluster.local"
   - "*.percona-rs0"
   - "*.percona-rs0.staging-percona"
   - "*.percona-rs0.staging-percona.svc.cluster.local"
   - "percona-rs0.staging-percona.svc.clusterset.local"
   - "*.percona-rs0.staging-percona.svc.clusterset.local"
   - "*.staging-percona.svc.clusterset.local"
   - "percona-mongos"
   - "percona-mongos.staging-percona"
   - "percona-mongos.staging-percona.svc.cluster.local"
   - "*.percona-mongos"
   - "*.percona-mongos.staging-percona"
   - "*.percona-mongos.staging-percona.svc.cluster.local"
   - "percona-cfg"
   - "percona-cfg.staging-percona"
   - "percona-cfg.staging-percona.svc.cluster.local"
   - "*.percona-cfg"
   - "*.percona-cfg.staging-percona"
   - "*.percona-cfg.staging-percona.svc.cluster.local"
   - "percona-mongos.staging-percona.svc.clusterset.local"
   - "*.percona-mongos.staging-percona.svc.clusterset.local"
   - "percona-cfg.staging-percona.svc.clusterset.local"
   - "*.percona-cfg.staging-percona.svc.clusterset.local"
   - "percona-mongos-0.staging-percona.svc.cluster.local"
   - "percona-mongos-1.staging-percona.svc.cluster.local"
   - "percona-mongos-2.staging-percona.svc.cluster.local"
   - "percona-mongos-0.staging-percona"
   - "percona-mongos-1.staging-percona"
   - "percona-mongos-2.staging-percona"
   - "percona-mongos-0"
   - "percona-mongos-1"
   - "percona-mongos-2"
   - "mongodb.xyz.example.com"
```

 

 

Avner Zini May 28, 2023 at 6:57 AM

  1. How do you create MongoDB ReplicaSet Cluster that supports TLS with a Certificate from a ClusterIssuer, I am sure there are multiple steps for this. A blog post is a great idea and can be constructive.

  2. same as me, two secrets. with the same content for all three parameters ( ca.crt, tls.crttls.key )

  3. I did try to generate certificates manually, but the ca is not letsencrypt ca, maybe I need to play with this a bit more.

  4. Yes, I'm sure .

  5. can you please share if you used clusterServiceDNSMode: Internal or External? and how it affects your "hosts" section.

Thank you

marc May 24, 2023 at 4:19 PM

> 1. Please elaborate on how you did it.

How I did what? These are multiple steps. I have just asked percona if they are interested in an article for their blog...

> 2. Did you create different secrets for ssl and ssl-internal?

Two secrets, but the certificates are with the same content.

> 3. TLS-docs of the operator.  - how it can be useful for public domains or clusterIssuer?

You have to create secrets containing this: https://docs.percona.com/percona-operator-for-mongodb/TLS.html#generate-certificates-manually 

How you do that doesn't matter. I've done it with step-ca and step-issuer. But this takes some time to understand. In my case about a week, but the CA was already running ...

> 4. the `subject.organization' have all the necessary `usages`.

Sure?

You can check your secrets like that:

`kubectl get secret/my-cluster-name-ssl-internal -o jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -noout -text`

Please check for the usage-fields and the `subject.organization`

Avner Zini May 24, 2023 at 2:49 PM

Thank you for your comment: 

  1. Please elaborate on how you did it.

  2. Did you create different secrets for ssl and ssl-internal? 

  3. TLS-docs of the operator.  - how it can be useful for public domains or clusterIssuer?

  4. the `subject.organization' have all the necessary `usages`.

this is my Certificate:

this certificate creates this secret named "minimal-cluster-tls": 

_apiVersion: v1

data:
  tls.crt: blablabla
  tls.key: blablabla_
_kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: stg-mongo-psmdb-rs0-0.comapny.com,stg-mongo-psmdb-rs0-1.comapny.com,stg-mongo-psmdb-rs0-2.comapny.com_
_cert-manager.io/certificate-name: minimal-cluster-tls
    cert-manager.io/common-name: stg-eng-mongo-psmdb-rs0-0.company.com
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: letsencrypt-cloudflare
    cert-manager.io/uri-sans: ""
  name: minimal-cluster-tls
  namespace: mongodb
type: kubernetes.io/tls_

I took from this secret the tls.crt and the tls.key and downloaded the ca.pam (base64) from Let's Encrypt website and insert it into the secret as ca.crt 
after all the above I have two identical secrets (ssl and ssl-internal).

this is my deployed values (values.yaml):

Thank you!

 

 

 

Details

Assignee

Reporter

Needs QA

Yes

Affects versions

Priority

Smart Checklist

Created May 14, 2023 at 10:53 AM
Updated March 5, 2024 at 4:29 PM