3. If I have a custom storageclass with “allowVolumeExpansion: false” it seems operator tries to expand storage anyway, but it fails and the cluster is still in ready state (I tried to scale from 1G to 5G).

$ k get psmdb
NAME ENDPOINT STATUS AGE
my-cluster-name my-cluster-name-rs0.test.svc.cluster.local ready 3m45s

$ k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
mongod-data-my-cluster-name-rs0-0 Bound pvc-5b933e28-5651-4a23-9e05-0438e5ee081a 1Gi RWO standard-test <unset> 3m48s
mongod-data-my-cluster-name-rs0-1 Bound pvc-c9e726a5-a420-4096-9573-a5c20d4cde30 1Gi RWO standard-test <unset> 3m23s
mongod-data-my-cluster-name-rs0-2 Bound pvc-5a270dd3-cde3-421a-859d-f0857cbab57b 1Gi RWO standard-test <unset> 2m57s

$ k get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-5a270dd3-cde3-421a-859d-f0857cbab57b 1Gi RWO Delete Bound test/mongod-data-my-cluster-name-rs0-2 standard-test <unset> 2m59s
pvc-5b933e28-5651-4a23-9e05-0438e5ee081a 1Gi RWO Delete Bound test/mongod-data-my-cluster-name-rs0-0 standard-test <unset> 3m49s
pvc-c9e726a5-a420-4096-9573-a5c20d4cde30 1Gi RWO Delete Bound test/mongod-data-my-cluster-name-rs0-1 standard-test <unset> 3m24s

$ k get storageclass standard-test -oyaml
allowVolumeExpansion: false
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
components.gke.io/component-name: pdcsi
components.gke.io/component-version: 0.18.12
components.gke.io/layer: addon
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":false,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"components.gke.io/component-name":"pdcsi","components.gke.io/component-version":"0.18.12","components.gke.io/layer":"addon","storageclass.kubernetes.io/is-default-class":"false"},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"gcp-compute-persistent-disk-csi-driver"},"name":"standard-test"},"parameters":{"type":"pd-balanced"},"provisioner":"pd.csi.storage.gke.io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
storageclass.kubernetes.io/is-default-class: "false"
creationTimestamp: "2024-04-29T13:49:18Z"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
k8s-app: gcp-compute-persistent-disk-csi-driver
name: standard-test
resourceVersion: "249562"
uid: 38fb7586-eb57-4cc7-9d85-9cfd9dadfc3f
parameters:
type: pd-balanced
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

operator logs:

2024-04-29T13:51:22.705Z INFO update Mongo version to 7.0.8-5 (fetched from db) {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "0f4c8349-2223-4e0f-9705-d6f0798e0851"}
2024-04-29T13:53:16.294Z INFO Resizing PVCs {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "8fbc9204-1552-4237-ac47-454eaf88557c", "requested": "5Gi", "actual": "1Gi", "pvcList": "mongod-data-my-cluster-name-rs0-0,mongod-data-my-cluster-name-rs0-1,mongod-data-my-cluster-name-rs0-2"}
2024-04-29T13:53:16.294Z INFO Deleting statefulset {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "8fbc9204-1552-4237-ac47-454eaf88557c", "name": "my-cluster-name-rs0"}
2024-04-29T13:53:16.311Z INFO Resizing PVC {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "8fbc9204-1552-4237-ac47-454eaf88557c", "name": "mongod-data-my-cluster-name-rs0-0"}
2024-04-29T13:53:16.520Z ERROR Reconciler error {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "8fbc9204-1552-4237-ac47-454eaf88557c", "error": "reconcile statefulsets: reconcile StatefulSet for rs0: reconcile PVCs for my-cluster-name-rs0: resize volumes if needed: update persistentvolumeclaim/mongod-data-my-cluster-name-rs0-0: persistentvolumeclaims \"mongod-data-my-cluster-name-rs0-0\" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize", "errorVerbose": "reconcile StatefulSet for rs0: reconcile PVCs for my-cluster-name-rs0: resize volumes if needed: update persistentvolumeclaim/mongod-data-my-cluster-name-rs0-0: persistentvolumeclaims \"mongod-data-my-cluster-name-rs0-0\" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileReplsets\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:463\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:401\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nreconcile statefulsets\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:403\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:222
2024-04-29T13:53:16.977Z INFO PVC resize completed {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "5bcaa49c-95b9-4320-a9d9-eedb90b56883"}
2024-04-29T13:53:17.556Z INFO Cluster state changed {"controller": "psmdb-controller", "object": {"name":"my-cluster-name","namespace":"test"}, "namespace": "test", "name": "my-cluster-name", "reconcileID": "5bcaa49c-95b9-4320-a9d9-eedb90b56883", "previous": "error", "current": "ready"}

We seem to have an issue (or two) with PVC/storage resizing.

1. When you try to resize to value to which k8s will not be able to resize, resize will fail but the cluster status will be ready and you can have some PVC with one size and others with old size.

$ k get psmdb
NAME ENDPOINT STATUS AGE
my-cluster-name my-cluster-name-mongos.test.svc.cluster.local ready 7m16s

$ k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
mongod-data-my-cluster-name-cfg-0 Bound pvc-c26d0478-6b2e-45e7-b8c4-64b20e0e3991 1Gi RWO standard-rwo <unset> 9m28s
mongod-data-my-cluster-name-cfg-1 Bound pvc-dab8b58a-2cea-4b40-af33-e3be54941802 1Gi RWO standard-rwo <unset> 9m3s
mongod-data-my-cluster-name-cfg-2 Bound pvc-7e26673f-9d7d-4a23-ba0e-54e93d8ad27a 1Gi RWO standard-rwo <unset> 8m29s
mongod-data-my-cluster-name-rs0-0 Bound pvc-7cc28897-c1a0-42fc-9940-4c6022efa968 1Gi RWO standard-rwo <unset> 9m27s
mongod-data-my-cluster-name-rs0-1 Bound pvc-a60edc32-8433-47be-85ec-c4dff6e93a32 1Gi RWO standard-rwo <unset> 9m2s
mongod-data-my-cluster-name-rs0-2 Bound pvc-634ab2e0-28a9-437f-a701-3149158d01d9 10000Gi RWO standard-rwo <unset> 8m38s

2. If you try to downsize there will be some error, but it's not very helpful (and I'm not even sure if we can do something about this)

message: 'Error: reconcile StatefulSet for rs0: update StatefulSet my-cluster-name-rs0:
StatefulSet.apps "my-cluster-name-rs0" is invalid: spec: Forbidden: updates to
statefulset spec for fields other than ''replicas'', ''ordinals'', ''template'',
''updateStrategy'', ''persistentVolumeClaimRetentionPolicy'' and ''minReadySeconds''
are forbidden'

Hello @Sergey Pronin

I believe that just got merged here actually allows for this! With this, you can simply use pvc-autoresizer annotations to allow for it to autoscale the Percona PVCs when they get too full based on your custom needs. I would see it as an extra entry in the documentation (that I'm willing to create!), unless there's something I'm missing and you need to do more internally than simply resize the PVC which would require to implement your own automated storage scaling solution.

Semi-related: I haven't had much success modifying my CR to update them, unlike what was mentioned here. For example if I create one with with 10Gi of request storage, wait for it to be ready and then modify it to 20Gi, the operator would throw this error:

Making any further change to the CR impossible (even delete the cluster) as it stays stuck in this error state, until I revert the CR changes and apply them. It's probably fine to leave the CR with the initial storage values, just thought it was worth mentioning.

Hope this was helpful!

PS: The permission "Browse Users & Groups" is required to be able to properly @ someone, so maybe it's something worth considering to grant to the "jira-users" group (or me )? I understand if it's a security concern though.