Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

[Disk Resize] Decimal disk sizes with undefined units cause cluster to fail or get stuck during resize after upgrade to v1.6.0rc1

Description

Steps to reproduce:

  1. Install Everest version v1.5.0

  2. Create MySQL, MongoDB, and PostgreSQL clusters with disk sizes defined using decimal values:

    • For example:

      • MySQL → 25.5 (will convert to Mi)

      • MySQL → 37.333 (will convert to m)

  3. Upgrade Everest to v1.6.0rc1

  4. Edit the DB cluster and increase the disk size to a large number, such as 9223372036854776.00

  5. Save the changes and observe the cluster status and error messages.

 

Actual Outcome:

Cluster with 25.5 (Mi): (pvc shows 26Gi)

  • Enters Down state

  • Error: requested storage (-8589934591Gi) is less than actual storage (26Gi)

Cluster with 37.333 (m): (pvc shows 38Gi)

  • Gets stuck in Resizing Volumes state

  • Errors: Error: failed to deploy pxc: updatePod for pxc: failed to create or update sts: update error: StatefulSet.apps "mysql-<name>-pxc" is invalid: spec: Forbidden
    Error: update persistentvolumeclaim/datadir-<pod>: Operation cannot be fulfilled... the object has been modified…

  • UI Behavior:

    • When editing to a large number, the UI does not send any unit type (e.g., 'Mi', 'Gi', 'm', etc.), resulting in backend misinterpretation.

Expected Outcome:

  • UI should enforce or normalize disk size inputs with proper unit suffixes (e.g., always in Gi).

  • All values should have a type.

 

Extra:

  • This issue is a continuation of behavior seen in: EVEREST-1955

  • After upgrading to v1.6.0rc1, editing these clusters causes storage resizing failures due to:

    • Unit misinterpretation (or absence of a unit)

    • Backend always displaying PVC size in Gi (for MySQL and MongoDB)

  • I have set the priority of this issue as Medium because it is an edge case where the user already has a DB cluster created with a decimal disk value in Everest v1.5.0. After upgrading to the latest version (v1.6.0rc1), where disk size editing is allowed, the user may enter a new value that either:

    • Converts to a decimal Gi value, or

    • Is a very large number.

    Both scenarios can lead to failure or inconsistencies during resizing.

  • Additionally, the UI should not convert the disk size unit from Gi to any other unit (like Mi, m, u, etc.) because:

    • It breaks the type + value compatibility between the frontend and backend, especially for MySQL and MongoDB, which expect consistent Gi units.

    • It causes confusion for users and can lead to unintended behavior or backend errors.

Environment

None

Attachments

1
  • 15 Apr 2025, 02:31 PM

Activity

Manish Chawla 
2 days ago

Discussed in defect triage, this will get fixed by operators. No fix on Everest side will be done. This issue can only be tested by changing CRDs manually.

Yusaf Awan 
April 16, 2025 at 9:59 AM

Findings by

  1. The PG operator rounds requested sizes up to the nearest Gi, e.g. 6400 Mi results in a 7 Gi PVC. If you then scale to any value ≤7 Gi (like 6500 Mi), the PVC size doesn't actually change, so the status doesn't enter "resizing." When scaling beyond 7 Gi, the resizing status appears as expected. This behavior is consistent, but it may be confusing to users since 6400 Mi results in a 7 Gi PVC. A UI improvement could be to reflect the actual size users receive (e.g., show 7 Gi instead of 6400 Mi).

  2. Where the PG operator doesn't even change the state of the DB if the disk change doesn't lead to an increase in Gi, e.g. moving from 6400 Mi to 6500 Mi, the PXCO on the other hand gets the DB onto the error state with the following message:
    Error: failed to deploy pxc: updatePod for pxc: failed to create or update sts: update error: StatefulSet.apps "mysql-9b7-pxc" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Yusaf Awan 
April 15, 2025 at 2:59 PM

I missed testing a scenario where we pass a very big value (ex:9223372036854775808.00), while creating or editing a cluster, either a new cluster remains stuck in initializing or on editing goes into Down state. I missed this as I did not go for such a number, considering Gi as type (we did not have any other type now as we do not have decimals). But I tried this right now on 1.6.0rc-1 and it fails.

Flagged

Details

Assignee

Reporter

Priority

Created April 15, 2025 at 2:31 PM
Updated 2 days ago