Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Add support for major version upgrade

Description

I've tried to reproduce:
https://docs.percona.com/percona-operator-for-postgresql/update.html

  1. Create new empty cluster from cr.yaml with ppg12 or ppg13

  2. load sample database/create any table

  3. Pause cluster, change pgBouncer.size to 0

  4. wait until full shutdown

  5. edit perconapgcluster: change images to ppg13 (for upgrade from ppg12) or to ppg14

In case of 12->13 or 12->14 upgrade, Postgresql pods are not coming to the ready state because wal_keep_segments parameter was removed from postgresql

13->14 upgrade fails due to:
2022-08-26 00:19:57,248 INFO: stderr=2022-08-26 00:19:57.245 UTC [981] FATAL: database files are incompatible with server
2022-08-26 00:19:57.245 UTC [981] DETAIL: The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 14.4.

In-place upgrade could be done with pg_upgrade, but pg_upgrade requires old version of PG to be in the image: https://docs.percona.com/postgresql/14/major-upgrade.html

 

Things to consider:
- manually test with custom extenstions

  • works: - add automated E2E test

  • doesn’t - create ticket for make it work

Environment

None

AFFECTED CS IDs

PS0001414

Smart Checklist

Activity

ege.gunes June 10, 2024 at 8:33 AM

We added a new CRD called PerconaPGUpgrade and a controller to reconcile major upgrades. Example file can be found in deploy/upgrade.yaml.

When you apply this file, you should see the following:
1. Cluster will be paused
2. Cluster will be annotated with pgv2.percona.com/allow-upgrade: <PerconaPGUpgrade.Name>
3. Jobs are created to migrate the data
4. Cluster will be resumed after upgrade finishes

To get information about upgrade our troubleshoot it you can check the status.conditions field. For example:

$ kubectl get perconapgupgrades.pgv2.percona.com 15-to-16 -o yaml apiVersion: pgv2.percona.com/v2 kind: PerconaPGUpgrade metadata: name: 15-to-16 spec: fromPostgresVersion: 15 image: perconalab/percona-postgresql-operator:main-upgrade postgresClusterName: cluster1 toPostgresVersion: 16 status: conditions: - lastTransitionTime: "2024-06-10T07:25:37Z" message: PostgresCluster instances still running observedGeneration: 1 reason: PGClusterNotShutdown status: "False" type: Progressing

Custom extensions
If user has custom extensions installed in their cluster, we need to consider them during upgrades. For this user needs to build and package their custom extension for the major version they want to upgrade. During upgrade we'll install extensions in the upgrade container using the same mechanism, by injecting init containers.

Caveats

  1. pg_upgrade duplicates the data in the same PVC for each major upgrade. Data for old version is not deleted automatically. That means user needs to be sure that their PVC have enough free space to store data.

  2. If something went wrong with upgrade, cluster will remain paused. User needs to delete PerconaPGUpgrade object and resume cluster themself to check what went wrong with upgrade.

Sergey Pronin September 14, 2023 at 12:20 PM

Moving this to a feature/improvement and tenatevely planning to deliver in 2.3.0

Sergey Pronin April 7, 2023 at 7:26 AM

Major version upgrade is not something we target at GA. We added it to MongoDB long after it was GA and it was quite fine. If you think that for PG it is a bit different story - please let me know.

Done

Details

Assignee

Reporter

Labels

Needs Review

Yes

Needs QA

Yes

Needs Doc

Yes

Needs Packaging

No

Story Points

Fix versions

Affects versions

Priority

Smart Checklist

Created August 26, 2022 at 12:41 AM
Updated August 8, 2024 at 9:14 AM
Resolved August 8, 2024 at 9:14 AM