Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Mysql and haproxy pods go into CreateContainerConfigError state

Description

Tested with Everest v1.4.0-rc1 version.

  1. Create an eks cluster with 3 nodes.

  2. Helm Install 1.4.0 with default everest namespace and all db operators.

  3. Create 3 node mysql, mongodb, pg databases in everest namespace.

  4. Change the admin password using cli.

  5. Add two users using cli, list users, delete a user.

  6. Login as new user and create 1 node mysql, mongodb, pg databases in everest namespace with monitoring enabled.

  7. The resources are not sufficient for the 1 node mysql db to come up, scale up the eks cluster to add more resources.

  8. The 1 node mysql db still does not come up. The pmm container is not able to come up in the mysql db and remains in the CreateContainerConfigError state.

The mysql pod events show this error Error: couldn't find key root in Secret everest/internal-mysql-1node

haproxy and pxc pod status

In the UI, the mysql database remains in Initializing status

Pxc container status

Adding logs for pxc operator, and pod description of haproxy, pxc.

Note: I was able to reproduce the issue with two mysql 1 node clusters. The kubernetes cluster had 2.75 CPU and after creating the mysql database with monitoring enabled the haproxy, mysql pods went into CreateContainerConfigError state.

Slack discussion:

Environment

None

Attachments

5

Activity

Show:

Manish Chawla December 27, 2024 at 9:28 AM

Tested with Everest v1.4.0-rc4 version.

Install using helm with default everest namespace.

Created several mysql databases with scheduled backps, pitr and monitoring enabled. The databases were able to come up successfully and no errors were observed.

Install the a1 db namespace with only pxc and created more mysql databases.

Also created mysql databases when the cluster resources were insufficient. The mysql db pods go into Pending status and after resources are available, they transition to the Running status.

These were the mysql databases created

The issue is fixed, hence closing it.

Diogo Recharte December 24, 2024 at 9:53 AM
Edited

Looks like the changes introduced in triggered the PXCO bug that causes this issue.

Because this issue is more severe than we are going to revert and move back to TODO. Once is fixed we can then re-introduce the changes that fix .

Manish Chawla December 24, 2024 at 7:53 AM

This issue got reproduced in Everest v1.4.0-rc3 version. The steps are:

Create an eks cluster with 3 nodes.

Install everest using helm without creating any db namespace.

Install a1 namespace - only mysql operator.

Create 3 node mysql database in a1 namespace.

Create a 1 node mysql database, add a monitoring instance and enable monitoring in the db wizard. The mysql and haproxy pods go into CreateContainerConfigError state

On the UI, the mysql database remains in Initializing status.

Mayank Shah December 18, 2024 at 9:54 AM

Seems like a PXC operator issue, created a detailed report here -

Manish Chawla December 17, 2024 at 3:54 PM
Edited

On deleting the internal-mysql-gca secrets, the mysql database comes up. It seems that the secrets are not correctly generated in this scenario.

Unresolved

Details

Assignee

Reporter

Labels

Fix versions

Affects versions

Priority

Smart Checklist

Created December 17, 2024 at 3:38 PM
Updated January 8, 2025 at 4:58 PM