Description

Description:

Deployed Percona xtraDB operator on GKE clusters, pxc pods are failing to start due the following error

PODS:

 
NAME                                               READY   STATUS                  RESTARTS   AGE
cluster1-haproxy-0                                 1/2     Running                 2          8m2s
cluster1-pxc-0                                     0/3     Init:CrashLoopBackOff   6          8m2s
percona-xtradb-cluster-operator-7c4786cb9d-8vl4d   1/1     Running                 0          37m

LOGS:

 

Any suggestion to get over this issue?

Environment

Cluster: GKE

Percona xtraDB Cluster Operator.

Attachments

4

Smart Checklist

Activity

Show:

Denys Kondratenko May 11, 2022 at 6:54 AM

so in my case it is a bug in minikube/host-pathprovisioner:

https://github.com/kubernetes/minikube/issues/12360#issuecomment-1123247475

 

it sets incorrect permissions `775` on a provisioned pathes that attached to volumes on all nodes except the master where it sets it to correct `777` so if pod got scheduled on master node it is ok, but on all others it will get permission denied if it doesn't run as root.

Denys Kondratenko April 3, 2022 at 12:44 PM
Edited

I can reproduce this bug when deploying pxc single node with DBaaS on minikube.

I am developing manifests for k8s and PMM runs on the same minikube:

Then I deploy single node pxc with help of DBaaS UI, and pxc init fails:

when I apply cr-minimal directly:

it works. Can't find anything in logs or anywhere, looks like StatefulSet is similar for both deployments instead of labels and annotations

 

Sridhar Vana. December 24, 2021 at 6:06 PM

Setting runAsGroup: 0 worked on my GKE Cluster.

 

 

Lalit Choudhary December 9, 2021 at 1:45 PM

it's an issue only for new cluster initialization, if cluster/POD already exists with files, change in the  runAsuser/ group will not result in a reported error.

 

Lalit Choudhary December 3, 2021 at 10:13 AM

Hi

Thank you for the report.

I'm able to reproduce the issue and the cause for this issue is change in the  runAsuser/ group, with default 99 user/group I don't see this issue but if I update cr.yaml as follows (podSecurityContext) which I can see in provided cr.yaml files I get permission issue for Pod initialization. 

 

 

with above changes  in cr.yaml following https://www.percona.com/doc/kubernetes-operator-for-pxc/gke.html 

My Test Example:

 

 

 

With Default  99 user/group (nobody) working fine:

 

 

 

Pinned fields
Click on the next to a field label to start pinning.
Details

Assignee

Reporter

Sridhar Vana.

Affects versions

Priority

Smart Checklist
Created September 17, 2021 at 6:14 AM
Updated March 5, 2024 at 5:44 PM
Loading...