Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

failed: asio.ssl:67567754: 'invalid padding'

Description

Using Operator 1.6.0 on GKE v1.16.15-gke.4300

I have a fresh install with no data in the database at this point. 

I used helm to install and configure.

I was able to successfully boot the cluster. I made a few additional password updates and configuration updates using the helm upgrade command with while apply a values.yaml file to adjust my configuration.

 

The following error happened when I added a backup configuration to my values.yaml.

 

Here is the output from k9s

│ NAME PF READY RESTARTS STATUS CPU MEM %CPU/R %CPU/L %MEM/R %MEM/L IP NODE AGE │
│ db-pxc-db-proxysql-0 ● 3/3 0 Running 16 28 2 0 3 0 10.0.2.27 gke-blackfisk-dev-01-default-pool-18b2446b-0jhc 9m55s │
│ db-pxc-db-proxysql-1 ● 3/3 0 Running 16 28 2 0 3 0 10.0.0.31 gke-blackfisk-dev-01-default-pool-18b2446b-2mbv 10m │
│ db-pxc-db-proxysql-2 ● 3/3 0 Running 16 27 2 0 2 0 10.0.1.24 gke-blackfisk-dev-01-default-pool-18b2446b-5z7y 10m │
│ db-pxc-db-pxc-0 ● 1/1 0 Running 21 454 3 0 47 0 10.0.0.23 gke-blackfisk-dev-01-default-pool-18b2446b-2mbv 12h │
│ db-pxc-db-pxc-1 ● 1/1 0 Running 12 457 2 0 47 0 10.0.1.19 gke-blackfisk-dev-01-default-pool-18b2446b-5z7y 12h │
│ db-pxc-db-pxc-2 ● 0/1 6 CrashLoopBackOff 0 0 0 0 0 0 10.0.2.26 gke-blackfisk-dev-01-default-pool-18b2446b-0jhc 14m │
│ operator-pxc-operator-6cc67dd6c9-4rx2f ● 1/1 0 Running 8 17 0 0 0 0 10.0.0.20 gke-blackfisk-dev-01-default-pool-18b2446b-2mbv 12h │
│ xb-cron-db-pxc-db-fs-pvc-20201114000016-372f8-mhssk ● 0/1 0 Completed 0 0 0 0 0 0 10.0.2.20 gke-blackfisk-dev-01-default-pool-18b2446b-0jhc 12h │

--------------

Error log from failed pod

│ pxc -------- │
│ pxc 2020-11-14T14:05:03.319698Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.1.19:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding') │
│ pxc This error is often caused by SSL issues. For more information, please see: │
│ pxc https://per.co.na/pxc/encrypt_cluster_traffic │
│ pxc -------- │
│ pxc 2020-11-14T14:05:04.319583Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.0.23:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding') │
│ pxc This error is often caused by SSL issues. For more information, please see: │
│ pxc https://per.co.na/pxc/encrypt_cluster_traffic │
│ pxc -------- │
│ pxc 2020-11-14T14:05:04.819958Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.1.19:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding') │
│ pxc This error is often caused by SSL issues. For more information, please see: │
│ pxc https://per.co.na/pxc/encrypt_cluster_traffic │
│ pxc -------- │
│ pxc 2020-11-14T14:05:05.820818Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.0.23:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding') │
│ pxc This error is often caused by SSL issues. For more information, please see: │
│ pxc https://per.co.na/pxc/encrypt_cluster_traffic │
│ pxc -------- │
│ pxc 2020-11-14T14:05:06.320189Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.1.19:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding') │
│ pxc This error is often caused by SSL issues. For more information, please see: │
│ pxc https://per.co.na/pxc/encrypt_cluster_traffic │
│ pxc -------- │
│ pxc 2020-11-14T14:05:07.320404Z 0 [ERROR] [MY-000000] [Galera] handshake with remote endpoint ssl://10.0.0.23:4567 failed: asio.ssl:67567754: 'invalid padding' ( 67567754: 'error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding') │
│ pxc This error is often caused by SSL issues. For more information, please see: │
│ pxc https://per.co.na/pxc/encrypt_cluster_traffic │
│ pxc -------- │
│ pxc 2020-11-14T14:05:07.324856Z 0 [Note] [MY-000000] [Galera] PC protocol downgrade 1 -> 0 │
│ pxc 2020-11-14T14:05:07.324904Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node │
│ pxc view ((empty)) │
│ pxc 2020-11-14T14:05:07.325074Z 0 [ERROR] [MY-000000] [Galera] failed to open gcomm backend connection: 110: failed to reach primary view (pc.wait_prim_timeout): 110 (Connection timed out) │
│ pxc at gcomm/src/pc.cpp:connect():159 │
│ pxc 2020-11-14T14:05:07.325100Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs_core.cpp:gcs_core_open():220: Failed to open backend connection: -110 (Connection timed out) │
│ pxc 2020-11-14T14:05:07.325463Z 0 [ERROR] [MY-000000] [Galera] gcs/src/gcs.cpp:gcs_open():1700: Failed to open channel 'db-pxc-db-pxc' at 'gcomm://db-pxc-db-pxc-0.db-pxc-db-pxc,db-pxc-db-pxc-1.db-pxc-db-pxc': -110 (Connection timed out) │
│ pxc 2020-11-14T14:05:07.325520Z 0 [ERROR] [MY-000000] [Galera] gcs connect failed: Connection timed out │
│ pxc 2020-11-14T14:05:07.325548Z 0 [ERROR] [MY-000000] [WSREP] Provider/Node (gcomm://db-pxc-db-pxc-0.db-pxc-db-pxc,db-pxc-db-pxc-1.db-pxc-db-pxc) failed to establish connection with cluster (reason: 7) │
│ pxc 2020-11-14T14:05:07.325565Z 0 [ERROR] [MY-010119] [Server] Aborting │
│ pxc 2020-11-14T14:05:07.325948Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.20-11.2) Percona XtraDB Cluster (GPL), Release rel11, Revision 9132e55, WSREP version 26.4.3. │
│ pxc 2020-11-14T14:05:07.326475Z 0 [Note] [MY-000000] [Galera] dtor state: CLOSED │
│ pxc 2020-11-14T14:05:07.326523Z 0 [Note] [MY-000000] [Galera] MemPool(TrxHandleSlave): hit ratio: 0, misses: 0, in use: 0, in pool: 0 │
│ pxc 2020-11-14T14:05:07.330099Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 │
│ pxc 2020-11-14T14:05:07.333769Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 │
│ pxc 2020-11-14T14:05:07.337399Z 0 [Note] [MY-000000] [Galera] apply mon: entered 0 │
│ pxc 2020-11-14T14:05:07.337444Z 0 [Note] [MY-000000] [Galera] cert index usage at exit 0 │
│ pxc 2020-11-14T14:05:07.337456Z 0 [Note] [MY-000000] [Galera] cert trx map usage at exit 0 │
│ pxc 2020-11-14T14:05:07.337462Z 0 [Note] [MY-000000] [Galera] deps set usage at exit 0 │
│ pxc 2020-11-14T14:05:07.337476Z 0 [Note] [MY-000000] [Galera] avg deps dist 0 │
│ pxc 2020-11-14T14:05:07.337483Z 0 [Note] [MY-000000] [Galera] avg cert interval 0 │
│ pxc 2020-11-14T14:05:07.337489Z 0 [Note] [MY-000000] [Galera] cert index size 0 │
│ pxc 2020-11-14T14:05:07.337611Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed. │
│ pxc 2020-11-14T14:05:07.337759Z 0 [Note] [MY-000000] [Galera] wsdb trx map usage 0 conn query map usage 0 │
│ pxc 2020-11-14T14:05:07.337794Z 0 [Note] [MY-000000] [Galera] MemPool(LocalTrxHandle): hit ratio: 0, misses: 0, in use: 0, in pool: 0 │
│ pxc 2020-11-14T14:05:07.339632Z 0 [Note] [MY-000000] [Galera] Flushing memory map to disk... │
│ pxc stream closed │

 

 

Any idea how to resolve this without deleting and reinstalling the entire database?

Environment

None

Smart Checklist

Activity

Sergey Pronin March 10, 2021 at 1:44 PM
Edited

I played with various parameters and configs, but was not able to reproduce the issue.

Closing this one as Aged.

 Please let us know if there is more here and we would gladly help to fix the issue.

Lalit Choudhary February 10, 2021 at 10:47 AM

Hi

Thank you for the report.

Do still see this issue?

Looking at the error looks like an issue with SSL certificates

Jeremy R DeYoung November 23, 2020 at 4:27 PM

I ended up deleting the entire GKE instance and rebuilding everything.

Won't Do

Details

Assignee

Reporter

Labels

Priority

Smart Checklist

Created November 14, 2020 at 2:10 PM
Updated March 5, 2024 at 6:02 PM
Resolved March 10, 2021 at 1:43 PM