Backup fails with Vault related error
Description
Environment
Attachments
Activity
Satya Bodapati March 20, 2025 at 11:29 AM
From the data you shared, it looks like a race condition. This corrupts the encryption information required for a tablespace, and so the backup fails., do you have the steps to set up the vault and the k8s cluster? Will I need a license to run the vault server?
fkorodi March 19, 2025 at 3:21 PM
it is random, not related exactly this two table in this demo
every table encrypted in this demo database
We are using Vault in a K8S cluster, I can’t use or configure another one. We are using kv-v1
Satya Bodapati March 19, 2025 at 3:13 PM
2025-03-04T18:23:52.007666-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./xtra_test/test_18139.ibd with space id 18150. Will check if encrytion key has been parsed at the end of backup.2025-03-04T18:23:52.049523-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./xtra_test/test_1006.ibd with space id 1017. Will check if encrytion key has been parsed at the end of backup.
Few questions:
1. Does it happen only with these two tables?
2. Are there other encrypted tables?
3. If it happens only with these two tables, is it possible to rebuild them (OPTIMIZE TABLE?)?
Satya Bodapati March 19, 2025 at 3:10 PM
we use a docker image to simulate a vault server. Is it possible for you use this and reproduce the issue?
docker run -d --security-opt seccomp=unconfined --cap-add=NET_ADMIN --rm -p 8200:8200 --name vault satyapercona/vault:latest
docker logs vault
Can you use this config, to set the server using this config, create encrypted tables, and take backup with xtrabackup?
fkorodi March 19, 2025 at 2:57 PM
It is too large for pastebin. The most relevant information is this (only the Streaming and Done: Streaming lines are removed):
2025-03-04T18:23:49.801737-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --open_files_limit=5000002025-03-04T18:23:49.801960-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --user=MHTBACKUPUSER --password=* --port=3308 --socket=/var/lib/mysql-3308/mysql.sock --backup=1 --extra-lsndir=/var/lib/mysql-3308/backup/Tue --stream=xbstream --parallel=1 --close_files=1 --lock-ddl=1 --slave-info=1 --safe-slave-backup=1 --target-dir=/var/lib/mysql-3308/backup/Tuextrabackup version 8.0.35-31 based on MySQL server 8.0.35 Linux (x86_64) (revision id: 55ec21d7)Can't locate English.pm in @INC (you may need to install the English module) (@INC contains: /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at - line 3.BEGIN failed--compilation aborted at - line 3.2025-03-04T18:23:49.819313-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: MHTBACKUPUSER, password: set, port: 3308, socket: /var/lib/mysql-3308/mysql.sock2025-03-04T18:23:49.850706-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.36-282025-03-04T18:23:50.028626-00:00 0 [Note] [MY-011825] [Xtrabackup] Not checking slave open temp tables for --safe-slave-backup because host is not a slave2025-03-04T18:23:50.028953-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...2025-03-04T18:23:50.042672-00:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise().2025-03-04T18:23:50.042765-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql-3308/data/2025-03-04T18:23:50.042796-00:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 500000, set to 50000002025-03-04T18:23:50.340181-00:00 0 [Warning] [MY-011825] [Xtrabackup] close-files specified. Use it at your own risk. If there are DDL operations like table DROP TABLE or RENAME TABLE during the backup, inconsistent backup will be produced.2025-03-04T18:23:50.340388-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration:2025-03-04T18:23:50.340415-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .2025-03-04T18:23:50.340433-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:100M;ibdata2:100M:autoextend2025-03-04T18:23:50.340495-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = ./2025-03-04T18:23:50.340514-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 22025-03-04T18:23:50.340533-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 503316482025-03-04T18:23:50.344281-00:00 0 [Note] [MY-011825] [Xtrabackup] Added plugin 'keyring_vault.so' to load list.2025-03-04T18:23:50.402203-00:00 0 [Warning] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Probing circeo-partner1-kv for being a mount point unsuccessful - skipped.'2025-03-04T18:23:50.441005-00:00 0 [Warning] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Probing circeo-partner1-kv/tde-keys for being a mount point unsuccessful - skipped.'2025-03-04T18:23:50.483657-00:00 0 [Warning] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Probing circeo-partner1-kv/tde-keys/dbupgrade-fra02-3308 for being a mount point unsuccessful - skipped.'2025-03-04T18:23:50.528732-00:00 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded2025-03-04T18:23:50.638833-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: MHTBACKUPUSER, password: set, port: 3308, socket: /var/lib/mysql-3308/mysql.sock2025-03-04T18:23:50.649960-00:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up.2025-03-04T18:23:50.739844-00:00 0 [Note] [MY-011825] [Xtrabackup] Starting to parse redo log at lsn = 3471534252025-03-04T18:23:50.740288-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:51.690187-00:00 0 [Note] [MY-012953] [InnoDB] Disabling background ibuf IO read threads.2025-03-04T18:23:51.740523-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:51.894983-00:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces2025-03-04T18:23:51.895092-00:00 0 [Note] [MY-012204] [InnoDB] Scanning './'2025-03-04T18:23:51.979647-00:00 0 [Note] [MY-012207] [InnoDB] Using 3 threads to scan 20011 tablespace files2025-03-04T18:23:51.982743-00:00 0 [Note] [MY-012208] [InnoDB] Completed space ID check of 2 files.2025-03-04T18:23:51.985130-00:00 0 [Warning] [MY-012091] [InnoDB] Allocated tablespace ID 1 for sys/sys_config, old maximum was 02025-03-04T18:23:52.007538-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Vault Server response does not have "data" member'2025-03-04T18:23:52.007619-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Vault has returned the following error(s): missing client token'2025-03-04T18:23:52.007647-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./xtra_test/test_18139.ibd can't be decrypted, please confirm that keyring is loaded.2025-03-04T18:23:52.007666-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./xtra_test/test_18139.ibd with space id 18150. Will check if encrytion key has been parsed at the end of backup.2025-03-04T18:23:52.049387-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-03-04T18:23:52.049451-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-03-04T18:23:52.049493-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./xtra_test/test_1006.ibd can't be decrypted, please confirm that keyring is loaded.2025-03-04T18:23:52.049523-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./xtra_test/test_1006.ibd with space id 1017. Will check if encrytion key has been parsed at the end of backup.2025-03-04T18:23:52.740819-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:53.741117-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:54.741454-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:55.741762-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:56.742072-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:57.742416-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:58.742777-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:23:59.743096-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:00.743421-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:01.743750-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:02.744031-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:03.744409-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:04.744726-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:05.745138-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:06.745426-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:07.745699-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:08.745996-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:09.746305-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:10.746593-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:11.746855-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:12.747173-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:13.747580-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (347155031)2025-03-04T18:24:14.116004-00:00 0 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_001'.2025-03-04T18:24:14.118832-00:00 0 [Note] [MY-013252] [InnoDB] Using undo tablespace './undo_002'.2025-03-04T18:24:14.122083-00:00 0 [Note] [MY-012910] [InnoDB] Opened 2 existing undo tablespaces.2025-03-04T18:24:14.130625-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_4597.ibd2025-03-04T18:24:14.131988-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_4597.ibd2025-03-04T18:24:14.132651-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_9013.ibd2025-03-04T18:24:14.133109-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_9013.ibd2025-03-04T18:24:14.142499-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_2741.ibd2025-03-04T18:24:14.142936-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_2741.ibd2025-03-04T18:24:14.148486-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_15093.ibd2025-03-04T18:24:14.148819-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_15093.ibd2025-03-04T18:24:14.154821-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_1141.ibd2025-03-04T18:24:14.155059-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_1141.ibd2025-03-04T18:24:14.160742-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_7669.ibd2025-03-04T18:24:14.160962-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_7669.ibd2025-03-04T18:24:14.164418-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_8821.ibd2025-03-04T18:24:14.164630-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_8821.ibd2025-03-04T18:24:14.169682-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_12085.ibd2025-03-04T18:24:14.169855-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_12085.ibd2025-03-04T18:24:14.172665-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_15157.ibd2025-03-04T18:24:14.172830-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_15157.ibd2025-03-04T18:24:14.177349-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_4149.ibd2025-03-04T18:24:14.177518-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_4149.ibd2025-03-04T18:24:14.181752-00:00 2 [Note] [MY-011825] [Xtrabackup] Streaming ./xtra_test/test_9141.ibd2025-03-04T18:24:14.181887-00:00 2 [Note] [MY-011825] [Xtrabackup] Done: Streaming ./xtra_test/test_9141.ibd
...
2025-03-04T18:25:08.417491-00:00 3 [Note] [MY-011825] [Xtrabackup] Streaming performance_schema/persisted_variab_196.sdi to <STDOUT>2025-03-04T18:25:08.418120-00:00 3 [Note] [MY-011825] [Xtrabackup] Done: Streaming performance_schema/persisted_variab_196.sdi to <STDOUT>2025-03-04T18:25:08.508478-00:00 0 [Note] [MY-011825] [Xtrabackup] Finished backing up non-InnoDB tables and files2025-03-04T18:25:08.508559-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing FLUSH NO_WRITE_TO_BINLOG BINARY LOGS2025-03-04T18:25:08.530230-00:00 0 [Note] [MY-011825] [Xtrabackup] Selecting LSN and binary log position from p_s.log_status2025-03-04T18:25:08.538445-00:00 0 [Note] [MY-011825] [Xtrabackup] Streaming /var/lib/mysql-3308/binlog/binlog.000015 to <STDOUT> up to position 1572025-03-04T18:25:08.538515-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Streaming /var/lib/mysql-3308/binlog/binlog.000015 to <STDOUT>2025-03-04T18:25:08.538791-00:00 0 [Note] [MY-011825] [Xtrabackup] Streaming <STDOUT>2025-03-04T18:25:08.538819-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Streaming file <STDOUT>2025-03-04T18:25:08.544793-00:00 0 [Note] [MY-011825] [Xtrabackup] Streaming <STDOUT>2025-03-04T18:25:08.544840-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Streaming file <STDOUT>2025-03-04T18:25:08.551624-00:00 0 [Note] [MY-011825] [Xtrabackup] Streaming <STDOUT>2025-03-04T18:25:08.551666-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Streaming file <STDOUT>2025-03-04T18:25:08.551755-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...2025-03-04T18:25:08.556960-00:00 0 [Note] [MY-011825] [Xtrabackup] The latest check point (for incremental): '347155031'2025-03-04T18:25:08.557002-00:00 0 [Note] [MY-011825] [Xtrabackup] Stopping log copying thread at LSN 3471550312025-03-04T18:25:08.557678-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 18150 is missing encryption information.2025-03-04T18:25:08.557706-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 1017 is missing encryption information.
Details
Assignee
UnassignedUnassignedReporter
fkorodifkorodiNeeds QA
YesPriority
Urgent
Details
Details
Assignee
Reporter
Needs QA
Priority
UrgentSmart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Smart Checklist
We have several MySQL environments with the following configuration:
Rocky Linux 9
Percona Server for MySQL 8.0.36-28
Percona Xtrabackup 8.0.35-32
We are using TDE with Vault setup. Everything works fine if the MySQL instance contains less than 8000 tablespace file (table).
But we have 2 MySQL instance with more than 8000 tablespace file. The Xtrabackup can’t back up every table in this two MySQL instance, because of the following errors:
2025-02-25T08:24:17.063962-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: MHTBACKUPUSER, password: set, port: 3306, socket: /var/lib/mysql/mysql.sock2025-02-25T08:24:17.070605-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.36-282025-02-25T08:24:17.139644-00:00 0 [Note] [MY-011825] [Xtrabackup] Not checking slave open temp tables for --safe-slave-backup because host is not a slave2025-02-25T08:24:17.139790-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...2025-02-25T08:24:17.140612-00:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise().2025-02-25T08:24:17.140645-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql/data/2025-02-25T08:24:17.140657-00:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 500000, set to 50000002025-02-25T08:24:18.078670-00:00 0 [Warning] [MY-011825] [Xtrabackup] close-files specified. Use it at your own risk. If there are DDL operations like table DROP TABLE or RENAME TABLE during the backup, inconsistent backup will be produced.2025-02-25T08:24:18.078797-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration:2025-02-25T08:24:18.078808-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .2025-02-25T08:24:18.078813-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:100M;ibdata2:100M:autoextend2025-02-25T08:24:18.078842-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = ./2025-02-25T08:24:18.078848-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 22025-02-25T08:24:18.078856-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 503316482025-02-25T08:24:18.080330-00:00 0 [Note] [MY-011825] [Xtrabackup] Added plugin 'keyring_vault.so' to load list.2025-02-25T08:24:18.119419-00:00 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded2025-02-25T08:24:18.239281-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: MHTBACKUPUSER, password: set, port: 3306, socket: /var/lib/mysql/mysql.sock2025-02-25T08:24:18.246253-00:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up.2025-02-25T08:24:18.341493-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)2025-02-25T08:24:19.341725-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)2025-02-25T08:24:20.341932-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)2025-02-25T08:24:21.342176-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)2025-02-25T08:24:22.342391-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)2025-02-25T08:24:23.342606-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)2025-02-25T08:24:23.432222-00:00 0 [Note] [MY-012953] [InnoDB] Disabling background ibuf IO read threads.2025-02-25T08:24:23.633510-00:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces2025-02-25T08:24:23.633565-00:00 0 [Note] [MY-012204] [InnoDB] Scanning './'2025-02-25T08:24:24.083279-00:00 0 [Note] [MY-012207] [InnoDB] Using 9 threads to scan 106943 tablespace files2025-02-25T08:24:24.085861-00:00 0 [Note] [MY-012208] [InnoDB] Completed space ID check of 2 files.2025-02-25T08:24:24.124210-00:00 0 [Warning] [MY-012091] [InnoDB] Allocated tablespace ID 20349174 for U_20230817194925LF13901/IDE_REGISTRATION_TYPE, old maximum was 02025-02-25T08:24:24.126820-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.126862-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.126893-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20220514105651/ACT_HI_ACTINST.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.126905-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20220514105651/ACT_HI_ACTINST.ibd with space id 20335445. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.127205-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.127237-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.127250-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_WEBACC20250216221005ACQSMK/RSK_TASK_AUTHORITY_PROFILE.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.127279-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_WEBACC20250216221005ACQSMK/RSK_TASK_AUTHORITY_PROFILE.ibd with space id 20436644. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.129143-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.129147-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.129179-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20230817183009/DOC_REDIRECT.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.129180-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20230817183009/DOC_REDIRECT.ibd with space id 20345924. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.130088-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.130118-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.130131-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20230817124854LF13901/PRC_PROCESS_CHANGE_LMNT.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.130138-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20230817124854LF13901/PRC_PROCESS_CHANGE_LMNT.ibd with space id 20342021. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.132273-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.132300-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.132313-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_ABCDHGFE/PAR_PARTY_ACCOUNT_RELATIONSHIP.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.132324-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_ABCDHGFE/PAR_PARTY_ACCOUNT_RELATIONSHIP.ibd with space id 20369653. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.134585-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.134633-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.134645-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_WBCC20241230085442CQRGLF15572/PAR_PARTY_EMPLOYER_DATA.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.134656-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_WBCC20241230085442CQRGLF15572/PAR_PARTY_EMPLOYER_DATA.ibd with space id 20409136. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.134586-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.135002-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.135011-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_WBCC20250130202804CQSMKLF15572/PAR_PARTY_TENANT_ORG_UNIT_DATA.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.135029-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_WBCC20250130202804CQSMKLF15572/PAR_PARTY_TENANT_ORG_UNIT_DATA.ibd with space id 20415668. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.136169-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'2025-02-25T08:24:24.136216-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'2025-02-25T08:24:24.136232-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20220514105201/SLS_APP_GOODS_SERVICES_RULE.ibd can't be decrypted, please confirm that keyring is loaded.2025-02-25T08:24:24.136245-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20220514105201/SLS_APP_GOODS_SERVICES_RULE.ibd with space id 20334278. Will check if encrytion key has been parsed at the end of backup.2025-02-25T08:24:24.342778-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)...2025-02-25T08:37:27.552653-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20335445 is missing encryption information.2025-02-25T08:37:27.552661-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20436644 is missing encryption information.2025-02-25T08:37:27.552666-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20345924 is missing encryption information.2025-02-25T08:37:27.552670-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20342021 is missing encryption information.2025-02-25T08:37:27.552675-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20369653 is missing encryption information.2025-02-25T08:37:27.552683-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20409136 is missing encryption information.2025-02-25T08:37:27.552687-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20415668 is missing encryption information.2025-02-25T08:37:27.552691-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20334278 is missing encryption information.We are using the following command to backup the instance:
2025-02-25T08:24:17.056810-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --user=MHTBACKUPUSER --password=* --port=3306 --socket=/var/lib/mysql/mysql.sock --backup=1 --extra-lsndir=/var/lib/mysql/backup/Tue --stream=xbstream --parallel=1 --close_files=1 --lock-ddl=1 --slave-info=1 --safe-slave-backup=1 --target-dir=/var/lib/mysql/backup/TueIn earlier Percona version (5.7) every backup succeeded without any error. Another observation:
In this example there are 9 threads to scan 106943 tablespace files → 8 error observed in the log (8 table with missing encryption information)
In the other database there are 3 threads to scan 22860 tablespace files → 2 error observed in the log (2 table with missing encryption information)
So it is maybe related to the thread usage. If there are no threads to scan the tablespace files, there are no error during the backup. This is not a Vault related issue, the MySQL instance starts without error, every table is accessible for the clients.