We have several MySQL environments with the following configuration:

• Rocky Linux 9

• Percona Server for MySQL 8.0.36-28

• Percona Xtrabackup 8.0.35-32

We are using TDE with Vault setup. Everything works fine if the MySQL instance contains less than 8000 tablespace file (table).

But we have 2 MySQL instance with more than 8000 tablespace file. The Xtrabackup can’t back up every table in this two MySQL instance, because of the following errors:

2025-02-25T08:24:17.063962-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: MHTBACKUPUSER, password: set, port: 3306, socket: /var/lib/mysql/mysql.sock
2025-02-25T08:24:17.070605-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.0.36-28
2025-02-25T08:24:17.139644-00:00 0 [Note] [MY-011825] [Xtrabackup] Not checking slave open temp tables for --safe-slave-backup because host is not a slave
2025-02-25T08:24:17.139790-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...
2025-02-25T08:24:17.140612-00:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise().
2025-02-25T08:24:17.140645-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql/data/
2025-02-25T08:24:17.140657-00:00 0 [Note] [MY-011825] [Xtrabackup] open files limit requested 500000, set to 5000000
2025-02-25T08:24:18.078670-00:00 0 [Warning] [MY-011825] [Xtrabackup] close-files specified. Use it at your own risk. If there are DDL operations like table DROP TABLE or RENAME TABLE during the backup, inconsistent backup will be produced.
2025-02-25T08:24:18.078797-00:00 0 [Note] [MY-011825] [Xtrabackup] using the following InnoDB configuration:
2025-02-25T08:24:18.078808-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_home_dir = .
2025-02-25T08:24:18.078813-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_data_file_path = ibdata1:100M;ibdata2:100M:autoextend
2025-02-25T08:24:18.078842-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_group_home_dir = ./
2025-02-25T08:24:18.078848-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_files_in_group = 2
2025-02-25T08:24:18.078856-00:00 0 [Note] [MY-011825] [Xtrabackup] innodb_log_file_size = 50331648
2025-02-25T08:24:18.080330-00:00 0 [Note] [MY-011825] [Xtrabackup] Added plugin 'keyring_vault.so' to load list.
2025-02-25T08:24:18.119419-00:00 0 [Note] [MY-011825] [Xtrabackup] inititialize_service_handles suceeded
2025-02-25T08:24:18.239281-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: MHTBACKUPUSER, password: set, port: 3306, socket: /var/lib/mysql/mysql.sock
2025-02-25T08:24:18.246253-00:00 0 [Note] [MY-011825] [Xtrabackup] Redo Log Archiving is not set up.
2025-02-25T08:24:18.341493-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)
2025-02-25T08:24:19.341725-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)
2025-02-25T08:24:20.341932-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)
2025-02-25T08:24:21.342176-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)
2025-02-25T08:24:22.342391-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)
2025-02-25T08:24:23.342606-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)
2025-02-25T08:24:23.432222-00:00 0 [Note] [MY-012953] [InnoDB] Disabling background ibuf IO read threads.

2025-02-25T08:24:23.633510-00:00 0 [Note] [MY-011825] [Xtrabackup] Generating a list of tablespaces
2025-02-25T08:24:23.633565-00:00 0 [Note] [MY-012204] [InnoDB] Scanning './'
2025-02-25T08:24:24.083279-00:00 0 [Note] [MY-012207] [InnoDB] Using 9 threads to scan 106943 tablespace files
2025-02-25T08:24:24.085861-00:00 0 [Note] [MY-012208] [InnoDB] Completed space ID check of 2 files.
2025-02-25T08:24:24.124210-00:00 0 [Warning] [MY-012091] [InnoDB] Allocated tablespace ID 20349174 for U_20230817194925LF13901/IDE_REGISTRATION_TYPE, old maximum was 0
2025-02-25T08:24:24.126820-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.126862-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.126893-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20220514105651/ACT_HI_ACTINST.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.126905-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20220514105651/ACT_HI_ACTINST.ibd with space id 20335445. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.127205-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.127237-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.127250-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_WEBACC20250216221005ACQSMK/RSK_TASK_AUTHORITY_PROFILE.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.127279-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_WEBACC20250216221005ACQSMK/RSK_TASK_AUTHORITY_PROFILE.ibd with space id 20436644. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.129143-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.129147-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.129179-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20230817183009/DOC_REDIRECT.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.129180-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20230817183009/DOC_REDIRECT.ibd with space id 20345924. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.130088-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.130118-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.130131-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20230817124854LF13901/PRC_PROCESS_CHANGE_LMNT.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.130138-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20230817124854LF13901/PRC_PROCESS_CHANGE_LMNT.ibd with space id 20342021. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.132273-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.132300-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.132313-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_ABCDHGFE/PAR_PARTY_ACCOUNT_RELATIONSHIP.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.132324-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_ABCDHGFE/PAR_PARTY_ACCOUNT_RELATIONSHIP.ibd with space id 20369653. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.134585-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.134633-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.134645-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_WBCC20241230085442CQRGLF15572/PAR_PARTY_EMPLOYER_DATA.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.134656-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_WBCC20241230085442CQRGLF15572/PAR_PARTY_EMPLOYER_DATA.ibd with space id 20409136. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.134586-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.135002-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.135011-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_WBCC20250130202804CQSMKLF15572/PAR_PARTY_TENANT_ORG_UNIT_DATA.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.135029-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_WBCC20250130202804CQSMKLF15572/PAR_PARTY_TENANT_ORG_UNIT_DATA.ibd with space id 20415668. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.136169-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not parse Vault Server response.'
2025-02-25T08:24:24.136216-00:00 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'Could not read key from Vault. Error while parsing error messages'
2025-02-25T08:24:24.136232-00:00 0 [Note] [MY-011825] [InnoDB] Encryption information in datafile: ./U_20220514105201/SLS_APP_GOODS_SERVICES_RULE.ibd can't be decrypted, please confirm that keyring is loaded.
2025-02-25T08:24:24.136245-00:00 0 [Note] [MY-011825] [Xtrabackup] Failed to decrypt table ./U_20220514105201/SLS_APP_GOODS_SERVICES_RULE.ibd with space id 20334278. Will check if encrytion key has been parsed at the end of backup.
2025-02-25T08:24:24.342778-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (9189851711636)

...

2025-02-25T08:37:27.552653-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20335445 is missing encryption information.
2025-02-25T08:37:27.552661-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20436644 is missing encryption information.
2025-02-25T08:37:27.552666-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20345924 is missing encryption information.
2025-02-25T08:37:27.552670-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20342021 is missing encryption information.
2025-02-25T08:37:27.552675-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20369653 is missing encryption information.
2025-02-25T08:37:27.552683-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20409136 is missing encryption information.
2025-02-25T08:37:27.552687-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20415668 is missing encryption information.
2025-02-25T08:37:27.552691-00:00 0 [ERROR] [MY-011825] [Xtrabackup] Space ID 20334278 is missing encryption information.

We are using the following command to backup the instance:

2025-02-25T08:24:17.056810-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --user=MHTBACKUPUSER --password=* --port=3306 --socket=/var/lib/mysql/mysql.sock --backup=1 --extra-lsndir=/var/lib/mysql/backup/Tue --stream=xbstream --para
llel=1 --close_files=1 --lock-ddl=1 --slave-info=1 --safe-slave-backup=1 --target-dir=/var/lib/mysql/backup/Tue

In earlier Percona version (5.7) every backup succeeded without any error. Another observation:

• In this example there are 9 threads to scan 106943 tablespace files → 8 error observed in the log (8 table with missing encryption information)

• In the other database there are 3 threads to scan 22860 tablespace files → 2 error observed in the log (2 table with missing encryption information)

So it is maybe related to the thread usage. If there are no threads to scan the tablespace files, there are no error during the backup. This is not a Vault related issue, the MySQL instance starts without error, every table is accessible for the clients.