Hi guys,

It seems to be a problem or a bug when using encrypted tablespace but for the current specific version of 5.7.28-31 only. This is not a problem as I've tested it with previous versions which are:

5.7.25-28.1.el7
5.7.26-29.1.el7
5.7.27-30.1.el7

The above list of versions works really good with backup and restore scenarios using xtrabackup 2.4.16 and no issues encountered. However, it has a problem with the current version (5.7.28-31) as mentioned earlier when using xtrabackup to take a backup. Here's the sample result below:

mysql -e "create schema sbtest;"

mysql sbtest -e "create table name (id bigint not null auto_increment primary key, pname varchar(50) not null) encryption='y';insert into name values(0, 'one two'),(0, 'three four'), (0, 'five six');"

[root@testnode4 ~]# xtrabackup --user=root --password=MyP@55 --backup --target-dir=/root/backup --datadir=/var/lib/mysql --databases=sbtest
{{xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --server-id=1 --server-id=1 --datadir=/var/lib/mysql }}
{{xtrabackup: recognized client arguments: --password=* --user=root --password=* --backup=1 --target-dir=/root/backup --databases=sbtest }}
191120 17:21:38 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root' (using password: YES).
191120 17:21:38 version_check Connected to MySQL server
191120 17:21:38 version_check Executing a version check against the server...
191120 17:21:38 version_check Done.
191120 17:21:38 Connecting to MySQL server host: localhost, user: root, password: set, port: not set, socket: not set
Using server version 5.7.28-31
xtrabackup version 2.4.16 based on MySQL server 5.7.26 Linux (x86_64) (revision id: c807cfa)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
InnoDB: Number of pools: 1
191120 17:21:38 Added plugin 'keyring_file.so' to load list.
191120 17:21:38 >> log scanned up to (2673253)
xtrabackup: Generating a list of tablespaces
Skipping db: ./mysql
Skipping db: ./sys
InnoDB: Encryption information in datafile: ./sbtest/name.ibd can't be decrypted, please check if a keyring plugin is loaded and initialized successfully.
17:21:38 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2c)[0xd6f54c]
xtrabackup(handle_fatal_signal+0x262)[0xa35d92]
/lib64/libpthread.so.0(+0xf5f0)[0x7f809e5ca5f0]
xtrabackup(_ZN8Datafile19validate_first_pageEPmb+0xfa7)[0xc95eb7]
xtrabackup(_Z31xb_load_single_table_tablespacePKcS0_b+0xbd)[0x72c76d]
xtrabackup(_Z32xb_load_single_table_tablespacesPFbPKcS0_E+0x309)[0x72cd19]
xtrabackup[0x72cf90]
xtrabackup(_Z22xtrabackup_backup_funcv+0xf12)[0x72f262]
xtrabackup(main+0xe15)[0x70b965]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f809c439505]
xtrabackup[0x723974]

and my.cnf
[root@testnode4 ~]# cat /etc/my.cnf
[MYSQLD]
user=mysql
basedir=/usr/
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
pid-file=/var/lib/mysql/mysql.pid
port=3306
log_error=/var/log/mysql/mysqld.log
log_warnings=2
# log_output = FILE
ignore-db-dir=lost+found
early-plugin-load=keyring_file.so
keyring_file_data=/var/lib/mysql-keyring/keyring
server_id=1

[xtrabackup]
keyring_file_data=/var/lib/mysql-keyring/keyring
server_id=1

and mysqld/PXB versions:
[root@testnode4 ~]# mysqld --version
mysqld Ver 5.7.28-31 for Linux on x86_64 (Percona Server (GPL), Release 31, Revision d14ef86)
[root@testnode4 ~]# xtrabackup --version
{{xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --server-id=1 --server-id=1 }}
xtrabackup version 2.4.16 based on MySQL server 5.7.26 Linux (x86_64) (revision id: c807cfa)

Thanks guys.

191120 17:21:04 Added plugin 'keyring_file.so' to load list.
191120 17:21:04 >> log scanned up to (2673253)
xtrabackup: Generating a list of tablespaces
Skipping db: ./mysql
Skipping db: ./sys
InnoDB: Encryption information in datafile: ./sbtest/name.ibd can't be decrypted, please check if a keyring plugin is loaded and initialized successfully.