"Encryption can't find master key" after SST when keyring_file is used if keyring.backup file exists

Description

When keyring_file plugin is used, if keyring.backup file exists when SST is running, mysqld will fail to decrypt tables upon the startup after SST completion. Technically, SST completes successfully, but the data in encrypted tables is useless.

Even though presence of keyring.backup file is unexpected, and this issue is probably rare in the wild, SST script should delete any leftover keyring.backup file.

Easiest way to reproduce is as follows (assuming keyring-file-data=/var/lib/mysql-keyring/keyring and there are encrypted tables with data):
1. Start a cluster and pick a node. Following actions are on that node.
2. Stop mysqld
3. Run cp -ip /var/lib/mysql-keyring/keyring{,.backup}
4. Force SST
5. After SST, observe in mysqld logs:

2021-03-11T08:59:52.980388Z 0 [ERROR] InnoDB: Encryption can't find master key, please check the keyring plugin is loaded. 2021-03-11T08:59:52.980404Z 0 [ERROR] InnoDB: Encryption information in datafile: ./demo/tde.ibd can't be decrypted, please check if a keyring plugin is loaded and initialized successfully. 2021-03-11T08:59:52.980420Z 0 [Warning] InnoDB: Ignoring tablespace `demo/tde` because it could not be opened.

Environment

None

AFFECTED CS IDs

CS0015929

Smart Checklist

Activity

puneet.kaushik November 16, 2021 at 11:32 AM

MTR test also passed 

puneet.kaushik November 16, 2021 at 11:24 AM

Verified fix in PXC 8.0.25

Noemi Lapresta July 15, 2021 at 12:51 PM

Verified fix in PXC 5.7.34-31.51. MTR test passes.

Sergey Kuzmichev March 11, 2021 at 12:20 PM

The mechanism of issue seems to be the following:

Done

Details

Assignee

Reporter

Needs Review

Yes

Time tracking

2d 1h 27m logged57m remaining

Affects versions

Priority

Smart Checklist

Created March 11, 2021 at 12:19 PM
Updated March 6, 2024 at 9:13 PM
Resolved July 1, 2021 at 12:35 PM

Flag notifications