SST fails when cleaning the datadir
General
Escalation
General
Escalation
Description
Environment
None
AFFECTED CS IDs
CS0038717
Activity
Show:
Julia Vural
updated the Sprintyesterday
MySQL Sprint July 2025
MySQL Sprint August 2025
Julia Vural
changed the Assigneeyesterday
aibek.bukabayev
Unassigned
Julia Vural
changed the AssigneeJune 25, 2025 at 11:14 AM
Unassigned
aibek.bukabayev
Kamil Holubicki
updated the SprintJune 25, 2025 at 8:30 AM
None
MySQL Sprint July 2025
Kamil Holubicki
updated the LabelsApril 9, 2025 at 5:55 AM
None
reviewed
Percona Bot
updated the AFFECTED USER LEVELMarch 6, 2024 at 8:37 PM
None
Internal
Percona Bot
updated the AFFECTED CS IDsMarch 6, 2024 at 8:37 PM
None
CS0038717
Secure Custom Fields for Jira (Security & Permission)
updated the AFFECTED USER LEVELDecember 20, 2023 at 2:27 PM
None
This field doesn't support this UI
Secure Custom Fields for Jira (Security & Permission)
updated the AFFECTED ZENDESK IDsDecember 20, 2023 at 2:27 PM
None
This field doesn't support this UI
Juan Arruti
changed the StatusAugust 22, 2023 at 2:07 AM
New
Open
Juan Arruti
updated the DescriptionAugust 22, 2023 at 2:07 AM
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance:
{code:java}
find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>/dev/null \+{code}
If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.
How to repeat:
1. Create 10 databases and fill them with empty tables using sysbench:
{code:java}
# database 1
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare
# database 2
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare
...
# database 10
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare
{code}
2. On a second node change the my.cnf option file with the following values:
{code:java}
datadir = /mysql_data_directory
innodb_data_home_dir = /mysql_data_directory
innodb_log_group_home_dir = /mysql_data_directory
innodb_undo_directory = /mysql_data_directory{code}
3. Restart the instance and let SST fills the new data directory
{code:java}
$ systemctl restart mysql{code}
4. Stop the instance and remove the grastate.dat file
{code:java}
$ systemctl stop mysql
$ rm /var/lib/mysql/grastate.dat{code}
5. Start the instance, SST fails with the following error:
{code:java}
2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory
2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory
2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory)
2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file.
2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code}
It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.
I've workaround this issue by omitting repeated files and directories using the following command:
{code:java}
$ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance:
{code:java}
find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>/dev/null \+{code}
If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.
How to repeat:
1. Create 10 databases and fill them with empty tables using sysbench:
{code:java}
# database 1
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare
# database 2
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare
...
# database 10
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare
{code}
2. On a second node change the my.cnf option file with the following values:
{code:java}
datadir = /mysql_data_directory
innodb_data_home_dir = /mysql_data_directory
innodb_log_group_home_dir = /mysql_data_directory
innodb_undo_directory = /mysql_data_directory{code}
3. Restart the instance and let SST fills the new data directory
{code:java}
$ systemctl restart mysql{code}
4. Stop the instance and remove the grastate.dat file
{code:java}
$ systemctl stop mysql
$ rm /var/lib/mysql/grastate.dat{code}
5. Start the instance, SST fails with the following error:
{code:java}
2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory
2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory
2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory)
2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file.
2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code}
It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.
I've workaround this issue by omitting repeated files and directories using the following command:
{code:java}
$ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
Juan Arruti
updated the DescriptionAugust 22, 2023 at 2:06 AM
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance:
{code:java}
find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>/dev/null \+{code}
If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.
How to repeat:
1. Create 10 databases and fill them with empty tables using sysbench:
{code:java}
# database 1
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare
# database 2
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare
...
# database 10
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare
{code}
2. On a second node change the my.cnf option file with the following values:
{code:java}
datadir = /mysql_data_directory
innodb_data_home_dir = /mysql_data_directory
innodb_log_group_home_dir = /mysql_data_directory
innodb_undo_directory = /mysql_data_directory{code}
3. Restart the instance and let SST fills the new data directory
{code:java}
$ systemctl restart mysql{code}
4. Stop the instance and remove the grastate.dat file
{code:java}
$ systemctl stop mysql
$ rm /var/lib/mysql/grastate.dat{code}
5. Start the instance, SST fails with the following error:
{code:java}
2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory
2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory
2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory)
2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file.
2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code}
It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.
I've workaround this issue by omitting repeated files and directories using the following command:
{code:java}
$ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance:
{code:java}
find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>/dev/null \+{code}
If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.
How to repeat:
1. Create 10 databases and fill them with empty tables using sysbench:
{code:java}
# database 1
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare
# database 2
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare
...
# database 10
sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare
{code}
2. On a second node change the my.cnf option file with the following values:
{code:java}
datadir = /mysql_data_directory
innodb_data_home_dir = /mysql_data_directory
innodb_log_group_home_dir = /mysql_data_directory
innodb_undo_directory = /mysql_data_directory{code}
3. Restart the instance and let SST fills the new data directory
{code:java}
$ systemctl restart mysql{code}
4. Stop the instance and remove the grastate.dat file
{code:java}
$ systemctl stop mysql
$ rm /var/lib/mysql/grastate.dat{code}
5. Start the instance, SST fails with the following error:
{code:java}
2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST.........
2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory
2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory
2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1
2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory)
2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file.
2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code}
It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.
I've workaround this issue by omitting repeated files and directories using the following command:
{code:java}
$ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
Juan Arruti
created the BugAugust 22, 2023 at 2:06 AM
Details
Details
Assignee
Unassigned
UnassignedReporter
Juan Arruti
Juan ArrutiLabels
Needs QA
Yes
Sprint
Affects versions
Priority
Created August 22, 2023 at 2:06 AM
Updated yesterday
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance:
find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>/dev/null \+
If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.
How to repeat:
1. Create 10 databases and fill them with empty tables using sysbench:
# database 1 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare # database 2 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare ... # database 10 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare
2. On a second node change the my.cnf option file with the following values:
datadir = /mysql_data_directory innodb_data_home_dir = /mysql_data_directory innodb_log_group_home_dir = /mysql_data_directory innodb_undo_directory = /mysql_data_directory
3. Restart the instance and let SST fills the new data directory
$ systemctl restart mysql
4. Stop the instance and remove the grastate.dat file
$ systemctl stop mysql $ rm /var/lib/mysql/grastate.dat
5. Start the instance, SST fails with the following error:
2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST......... 2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory 2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory 2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1 2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory) 2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file. 2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1
It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.
I've workaround this issue by omitting repeated files and directories using the following command:
$ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --