SST fails when cleaning the datadir

Description

Environment

None

AFFECTED CS IDs

CS0038717

Activity

Show:
Julia Vural
updated the Sprint
yesterday
MySQL Sprint July 2025
MySQL Sprint August 2025
Julia Vural
changed the Assignee
yesterday
aibek.bukabayev
Unassigned
Julia Vural
changed the Assignee
June 25, 2025 at 11:14 AM
Unassigned
aibek.bukabayev
Kamil Holubicki
updated the Sprint
June 25, 2025 at 8:30 AM
None
MySQL Sprint July 2025
Kamil Holubicki
updated the Labels
April 9, 2025 at 5:55 AM
None
reviewed
Percona Bot
updated the AFFECTED USER LEVEL
March 6, 2024 at 8:37 PM
None
Internal
Percona Bot
updated the AFFECTED CS IDs
March 6, 2024 at 8:37 PM
None
CS0038717
Secure Custom Fields for Jira (Security & Permission)
updated the AFFECTED USER LEVEL
December 20, 2023 at 2:27 PM
None
This field doesn't support this UI
Secure Custom Fields for Jira (Security & Permission)
updated the AFFECTED ZENDESK IDs
December 20, 2023 at 2:27 PM
None
This field doesn't support this UI
Juan Arruti
changed the Status
August 22, 2023 at 2:07 AM
New
Open
Juan Arruti
updated the Description
August 22, 2023 at 2:07 AM
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance: {code:java} find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1  -regex $cpat  -prune  -o -exec rm -rfv {} 1>/dev/null \+{code} If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.  How to repeat: 1. Create 10 databases and fill them with empty tables using sysbench: {code:java} # database 1 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare # database 2 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare ... # database 10 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare {code} 2. On a second node change the my.cnf option file with the following values: {code:java} datadir                         =       /mysql_data_directory innodb_data_home_dir            =       /mysql_data_directory innodb_log_group_home_dir       =       /mysql_data_directory innodb_undo_directory           =       /mysql_data_directory{code} 3. Restart the instance and let SST fills the new data directory {code:java} $ systemctl restart mysql{code} 4. Stop the instance and remove the grastate.dat file {code:java} $ systemctl stop mysql $ rm /var/lib/mysql/grastate.dat{code} 5. Start the instance, SST fails with the following error: {code:java} 2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST......... 2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory 2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory 2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1 2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory) 2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file. 2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code} It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.  I've workaround this issue by omitting repeated files and directories using the following command: {code:java} $ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat  -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance: {code:java} find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1  -regex $cpat  -prune  -o -exec rm -rfv {} 1>/dev/null \+{code} If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.  How to repeat: 1. Create 10 databases and fill them with empty tables using sysbench: {code:java} # database 1 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare # database 2 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare ... # database 10 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare {code} 2. On a second node change the my.cnf option file with the following values: {code:java} datadir                         =       /mysql_data_directory innodb_data_home_dir            =       /mysql_data_directory innodb_log_group_home_dir       =       /mysql_data_directory innodb_undo_directory           =       /mysql_data_directory{code} 3. Restart the instance and let SST fills the new data directory {code:java} $ systemctl restart mysql{code} 4. Stop the instance and remove the grastate.dat file {code:java} $ systemctl stop mysql $ rm /var/lib/mysql/grastate.dat{code} 5. Start the instance, SST fails with the following error: {code:java} 2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST......... 2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory 2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory 2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1 2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory) 2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file. 2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code} It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.  I've workaround this issue by omitting repeated files and directories using the following command: {code:java} $ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat  -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
Juan Arruti
updated the Description
August 22, 2023 at 2:06 AM
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance: {code:java} find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1  -regex $cpat  -prune  -o -exec rm -rfv {} 1>/dev/null \+{code} If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.  How to repeat: 1. Create 10 databases and fill them with empty tables using sysbench:   {code:java} # database 1 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare # database 2 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare ... # database 10 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare {code}   2. On a second node change the my.cnf option file with the following values: {code:java} datadir                         =       /mysql_data_directory innodb_data_home_dir            =       /mysql_data_directory innodb_log_group_home_dir       =       /mysql_data_directory innodb_undo_directory           =       /mysql_data_directory{code} 3. Restart the instance and let SST fills the new data directory {code:java} $ systemctl restart mysql{code} 4. Stop the instance and remove the grastate.dat file {code:java} $ systemctl stop mysql $ rm /var/lib/mysql/grastate.dat{code} 5. Start the instance, SST fails with the following error: {code:java} 2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST......... 2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory 2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory 2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1 2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory) 2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file. 2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code} It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.  I've workaround this issue by omitting repeated files and directories using the following command: {code:java} $ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat  -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
The wsrep_sst_xtrabackup-v2 script runs the following command to clean the existing instance: {code:java} find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1  -regex $cpat  -prune  -o -exec rm -rfv {} 1>/dev/null \+{code} If someone has explicitly set the same directory for some of the above variables, SST may fail when cleaning the directory due to a find command error.  How to repeat: 1. Create 10 databases and fill them with empty tables using sysbench: {code:java} # database 1 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test1 prepare # database 2 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test2 prepare ... # database 10 sysbench --test=/usr/share/sysbench/tests/include/oltp_legacy/oltp.lua --report-interval=1 --oltp-tables-count=1000 --oltp-table-size=0 --max-time=1800 --oltp-read-only=off --max-requests=0 --num-threads=8 --rand-type=uniform --db-driver=mysql --mysql-user=sysbench --mysql-password=secret --mysql-db=test10 prepare {code} 2. On a second node change the my.cnf option file with the following values: {code:java} datadir                         =       /mysql_data_directory innodb_data_home_dir            =       /mysql_data_directory innodb_log_group_home_dir       =       /mysql_data_directory innodb_undo_directory           =       /mysql_data_directory{code} 3. Restart the instance and let SST fills the new data directory {code:java} $ systemctl restart mysql{code} 4. Stop the instance and remove the grastate.dat file {code:java} $ systemctl stop mysql $ rm /var/lib/mysql/grastate.dat{code} 5. Start the instance, SST fails with the following error: {code:java} 2023-08-22T01:04:41.698535Z 0 [Note] [MY-000000] [WSREP-SST] Proceeding with SST......... 2023-08-22T01:04:42.180202Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/performance_schema’: No such file or directory 2023-08-22T01:04:42.180248Z 0 [Note] [MY-000000] [WSREP-SST] find: ‘/mysql_data_directory/test3’: No such file or directory 2023-08-22T01:04:42.184350Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:1 2023-08-22T01:05:10.235706Z 0 [Note] [MY-000000] [WSREP-SST] ^Gxbstream: Can't create directory './test8/' (OS errno 2 - No such file or directory) 2023-08-22T01:05:10.235742Z 0 [Note] [MY-000000] [WSREP-SST] /usr/bin/pxc_extra/pxb-8.0/bin/xbstream: failed to create file. 2023-08-22T01:05:10.235754Z 0 [Note] [MY-000000] [WSREP-SST] exit code: 1{code} It looks like the duplicate values make the find command fail. It does not work with a regular /var/lib/mysql data directory, it seems the path should be longer to trigger it; maybe some buffer is filled.  I've workaround this issue by omitting repeated files and directories using the following command: {code:java} $ find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat  -prune -o -print0 | sort -z | uniq -z | xargs -0 rm -rfv --{code}
Juan Arruti
created the Bug
August 22, 2023 at 2:06 AM

Details

Assignee

Reporter

Labels

Needs QA

Affects versions

Priority

Created August 22, 2023 at 2:06 AM
Updated yesterday

Flag notifications