TokuDB does not clean up temporary files after crash recovery
General
Escalation
General
Escalation
Description
Working on TDB-28 I found this.
How to repeat:
1. Start Percona Server with TokuDB enabled 2. Run some long-running job. I used sysbench prepare and alter table sbtest1 partition by hash(id) partitions 6; 3. Change max file size limit for mysqld process: prlimit -f65537 -p `cat /home/sveta/build/ps-5.7/mysql-test/var/run/mysqld.1.pid` 4. Server will crash 5. Restart it 6. Crash recovery will happen 7. Database directory will contain temporary files:
{{$ ls -l /home/sveta/build/ps-5.7/mysql-test/var/mysqld.1/data/sbtest/ total 55516 rw-rw--- 1 sveta sveta 65 May 22 15:07 db.opt rw-rw--- 1 sveta sveta 8632 May 22 15:38 sbtest1.frm rw-rw--- 1 sveta sveta 4194304 May 22 23:46 sbtest1_key_k_1_1ae_2_1d.tokudb rw-rw--- 1 sveta sveta 50331648 May 22 23:46 sbtest1_main_1ae_3_1d.tokudb rw-rw--- 1 sveta sveta 65536 May 22 23:46 sbtest1_status_1ae_4_1d.tokudb rw-rw--- 1 sveta sveta 8632 May 22 15:07 #sql-3136_26.frm rw-rw--- 1 sveta sveta 8632 May 22 15:33 #sql-3647_3.frm rw-rw--- 1 sveta sveta 32768 May 22 15:34 _sql_3647_3_key_k_1_1a3_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:34 _sql_3647_3_main_1a3_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:34 _sql_3647_3_status_1a3_1_1d.tokudb rw-rw--- 1 sveta sveta 8632 May 22 15:41 #sql-3743_3.frm rw-rw--- 1 sveta sveta 44 May 22 15:41 #sql-3743_3.par rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p0_key_k_1_1b4_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p0_main_1b4_2_1d.tokudb rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p0_status_1b4_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p1_key_k_1_1b5_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p1_main_1b5_2_1d.tokudb rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p1_status_1b5_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p2_key_k_1_1b6_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p2_main_1b6_2_1d.tokudb rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p2_status_1b6_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p3_key_k_1_1b7_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p3_main_1b7_2_1d.tokudb rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p3_status_1b7_1_1d.tokudb rw-rw--- 1 sveta sveta 8632 May 22 15:53 #sql-38d1_3.frm rw-rw--- 1 sveta sveta 44 May 22 15:53 #sql-38d1_3.par rw-rw--- 1 sveta sveta 8632 May 22 16:01 #sql-3b7c_3.frm rw-rw--- 1 sveta sveta 44 May 22 16:01 #sql-3b7c_3.par rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p0_key_k_1_1c4_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p0_main_1c4_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p0_status_1c4_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p1_key_k_1_1c5_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p1_main_1c5_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p1_status_1c5_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p2_key_k_1_1c6_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p2_main_1c6_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p2_status_1c6_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p3_key_k_1_1c7_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p3_main_1c7_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p3_status_1c7_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p4_key_k_1_1c8_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p4_main_1c8_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p4_status_1c8_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p5_key_k_1_1c9_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p5_main_1c9_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p5_status_1c9_1_1d.tokudb rw-rw--- 1 sveta sveta 8632 May 22 23:42 #sql-5833_3.frm rw-rw--- 1 sveta sveta 44 May 22 23:42 #sql-5833_3.par rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p0_key_k_1_1da_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p0_main_1da_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p0_status_1da_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p1_key_k_1_1db_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p1_main_1db_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p1_status_1db_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p2_key_k_1_1dc_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p2_main_1dc_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p2_status_1dc_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p3_key_k_1_1dd_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p3_main_1dd_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p3_status_1dd_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p4_key_k_1_1de_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p4_main_1de_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p4_status_1de_1_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p5_key_k_1_1df_3_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p5_main_1df_2_1d.tokudb rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p5_status_1df_1_1d.tokudb
}}
Environment
None
Activity
Julia Vural
March 4, 2025 at 9:01 PM
It appears that this issue is no longer being worked on, so we are closing it for housekeeping purposes. If you believe the issue still exists, please open a new ticket after confirming it's present in the latest release.
George Lorch
April 24, 2018 at 7:25 PM
I believe this is the same issue as what I found for . Vlad is going to try to look into it. Since I think these are the same issue I am going to close that one as duplicate and track any work on this here.
George Lorch
May 24, 2017 at 5:33 AM
So which are the temporary files? AFAIK PerconaFT does not create any 'temporary' files at all during recovery. Are there matching file maps in the information_schema.tokudb_file_map table for all files? How does InnoDB behave with file_per_table on in the same situation? Just a gut feel, but this really looks more like a potential server problem when crashing during a non-transactional DDL operation, or possibly some negative interaction/impedance mismatch between the non-transactional DDL and the fully transactional TokuDB/PerconaFT operations.
Medium
Working on TDB-28 I found this.
How to repeat:
1. Start Percona Server with TokuDB enabled
2. Run some long-running job. I used sysbench prepare and
alter table sbtest1 partition by hash(id) partitions 6;3. Change max file size limit for mysqld process:
prlimit -f65537 -p `cat /home/sveta/build/ps-5.7/mysql-test/var/run/mysqld.1.pid`4. Server will crash
5. Restart it
6. Crash recovery will happen
7. Database directory will contain temporary files:
{{$ ls -l /home/sveta/build/ps-5.7/mysql-test/var/mysqld.1/data/sbtest/
total 55516
rw-rw--- 1 sveta sveta 65 May 22 15:07 db.opt
rw-rw--- 1 sveta sveta 8632 May 22 15:38 sbtest1.frm
rw-rw--- 1 sveta sveta 4194304 May 22 23:46 sbtest1_key_k_1_1ae_2_1d.tokudb
rw-rw--- 1 sveta sveta 50331648 May 22 23:46 sbtest1_main_1ae_3_1d.tokudb
rw-rw--- 1 sveta sveta 65536 May 22 23:46 sbtest1_status_1ae_4_1d.tokudb
rw-rw--- 1 sveta sveta 8632 May 22 15:07 #sql-3136_26.frm
rw-rw--- 1 sveta sveta 8632 May 22 15:33 #sql-3647_3.frm
rw-rw--- 1 sveta sveta 32768 May 22 15:34 _sql_3647_3_key_k_1_1a3_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:34 _sql_3647_3_main_1a3_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:34 _sql_3647_3_status_1a3_1_1d.tokudb
rw-rw--- 1 sveta sveta 8632 May 22 15:41 #sql-3743_3.frm
rw-rw--- 1 sveta sveta 44 May 22 15:41 #sql-3743_3.par
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p0_key_k_1_1b4_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p0_main_1b4_2_1d.tokudb
rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p0_status_1b4_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p1_key_k_1_1b5_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p1_main_1b5_2_1d.tokudb
rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p1_status_1b5_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p2_key_k_1_1b6_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p2_main_1b6_2_1d.tokudb
rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p2_status_1b6_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p3_key_k_1_1b7_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 15:41 _sql_3743_3_P_p3_main_1b7_2_1d.tokudb
rw-rw--- 1 sveta sveta 16384 May 22 15:41 _sql_3743_3_P_p3_status_1b7_1_1d.tokudb
rw-rw--- 1 sveta sveta 8632 May 22 15:53 #sql-38d1_3.frm
rw-rw--- 1 sveta sveta 44 May 22 15:53 #sql-38d1_3.par
rw-rw--- 1 sveta sveta 8632 May 22 16:01 #sql-3b7c_3.frm
rw-rw--- 1 sveta sveta 44 May 22 16:01 #sql-3b7c_3.par
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p0_key_k_1_1c4_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p0_main_1c4_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p0_status_1c4_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p1_key_k_1_1c5_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p1_main_1c5_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p1_status_1c5_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p2_key_k_1_1c6_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p2_main_1c6_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p2_status_1c6_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p3_key_k_1_1c7_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p3_main_1c7_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p3_status_1c7_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p4_key_k_1_1c8_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p4_main_1c8_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p4_status_1c8_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p5_key_k_1_1c9_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p5_main_1c9_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:41 _sql_3b7c_3_P_p5_status_1c9_1_1d.tokudb
rw-rw--- 1 sveta sveta 8632 May 22 23:42 #sql-5833_3.frm
rw-rw--- 1 sveta sveta 44 May 22 23:42 #sql-5833_3.par
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p0_key_k_1_1da_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p0_main_1da_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p0_status_1da_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p1_key_k_1_1db_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p1_main_1db_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p1_status_1db_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p2_key_k_1_1dc_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p2_main_1dc_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p2_status_1dc_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p3_key_k_1_1dd_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p3_main_1dd_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p3_status_1dd_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p4_key_k_1_1de_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p4_main_1de_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p4_status_1de_1_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p5_key_k_1_1df_3_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p5_main_1df_2_1d.tokudb
rw-rw--- 1 sveta sveta 32768 May 22 23:42 _sql_5833_3_P_p5_status_1df_1_1d.tokudb
}}