Problem:

Inside compress_write and encrypt_write we take thd->ctrl_mutex to
validate only one thread is writting at the time. In case of failure at
xb_crypt_write_chunk / ds_write / write_uint64_le or write_uint32_le we
will terminate the threads without releasing the mutex the thread owns.
In case other thread is waiting on this mutex, it will hang forever.

Fix:

Ensure threads release ctrl_mutex and data_mutex as part of error
handling.
Also added capabilities to detect if we are running on a debug binary.
Note that broken pipe test will not skip in case of non debug. Because
the test for --encrypt can be easily reproducible.

Confirmed with , this issue (at least the way it's reported here) doesn't affect 8.0. We don't use data_mutex nor ctrl_mutex on 8.0.