since 8.0.37 and telemetry mysql cannot restart without kill -9

Description

hi,

 

since we upgraded to 8.0.37 we got a lot of issues on mysql from perconna. The telemetry daemon started to launch as a daemon running all the time and we had to stop it and opt out of this. So we stop the telemetry daemon and mask the service as we cant deinstallit without deinstalling the mysql daemon.

then if you add the percona_telemetry_disable=1 to the cnf and restart mysql it just crash and stay. you cannot stop mysql apart a kill -9 and:

 

2024-08-25T03:20:21.524529Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-08-26T03:20:21.524791Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-08-27T03:20:21.524994Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-08-28T03:20:21.525214Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-08-29T03:20:21.525439Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-08-30T03:20:21.526970Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-08-31T03:20:21.527174Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-01T03:20:21.527767Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-02T03:20:21.528537Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-03T03:20:21.529330Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-04T03:20:21.530240Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-05T03:20:21.531375Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-06T03:20:21.532122Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-07T03:20:21.532743Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-08T03:20:21.532992Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-09T03:20:21.533727Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-10T03:20:21.534571Z 0 [Warning] [MY-011071] [Server] Component percona_telemetry reported: 'Collecting db_instance_id failed. It may be caused by server still initializing.'
2024-09-10T07:13:06.042150Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.37-29).
2024-09-10T07:13:08.056554Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 235024 user: 'root'.
2024-09-10T07:13:08.056639Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 419936 user: 'root'.
2024-09-10T07:13:08.056644Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 538705 user: 'root'.
2024-09-10T07:13:08.056647Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 15946 user: 'root'.
2024-09-10T07:13:08.056651Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 243490 user: 'root'.
2024-09-10T07:13:08.056655Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 632379 user: 'preprodUSR'.
2024-09-10T07:13:08.056696Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 57747 user: 'root'.
2024-09-10T07:13:08.056700Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 361083 user: 'root'.
2024-09-10T07:13:08.056703Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 556139 user: 'root'.
2024-09-10T07:13:08.056707Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 628515 user: 'root'.
2024-09-10T07:13:08.056710Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 483724 user: 'root'.
2024-09-10T07:13:08.056714Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 103325 user: 'root'.
2024-09-10T07:13:08.056717Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 7990 user: 'root'.
2024-09-10T07:13:08.056723Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 147502 user: 'root'.
2024-09-10T07:13:08.056727Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 547654 user: 'root'.
2024-09-10T07:13:08.056730Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 187687 user: 'root'.
2024-09-10T07:13:08.056733Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 252231 user: 'root'.
2024-09-10T07:13:08.056736Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 303687 user: 'root'.

 

telemetry is read only and you cannot restart mysql if you change the cnf. My guess here is that it try to contact the telemetry that is not there and so it just crash the whole service. Just a guess but as since the appearance of telemetry all goes wild i think its the usual suspect

 

best reagrds,

Ghislain.

Environment

debian 10 and debian12 machines on amd64 tested, same issue

Activity

Show:

Kamil Holubicki October 10, 2024 at 4:02 PM
Edited

HI ,

Thank you for your feedback. That means that your issue is exactly the same as .

The issue will be fixed in 8.0.40.

BTW, This is really not a spying process :). We are not spying on any user data; we just want to learn more about the way our products are used. This is to provide better services. You can learn more about Percona Telemetry here. Of course, at any time, you can disable the telemetry if that better suits you.

ghislain October 10, 2024 at 7:49 AM

not exactly,

what i said is that i cannot start the version prior to that adding of the spy system. i meant i cannot proactively disable it ( i didnt know about the loose_xxx possibility) and so an upgrade would automatically enable the spy agent and start building thread and crash the mysql server when i try to restart it (well stall it not crash).

If i use loose_ xxx then all is working i can disable it and prevent it from starting if i upgrade to the version that added it.

I could also use the ENV variable but as is came as a surprise package without prior knowledge if you dont follow the blog daily we have been cautgh by surprise with it. when we saw this it was too late and puppet upgraded the servers making them all impossible to restart without kill -9.

So no issue a start. The issue was when we tried to stop or restart the server as it would never close and kill -9 of mysqld was the only way if the agent was running for too long ( we had case where it eventualy restarted after one hour or so but i cannot block production server for so long without even being sure it will restart later).

hopes that clarify, i dont know when i can do another test with a root account if i can this week i will post the result here.

regards,

Ghislain.

Kamil Holubicki October 10, 2024 at 7:30 AM

Hi , As said above, PS-9453 explains the problem with server shutdown.

What is still unclear to me is that you said the server could not start if you add percona_telemetry_disable=1 to my. cnf. Could you provide the coredump from the server in a faulty/stuck state that happens during the startup?

 

Yet another question: Would it be possible to check with standard root user present?

Aaditya Dubey October 10, 2024 at 7:17 AM

Hi

Thank you for the updates.
This issue is now verified, and it could be linked with

However, we are keeping this ticket OPEN to evaluate the issue further and ensure that the related issue is fixed.

ghislain October 10, 2024 at 5:13 AM

hi kamil, no i rename the root user to mysqlroot on allmost all my servers.

Duplicate

Details

Security Level Help

None

Assignee

Reporter

Regression Issue

Yes

Needs QA

Yes

Affects versions

Priority

Smart Checklist

Created September 10, 2024 at 7:46 AM
Updated October 21, 2024 at 3:52 PM
Resolved October 21, 2024 at 3:50 PM