Handle max_allowed_packet setting being incorrect in orchestrator

Description

I recently saw this problem reported by orchestrator 3.2.6-5

 

I have orchestrator configured with the following settings:

and it seems that under some circumstances the packet sent by orchestrator is too large.  The MySQL server running as a backend to orchestrator runs 8.0.30 and its current configuration shows max_allowed_packet = 64M.

The go driver's hard-coded setting is: max_allowed_packet=4MB, but it's possible to use a value of 0 but this triggers each connection to query the server's setting and is not really desirable.

There appears to be no way in orchestrator to provide a specific value to use for max_allowed_packet and the logical setting would be via configuration management to make it match the server's setting.

So maybe it might be sensible to be able to have an orchestrator parameter MySQLOrchestratorMaxAllowedPacket which could be fed to the DSN which orchestrator uses when opening a new connection so the setting is configured properly.

The only thing I see as being a bit odd is the go driver is indicating that the server's max_allowed_packet setting is too small and the error message suggests fixing the setting on the server.  If the 4MB setting that I believe is being used by orchestrator is correct the actual error message is incorrect as the server's size is already larger.  Perhaps that should be reported upstream as an error whose message is not entirely correct?

Either way orchestrator does not provide a way to adjust this so the configuration setting looks to be needed if this error is seen.

Environment

None

AFFECTED CS IDs

CS0030517

Smart Checklist

Activity

Show:

Simon Mudd November 20, 2023 at 9:21 AM

Yes, but while connections are persistent there will be reconnections. I currently have the MySQLConnectionLifetimeSeconds set at 240.  This ensures that orchestrator will reconnect to the backend after a period of time. Context: firewall changes to the MySQL hosts may stop allowing new connections from orchestrator. Orchestrator will be unaware of this and if it wants to relocate or handle failover scenarios then a new connection sent to orchestrator may not work and this will cause problems. So the connections to the servers monitored by MySQL are not permanent but are only persisted for a short period of time.

So this request was making it possible to configure the max_allowed_packet setting when talking to MySQL backends as currently the value is determined exclusively by the go driver. I stand by my request that for some commands sent by orchestrator this setting may be too small and it would be good to be able to provide an appropriate setting.

Yves Trudeau November 15, 2023 at 7:56 PM
Edited

Hi Simon,

   You seem to assume that the variable is queried every time there is a poll. Orchestrator uses persistent connections, this will be done only when connecting for the first time, not for every poll.

 

Simon Mudd August 3, 2023 at 11:00 AM

For context on 2 clusters I manage (in active/active mode):

  • I see their individual 95 percentile poll latency metrics at 360ms for one cluster and 620ms for the other one, so poll latency is important as with this sort of latency orchestrator can be busy doing a large number of polls in parallel.  Doing extra queries which increase this by 10-30 ms for no good reason is not helpful I believe.

Simon Mudd August 3, 2023 at 10:52 AM

Hi Yves,

My reasoning is the following:

  • I know some people using orchestrator with the poll period set at 1 second.

  • I know some people using orchestrator polling thousands of servers in the poll period so orchestrator is pretty busy

  • so on heavily used systems adding extra queries and extra round trips is really not a good idea, this makes the poll period last longer

  • remember the query time consists of 2 parts and if you monitor multi AZ/DC systems then even if the query latency on the MySQL server in question is short the latency to send the query and get back a response may be much longer.

So on quiet systems letting orchestrator do an extra query to figure out the appropriate max_allowed_packet setting, and then setting it on the server with a session based variable change may be fine, for busy systems this extra overhead is unnecessary especially if you know what the value is anyway and can thus configure it in the connect DSN.

 

That seems to lead to an optional setting in orchestrator for such connections to allow for 0 (do an extra query to figure out the server's value) or 999 (an explicit value to use as the XXXMaxAllowedPacket setting.  By all means make the default 0, but provide a way for the user to be able to overwrite this with a specific setting which reduces the extra latency for each polled connection.

Yves Trudeau July 25, 2023 at 7:09 PM

Hi,

   regarding this:

Why do you think it is not really desirable?  It is just at most a network round-trip and a very simple query.

Done

Details

Assignee

Reporter

Needs Review

Yes

Needs QA

Yes

Components

Priority

Smart Checklist

Created September 27, 2022 at 4:39 PM
Updated July 11, 2024 at 5:40 PM
Resolved March 1, 2024 at 12:37 PM