Handle max_allowed_packet setting being incorrect in orchestrator
Description
Environment
AFFECTED CS IDs
Smart Checklist
Activity

Simon Mudd November 20, 2023 at 9:21 AM
Yes, but while connections are persistent there will be reconnections. I currently have the MySQLConnectionLifetimeSeconds set at 240. This ensures that orchestrator will reconnect to the backend after a period of time. Context: firewall changes to the MySQL hosts may stop allowing new connections from orchestrator. Orchestrator will be unaware of this and if it wants to relocate or handle failover scenarios then a new connection sent to orchestrator may not work and this will cause problems. So the connections to the servers monitored by MySQL are not permanent but are only persisted for a short period of time.
So this request was making it possible to configure the max_allowed_packet setting when talking to MySQL backends as currently the value is determined exclusively by the go driver. I stand by my request that for some commands sent by orchestrator this setting may be too small and it would be good to be able to provide an appropriate setting.

Yves Trudeau November 15, 2023 at 7:56 PMEdited
Hi Simon,
You seem to assume that the variable is queried every time there is a poll. Orchestrator uses persistent connections, this will be done only when connecting for the first time, not for every poll.

Simon Mudd August 3, 2023 at 11:00 AM
For context on 2 clusters I manage (in active/active mode):
I see their individual 95 percentile poll latency metrics at 360ms for one cluster and 620ms for the other one, so poll latency is important as with this sort of latency orchestrator can be busy doing a large number of polls in parallel. Doing extra queries which increase this by 10-30 ms for no good reason is not helpful I believe.

Simon Mudd August 3, 2023 at 10:52 AM
Hi Yves,
My reasoning is the following:
I know some people using orchestrator with the poll period set at 1 second.
I know some people using orchestrator polling thousands of servers in the poll period so orchestrator is pretty busy
so on heavily used systems adding extra queries and extra round trips is really not a good idea, this makes the poll period last longer
remember the query time consists of 2 parts and if you monitor multi AZ/DC systems then even if the query latency on the MySQL server in question is short the latency to send the query and get back a response may be much longer.
So on quiet systems letting orchestrator do an extra query to figure out the appropriate max_allowed_packet setting, and then setting it on the server with a session based variable change may be fine, for busy systems this extra overhead is unnecessary especially if you know what the value is anyway and can thus configure it in the connect DSN.
That seems to lead to an optional setting in orchestrator for such connections to allow for 0 (do an extra query to figure out the server's value) or 999 (an explicit value to use as the XXXMaxAllowedPacket setting. By all means make the default 0, but provide a way for the user to be able to overwrite this with a specific setting which reduces the extra latency for each polled connection.

Yves Trudeau July 25, 2023 at 7:09 PM
Hi,
regarding this:
Why do you think it is not really desirable? It is just at most a network round-trip and a very simple query.
Details
Assignee
Yves TrudeauYves TrudeauReporter
Simon MuddSimon MuddLabels
Needs Review
YesNeeds QA
YesComponents
Fix versions
Priority
Medium
Details
Details
Assignee

Reporter

Labels
Needs Review
Needs QA
Components
Fix versions
Priority
Smart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Smart Checklist

I recently saw this problem reported by orchestrator 3.2.6-5
I have orchestrator configured with the following settings:
and it seems that under some circumstances the packet sent by orchestrator is too large. The MySQL server running as a backend to orchestrator runs 8.0.30 and its current configuration shows
max_allowed_packet = 64M.
The go driver's hard-coded setting is:
max_allowed_packet=4MB
, but it's possible to use a value of 0 but this triggers each connection to query the server's setting and is not really desirable.There appears to be no way in orchestrator to provide a specific value to use for
max_allowed_packet
and the logical setting would be via configuration management to make it match the server's setting.So maybe it might be sensible to be able to have an orchestrator parameter
MySQLOrchestratorMaxAllowedPacket
which could be fed to the DSN which orchestrator uses when opening a new connection so the setting is configured properly.The only thing I see as being a bit odd is the go driver is indicating that the server's
max_allowed_packet
setting is too small and the error message suggests fixing the setting on the server. If the 4MB setting that I believe is being used by orchestrator is correct the actual error message is incorrect as the server's size is already larger. Perhaps that should be reported upstream as an error whose message is not entirely correct?Either way orchestrator does not provide a way to adjust this so the configuration setting looks to be needed if this error is seen.