Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

pmm-agent crash because some statements in slow log

Description

# pmm-agent --version
ProjectName: pmm-agent
Version: 2.38.1
PMMVersion: 2.38.1
Timestamp: 2023-07-11 16:50:12 (UTC)
FullCommit: f7772e88783b177278830fbdf23b4aa526f33bf2

##########################################################
# mysqld --version
mysqld  Ver 5.7.42-46 for Linux on x86_64 (Percona Server (GPL), Release 46, Revision e1995a8bb71)

##########################################################
pmm-agent log

pmm-agent[3176]: panic: runtime error: index out of range [55] with length 55 pmm-agent[3176]: goroutine 99 [running]: pmm-agent[3176]: github.com/percona/go-mysql/query.Fingerprint({0xc0001caec0?, 0xc00072d74c?}) pmm-agent[3176]: /tmp/go/pkg/mod/github.com/percona/go-mysql@v0.0.0-20210427141028-73d29c6da78c/query/query.go:779 +0x3c4c pmm-agent[3176]: github.com/percona/pmm/agent/agents/mysql/slowlog.(*SlowLog).processFile(0xc000710330, {0x16bdfa0, 0xc000180640}, {0xc0001c8000, 0x1d}, 0x0?) pmm-agent[3176]: /tmp/go/src/github.com/percona/pmm/agent/agents/mysql/slowlog/slowlog.go:372 +0x122a pmm-agent[3176]: github.com/percona/pmm/agent/agents/mysql/slowlog.(*SlowLog).Run.func3() pmm-agent[3176]: /tmp/go/src/github.com/percona/pmm/agent/agents/mysql/slowlog/slowlog.go:145 +0xdc pmm-agent[3176]: created by github.com/percona/pmm/agent/agents/mysql/slowlog.(*SlowLog).Run pmm-agent[3176]: /tmp/go/src/github.com/percona/pmm/agent/agents/mysql/slowlog/slowlog.go:143 +0x329

##########################################################

  1. mysql slow log content

# Time: 2023-07-18T14:16:08.010973+03:00 # User@Host: SW[SW] @  [172.16.10.215]  Id: 1356746 # Schema: SW  Last_errno: 0  Killed: 0 # Query_time: 0.000143  Lock_time: 0.000047  Rows_sent: 1  Rows_examined: 1  Rows_affected: 0  Bytes_sent: 1363 # Tmp_tables: 0  Tmp_disk_tables: 0  Tmp_table_sizes: 0 # InnoDB_trx_id: 7C776D6FDD # QC_Hit: No  Full_scan: No  Full_join: No  Tmp_table: No  Tmp_table_on_disk: No # Filesort: No  Filesort_on_disk: No  Merge_passes: 0 #   InnoDB_IO_r_ops: 0  InnoDB_IO_r_bytes: 0  InnoDB_IO_r_wait: 0.000000 #   InnoDB_rec_lock_wait: 0.000000  InnoDB_queue_wait: 0.000000 #   InnoDB_pages_distinct: 1 # Log_slow_rate_type: query  Log_slow_rate_limit: 100 SET timestamp=1689678968; SELECT `swd_user`.* FROM `swd_user` WHERE (`id` IN(1)); # Time: 2023-07-18T14:16:14.547903+03:00 # User@Host: SW[SW] @  [172.16.10.215]  Id: 1356743 # Schema: SW  Last_errno: 0  Killed: 0 # Query_time: 0.002315  Lock_time: 0.000000  Rows_sent: 0  Rows_examined: 0  Rows_affected: 0  Bytes_sent: 11 # Tmp_tables: 0  Tmp_disk_tables: 0  Tmp_table_sizes: 0 # InnoDB_trx_id: 7C776D71F0 # QC_Hit: No  Full_scan: No  Full_join: No  Tmp_table: No  Tmp_table_on_disk: No # Filesort: No  Filesort_on_disk: No  Merge_passes: 0 #   InnoDB_IO_r_ops: 0  InnoDB_IO_r_bytes: 0  InnoDB_IO_r_wait: 0.000000 #   InnoDB_rec_lock_wait: 0.000000  InnoDB_queue_wait: 0.000000 #   InnoDB_pages_distinct: 4 # Log_slow_rate_type: query  Log_slow_rate_limit: 100 SET timestamp=1689678974; COMMIT;

 

 

 

How to test

None

How to document

None

Activity

Aaditya Dubey August 25, 2023 at 10:13 AM

Hi ,

Thank you for the report.
Able to repeat the issue, Please find my test case below:

Below PMM Versions used for testing: 1. PMM 2.38.1 2. PMM 2.39.0 MySQL 8.0.27 used as server added to PMM Below SQL statement used to repeat the issue: USE test; CREATE TABLE `t1` (   `id` int DEFAULT NULL,   `name` char(1) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci; INSERT INTO t1(id,name) values(1,'A'),(2,'B'),(3,'C'),(4,'D'); select * from t1 where `id` IN(1); $ journalctl -u pmm-agent.service -f -- Logs begin at Thu 2023-05-18 13:01:59 UTC. -- Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.187+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=main Aug 25 09:34:39 localhost pmm-agent[8825]: INFO[2023-08-25T09:34:39.188+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=main Aug 25 09:34:40 localhost pmm-agent[8825]: WARN[2023-08-25T09:34:40.294+00:00] ts=2023-08-25T09:34:40.294Z caller=node_exporter.go:186 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."  agentID=/agent_id/168b1f16-6fb6-41da-a0c8-e85c09bde38b component=agent-process type=node_exporter Aug 25 09:38:28 localhost pmm-agent[8825]: panic: runtime error: index out of range [36] with length 36 Aug 25 09:38:28 localhost pmm-agent[8825]: goroutine 140 [running]: Aug 25 09:38:28 localhost pmm-agent[8825]: github.com/percona/go-mysql/query.Fingerprint({0xc0006f2870?, 0xc00046d74c?}) Aug 25 09:38:28 localhost pmm-agent[8825]:         /tmp/go/pkg/mod/github.com/percona/go-mysql@v0.0.0-20210427141028-73d29c6da78c/query/query.go:779 +0x3c4c Aug 25 09:38:28 localhost pmm-agent[8825]: github.com/percona/pmm/agent/agents/mysql/slowlog.(*SlowLog).processFile(0xc0000138d8, {0x16bdfa0, 0xc00083c0a0}, {0xc0003ea380, 0x38}, 0x4024000000000000?) Aug 25 09:38:28 localhost pmm-agent[8825]:         /tmp/go/src/github.com/percona/pmm/agent/agents/mysql/slowlog/slowlog.go:372 +0x122a Aug 25 09:38:28 localhost pmm-agent[8825]: github.com/percona/pmm/agent/agents/mysql/slowlog.(*SlowLog).Run.func3() Aug 25 09:38:28 localhost pmm-agent[8825]:         /tmp/go/src/github.com/percona/pmm/agent/agents/mysql/slowlog/slowlog.go:145 +0xdc Aug 25 09:38:28 localhost pmm-agent[8825]: created by github.com/percona/pmm/agent/agents/mysql/slowlog.(*SlowLog).Run Aug 25 09:38:28 localhost pmm-agent[8825]:         /tmp/go/src/github.com/percona/pmm/agent/agents/mysql/slowlog/slowlog.go:143 +0x329 Aug 25 09:38:28 localhost systemd[1]: pmm-agent.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Aug 25 09:38:28 localhost systemd[1]: pmm-agent.service: Failed with result 'exit-code'. Aug 25 09:38:30 localhost systemd[1]: pmm-agent.service: Scheduled restart job, restart counter is at 2. Aug 25 09:38:30 localhost systemd[1]: Stopped pmm-agent. Aug 25 09:38:30 localhost systemd[1]: Started pmm-agent. Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.203+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.204+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.204+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.204+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.204+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.205+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.206+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.206+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.206+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.206+00:00] Runner capacity set to 32.                    component=runner Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.207+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=main Aug 25 09:38:30 localhost pmm-agent[10990]: INFO[2023-08-25T09:38:30.208+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=main Aug 25 09:38:31 localhost pmm-agent[10990]: WARN[2023-08-25T09:38:31.329+00:00] ts=2023-08-25T09:38:31.329Z caller=node_exporter.go:186 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."  agentID=/agent_id/168b1f16-6fb6-41da-a0c8-e85c09bde38b component=agent-process type=node_exporter

Sending the concern to engineering for further review and updates.

Details

Assignee

Reporter

Priority

Components

Labels

Needs QA

Yes

Needs Doc

Yes

Affects versions

Smart Checklist

Created July 18, 2023 at 12:02 PM
Updated November 13, 2024 at 8:32 AM

Flag notifications