after upgrade pmm-client (1.6.1-1) can't start mysql:metrics - can't find .my.cnf
Description
How to test
How to document
Smart Checklist
Activity
yesoos February 1, 2018 at 7:45 PM
Seems that pmm-admin is generating (or forcing systemd) to generate this service definition:
/etc/systemd/system/pmm-mysql-metrics-42002.service
Problem is that Environment variable is bogus in this definition, it is adding "?parseTime=true&time_zone='%2b00%3a00'&loc=UTC" to the Environment which causes issues for systemd.
After removing that and doing systemctl daemon-reload mysql:metrics start ok from pmm-admin command.
yesoos February 1, 2018 at 7:16 PM
Errors in log are probably shown from pmm-admin trying to start or something. Both of the processes started manually with DATA_SOURCE_NAME set work ok. So there is clearly a problem somewhere with pmm-admin setting the env variable.
yesoos February 1, 2018 at 7:13 PM
After purging everything (along with pmm user in mysql) and installing version 1.7.0 the problem is still there.
But when starting mysqld_exporter manually (with correct DATA_SOURCE_NAME set) it works. On one server in log I can see it works, on other server there is the same error. But both servers correctly report metrics to PMM server.
None of the processes are correctly identified by pmm-admin.
yesoos January 26, 2018 at 2:37 PM
Michael,
If that is going to help somehow. It happend also on the second node of the cluster (it's a two node cluster). Those were the only things that were upgraded/removed today (below). After that even downgrading to version 1.5.3 or 1.6.0 did not help.
Start-Date: 2018-01-26 07:46:54
Commandline: apt-get dist-upgrade
Install: libisccfg160:amd64 (1:9.11.2+dfsg-5~bpo9+1, automatic), linux-image-4.14.0-0.bpo.3-amd64:amd64 (4.14.13-1~bpo9+1, automatic), libirs160:amd64 (1:9.11.2+dfsg-5~bpo9+1, automatic), libisc166:amd64 (1:9.11.2+dfsg-5~bpo9+1, automatic), liblwres160:a
md64 (1:9.11.2+dfsg-5~bpo9+1, automatic), libdns169:amd64 (1:9.11.2+dfsg-5~bpo9+1, automatic), libisccc160:amd64 (1:9.11.2+dfsg-5~bpo9+1, automatic), libbind9-160:amd64 (1:9.11.2+dfsg-5~bpo9+1, automatic), libjson-c3:amd64 (0.12.1-1.1, automatic)
Upgrade: libdns-export162:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), libisccfg140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), manpages-de:amd64 (2.2-1~bpo9+1, 2.3-1~bpo9+1), iproute:amd64 (1:4.9.0-1+deb9u1
, 1:4.14.1-1~bpo9+1), libsystemd0:amd64 (234-3~bpo9+1, 236-3~bpo9+1), libirs141:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), bind9-host:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.11.2+dfsg-5~bpo9+1), dnsutils:amd64 (1:9.10.3.dfsg.P4-
12.3+deb9u3, 1:9.11.2+dfsg-5~bpo9+1), libisc160:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), udev:amd64 (234-3~bpo9+1, 236-3~bpo9+1), libfastjson4:amd64 (0.99.7-1~bpo9+1, 0.99.8-1~bpo9+1), percona-xtradb-cluster-common-5.7:amd64 (5
.7.19-29.22-3.stretch, 5.7.20-29.24-1.stretch), bind9utils:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.11.2+dfsg-5~bpo9+1), percona-xtradb-cluster-server-5.7:amd64 (5.7.19-29.22-3.stretch, 5.7.20-29.24-1.stretch), libudev1:amd64 (234-3~bpo9+1, 236-3~bpo9+1)
, rsyslog:amd64 (8.30.0-3~bpo9+1, 8.32.0-1~bpo9+1), percona-xtradb-cluster-client-5.7:amd64 (5.7.19-29.22-3.stretch, 5.7.20-29.24-1.stretch), libisc-export160:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), linux-image-amd64:amd64 (4.
13+86~bpo9+1, 4.14+89~bpo9+1), systemd-sysv:amd64 (234-3~bpo9+1, 236-3~bpo9+1), liblwres141:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), systemd:amd64 (234-3~bpo9+1, 236-3~bpo9+1), iproute2:amd64 (4.9.0-1+deb9u1, 4.14.1-1~bpo9+1),
bind9:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.11.2+dfsg-5~bpo9+1), pmm-client:amd64 (1.5.3-1.stretch, 1.6.1-1.stretch), libdns162:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), libxml2:amd64 (2.9.4+dfsg1-2.2+deb9u1, 2.9.4+dfsg1-2.2+
deb9u2), percona-toolkit:amd64 (3.0.5-1.stretch, 3.0.6-1.stretch), rsync:amd64 (3.1.2-1, 3.1.2-1+deb9u1), libisccc140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-12.3+deb9u4), libbind9-140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u3, 1:9.10.3.dfsg.P4-1
2.3+deb9u4), percona-xtradb-cluster-57:amd64 (5.7.19-29.22-3.stretch, 5.7.20-29.24-1.stretch), sensible-utils:amd64 (0.0.9, 0.0.9+deb9u1), libssl1.0.2:amd64 (1.0.2l-2+deb9u1, 1.0.2l-2+deb9u2)
End-Date: 2018-01-26 07:47:17
Start-Date: 2018-01-26 07:47:45
Commandline: apt-get --purge autoremove
Purge: linux-image-4.12.0-0.bpo.1-amd64:amd64 (4.12.6-1~bpo9+1), libisccfg140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4), linux-image-4.9.0-3-amd64:amd64 (4.9.30-2+deb9u5), libirs141:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4), libisc160:amd64 (1:9.10.3.dfsg.P4-12.3+
deb9u4), linux-image-4.11.0-0.bpo.1-amd64:amd64 (4.11.6-1~bpo9+1), liblwres141:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4), libdns162:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4), libisccc140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u4), libbind9-140:amd64 (1:9.10.3.dfsg.P4-12
.3+deb9u4)
End-Date: 2018-01-26 07:47:49
yesoos January 26, 2018 at 2:30 PM
Hi Michael,
I have purged, reinstalled and reconfigured pmm-client. mysql:metric still gives the same error. I'm not using --defaults-file with pmm-admin.
time="2018-01-26T14:26:33Z" level=info msg="Starting mysqld_exporter (version=1.6.1, branch=master, revision=4114da2e398fb94279eeb23acaa2082236b712e6)" source="mysqld_exporter.go:743"
time="2018-01-26T14:26:33Z" level=info msg="Build context (go=go1.9.2, user=, date=)" source="mysqld_exporter.go:744"
time="2018-01-26T14:26:33Z" level=fatal msg="failed reading ini file: open .my.cnf: no such file or directory" source="mysqld_exporter.go:750"
After running latest upgrades of kernel (4.14), percona xtradb cluster and pmm-client mysql:metrics stopped working. It shows error:
time="2018-01-26T08:55:58Z" level=info msg="Starting mysqld_exporter (version=1.6.1, branch=master, revision=4114da2e398fb94279eeb23acaa2082236b712e6)" source="mysqld_exporter.go:743" time="2018-01-26T08:55:58Z" level=info msg="Build context (go=go1.9.2, user=, date=)" source="mysqld_exporter.go:744" time="2018-01-26T08:55:58Z" level=fatal msg="failed reading ini file: open .my.cnf: no such file or directory" source="mysqld_exporter.go:750"The file is in my $HOME
root@db1:~# ls -ltr $HOME/.my.cnf -rw-r--r-- 1 root root 125 Sep 5 17:52 /root/.my.cnf root@db1:~#I've tested 1.6.0, 1.6.1, 1.5.3 - nothing works at this point.