Description
How to test
2 scenarios to test:
standard PMM installation: regression test
PMM with external Victoria Metrics
Enable external Victoria metrics with PMM_VM_URL
Test authentication on external VM via the following ENV variables
VMAGENT_remoteWrite_basicAuth_username={username}
VMAGENT_remoteWrite_basicAuth_password={password}
regression test
How to document
We added new env variable PMM_VM_URL which should point to external victoria metrics and should have format http(s)://hostname:port/path.
If External victoria metrics requires basic authentication following env vars should be used
VMAGENT_remoteWrite_basicAuth_username={username}
VMAGENT_remoteWrite_basicAuth_password={password}
In case other authentication method used on VictoriaMetrics side users can use any of vmagent env variables by prepending VMAGENT_ prefix.
Internal victoria metrics stops when external victoria metrics is configured. In this case we use VM Agent on PMM Server to pull metrics from agents configured in pulls metrics mode and remote nodes and push collected data to external victoria metrics. VM Agents running on PMM Clients pushes data directly to external victoria metrics. Make sure that they can connect to external victoria metrics.
Attachments
- 18 Sep 2023, 05:30 AM
- 08 Sep 2023, 06:24 AM
- 04 Sep 2023, 09:34 AM
- 29 Aug 2023, 12:53 PM
relates to
Smart Checklist
hideActivity
Nailya Kutlubaeva September 8, 2023 at 6:24 AM
I started PMM server with the external VM variable, but default VM was started to of some reason.
[root@900fdf1cedc8 opt] # supervisorctl status
alertmanager RUNNING pid 29, uptime 0:04:43
clickhouse RUNNING pid 15, uptime 0:04:43
dbaas-controller STOPPED Not started
grafana RUNNING pid 125, uptime 0:04:41
nginx RUNNING pid 25, uptime 0:04:43
pmm-agent RUNNING pid 33, uptime 0:04:43
pmm-managed RUNNING pid 32, uptime 0:04:43
pmm-update-perform STOPPED Not started
pmm-update-perform-init EXITED Sep 07 02:28 PM
postgresql RUNNING pid 14, uptime 0:04:43
prometheus STOPPED Not started
qan-api2 RUNNING pid 335, uptime 0:04:36
victoriametrics RUNNING pid 27, uptime 0:04:43
vmalert RUNNING pid 465, uptime 0:04:32
vmproxy RUNNING pid 484, uptime 0:04:31
Nailya Kutlubaeva September 4, 2023 at 9:33 AM
I see the same error with the new FB, attached the logs
Nailya Kutlubaeva August 29, 2023 at 12:51 PM
I installed external VictoriaMetrics on the hot, checked that VictoriaMetrics is available with URL and started PMM server with providing PMM_VM_URL variable, but as I see from logs something is wrong:
level=warning msg="unknown environment variable \"PMM_VM_URL=http://10.30.3.15:8428\"" component=server
........
time="2023-08-29T12:04:35.828+00:00" level=debug msg="External VM is used, VM healthcheck is skipped" component=victoriametrics
time="2023-08-29T12:04:35.828+00:00" level=debug msg="VMAlert health: OK" component=vmalert
time="2023-08-29T12:04:35.828+00:00" level=debug msg="\nRequest:\n\nResponse:\n\n" request=38c63301-4664-11ee-b3c0-0242ac110002
time="2023-08-29T12:04:35.829+00:00" level=info msg="RPC /server.Server/Readiness done in 3.866143ms." request=38c63301-4664-11ee-b3c0-0242ac110002
time="2023-08-29T12:04:35.829+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl reread\"..." component=supervisord
time="2023-08-29T12:04:35.911+00:00" level=debug msg="0 alerts to send, exiting." component=alertmanager
time="2023-08-29T12:04:36.007+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl stop victoriametrics\"..." component=supervisord
time="2023-08-29T12:04:36.243+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl remove victoriametrics\"..." component=supervisord
time="2023-08-29T12:04:36.244+00:00" level=debug msg="Got event: &{Time:2023-08-29 12:04:36.141 +0000 UTC Type:STOPPING Program:victoriametrics}" component=supervisord
time="2023-08-29T12:04:36.387+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl add victoriametrics\"..." component=supervisord
time="2023-08-29T12:04:36.529+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl reread\"..." component=supervisord
time="2023-08-29T12:04:36.695+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl stop victoriametrics\"..." component=supervisord
time="2023-08-29T12:04:36.840+00:00" level=warning msg="/usr/local/bin/supervisorctl stop victoriametrics failed: exit status 1" component=supervisord
time="2023-08-29T12:04:36.840+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl remove victoriametrics\"..." component=supervisord
time="2023-08-29T12:04:36.982+00:00" level=warning msg="/usr/local/bin/supervisorctl remove victoriametrics failed: exit status 1" component=supervisord
time="2023-08-29T12:04:36.982+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl add victoriametrics\"..." component=supervisord
time="2023-08-29T12:04:37.128+00:00" level=error msg="Failed to save/reload: /usr/local/bin/supervisorctl add victoriametrics failed: exit status 1." component=supervisord
time="2023-08-29T12:04:37.128+00:00" level=warning msg="Failed to update supervisord configuration: /usr/local/bin/supervisorctl add victoriametrics failed: exit status 1." component=setup
time="2023-08-29T12:04:37.128+00:00" level=warning msg="Retrying in 2s." component=setup
User story:
As a user, I want to use an external Victoria Metrics instance.
Acceptance criteria:
We can use a separate Victoria Metrics instance outside the container
We can setup Victoria Metrics hostname/port and credentials with environment variables.
PMM Clients push metrics to Victoria Metrics.
Grafana uses external Victoria Metrics
Out of scope:
Changing settings from UI (hostname, login/password, and so on)
Suggested implementation:
-
How to test: