Description

User story:

As a user, I want to use an external Victoria Metrics instance.

 

Acceptance criteria:

  • We can use a separate Victoria Metrics instance outside the container

  • We can setup Victoria Metrics hostname/port and credentials with environment variables.

  • PMM Clients push metrics to Victoria Metrics.

  • Grafana uses external Victoria Metrics

Out of scope:

  • Changing settings from UI (hostname, login/password, and so on)

Suggested implementation:

 -

How to test:

How to test

2 scenarios to test:

  1. standard PMM installation: regression test

  2. PMM with external Victoria Metrics

    1. Enable external Victoria metrics with PMM_VM_URL

    2. Test authentication on external VM via the following ENV variables

      1. VMAGENT_remoteWrite_basicAuth_username={username}

      2.  
        VMAGENT_remoteWrite_basicAuth_password={password} 

    3. regression test

How to document

We added new env variable PMM_VM_URL which should point to external victoria metrics and should have format http(s)://hostname:port/path.

If External victoria metrics requires basic authentication following env vars should be used
VMAGENT_remoteWrite_basicAuth_username={username}
VMAGENT_remoteWrite_basicAuth_password={password}
In case other authentication method used on VictoriaMetrics side users can use any of vmagent env variables by prepending VMAGENT_ prefix.

Internal victoria metrics stops when external victoria metrics is configured. In this case we use VM Agent on PMM Server to pull metrics from agents configured in pulls metrics mode and remote nodes and push collected data to external victoria metrics. VM Agents running on PMM Clients pushes data directly to external victoria metrics. Make sure that they can connect to external victoria metrics.

Attachments

4
  • 18 Sep 2023, 05:30 AM
  • 08 Sep 2023, 06:24 AM
  • 04 Sep 2023, 09:34 AM
  • 29 Aug 2023, 12:53 PM

Smart Checklist

hide

Activity

Show:

Nailya Kutlubaeva September 8, 2023 at 6:24 AM

I started PMM server with the external VM variable, but default VM was started to of some reason.

[root@900fdf1cedc8 opt] # supervisorctl status alertmanager RUNNING pid 29, uptime 0:04:43 clickhouse RUNNING pid 15, uptime 0:04:43 dbaas-controller STOPPED Not started grafana RUNNING pid 125, uptime 0:04:41 nginx RUNNING pid 25, uptime 0:04:43 pmm-agent RUNNING pid 33, uptime 0:04:43 pmm-managed RUNNING pid 32, uptime 0:04:43 pmm-update-perform STOPPED Not started pmm-update-perform-init EXITED Sep 07 02:28 PM postgresql RUNNING pid 14, uptime 0:04:43 prometheus STOPPED Not started qan-api2 RUNNING pid 335, uptime 0:04:36 victoriametrics RUNNING pid 27, uptime 0:04:43 vmalert RUNNING pid 465, uptime 0:04:32 vmproxy RUNNING pid 484, uptime 0:04:31

Nailya Kutlubaeva September 4, 2023 at 9:33 AM

I see the same error with the new FB, attached the logs

Nailya Kutlubaeva August 29, 2023 at 12:51 PM

I installed external VictoriaMetrics on the hot, checked that VictoriaMetrics is available with URL and started PMM server with providing PMM_VM_URL variable, but as I see from logs something is wrong:

level=warning msg="unknown environment variable \"PMM_VM_URL=http://10.30.3.15:8428\"" component=server ........ time="2023-08-29T12:04:35.828+00:00" level=debug msg="External VM is used, VM healthcheck is skipped" component=victoriametrics time="2023-08-29T12:04:35.828+00:00" level=debug msg="VMAlert health: OK" component=vmalert time="2023-08-29T12:04:35.828+00:00" level=debug msg="\nRequest:\n\nResponse:\n\n" request=38c63301-4664-11ee-b3c0-0242ac110002 time="2023-08-29T12:04:35.829+00:00" level=info msg="RPC /server.Server/Readiness done in 3.866143ms." request=38c63301-4664-11ee-b3c0-0242ac110002 time="2023-08-29T12:04:35.829+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl reread\"..." component=supervisord time="2023-08-29T12:04:35.911+00:00" level=debug msg="0 alerts to send, exiting." component=alertmanager time="2023-08-29T12:04:36.007+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl stop victoriametrics\"..." component=supervisord time="2023-08-29T12:04:36.243+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl remove victoriametrics\"..." component=supervisord time="2023-08-29T12:04:36.244+00:00" level=debug msg="Got event: &{Time:2023-08-29 12:04:36.141 +0000 UTC Type:STOPPING Program:victoriametrics}" component=supervisord time="2023-08-29T12:04:36.387+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl add victoriametrics\"..." component=supervisord time="2023-08-29T12:04:36.529+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl reread\"..." component=supervisord time="2023-08-29T12:04:36.695+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl stop victoriametrics\"..." component=supervisord time="2023-08-29T12:04:36.840+00:00" level=warning msg="/usr/local/bin/supervisorctl stop victoriametrics failed: exit status 1" component=supervisord time="2023-08-29T12:04:36.840+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl remove victoriametrics\"..." component=supervisord time="2023-08-29T12:04:36.982+00:00" level=warning msg="/usr/local/bin/supervisorctl remove victoriametrics failed: exit status 1" component=supervisord time="2023-08-29T12:04:36.982+00:00" level=debug msg="Running \"/usr/local/bin/supervisorctl add victoriametrics\"..." component=supervisord time="2023-08-29T12:04:37.128+00:00" level=error msg="Failed to save/reload: /usr/local/bin/supervisorctl add victoriametrics failed: exit status 1." component=supervisord time="2023-08-29T12:04:37.128+00:00" level=warning msg="Failed to update supervisord configuration: /usr/local/bin/supervisorctl add victoriametrics failed: exit status 1." component=setup time="2023-08-29T12:04:37.128+00:00" level=warning msg="Retrying in 2s." component=setup
Done
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Unassigned

Reporter

Nikita Beletskii(Deactivated)

Priority

Needs QA

Yes

Needs Doc

Yes

Planned Version/s

Fix versions

Story Points

5

Smart Checklist Progress

0/4

Smart Checklist

Created December 24, 2021 at 1:27 PM
Updated August 8, 2024 at 5:01 AM
Resolved October 5, 2023 at 3:21 PM

Flag notifications