[Research] Tune VMagent resource usage

Description

The problem: vmagent uses quite a bit of memory.
200MB of memory used (VM is much higher) for something which just scrapes the data and pushed it over is kind of a lot.
 

Expectations: it would be natural for it to use similar memory as node_exporter/mysqld_exporter use as functionality is similar - they get data in one format and expose in another basically..
 Note: The expectations can be changed if VMgent start  collecting data locally

 

 

Details:

 
The following options can be used for reducing the memory usage for vmagent:
 

  • Passing `-streamParse` command-line flag to it. In this case it will read metrics from targets in a streaming manner. This allows saving RAM when a target exposes a big number of metrics. See https://docs.victoriametrics.com/vmagent.html#stream-parsing-mode  Already done

  • Passing `-memory.allowedBytes=100MB` command-line flag to it. This will limit in-memory buffers vmagent can use. By default vmagent can use up to 60% of available memory on the host server (unless it is limited with cgroups).

  • Passing `GOMAXPROCS=1` environment variable to vmagent. This may reduce memory usage, since vmagent pre-allocates some in-memory buffers per available CPU core. By default it can use all the available CPU cores on the host server (unless it is limited with cgroups). GOMAXPROCS=1 limits available CPU cores to 1.
     
    If these options won't help, then it would be great collecting memory profile from vmagent and investigating the source of high memory usage. See https://docs.victoriametrics.com/vmagent.html#profiling .

 

Also analyze new features of VM and open subtasks or new feature to implement them.
 

How to test

None

How to document

None

Attachments

1
  • 21 May 2021, 10:26 AM

Smart Checklist

Activity

Nurlan Moldomurov January 4, 2022 at 1:20 PM

I made a small research regarding memory usage.
Installed 3 systems: # dev-latest

  1. dev-latest with allowed 150MB(30MB per agent)

  2. dev-latest with Victoria metrics 1.69

In the first case we have memory usage between 135MiB and 165MiB

In the second case memory usage is between 120MiB and 145MiB
In the third one it is around 100 MiB

So we agreed to just upgrade VM for now.

-promscrape.minResponseSizeForStreamParse has default value 1MB, I think it's good enough for now. No need to tweak it.

Denys Kondratenko December 14, 2021 at 5:44 PM

we need to review and re-test new vm 1.69 version and see.

 

There is also `-promscrape.minResponseSizeForStreamParse` feature added in 1.68, maybe we should use it.

Taras Kozub November 18, 2021 at 12:34 PM

,  please discuss this item

Done

Details

Assignee

Reporter

Priority

Components

Story Points

Smart Checklist

Created May 21, 2021 at 10:34 AM
Updated March 6, 2024 at 2:42 AM
Resolved January 4, 2022 at 11:48 AM

Flag notifications