Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

pmm-transferer import fails with message "failed to write chunk: non-OK response from victoria metrics: 413"

Description

Issue:
---------
import process fails with the error:

Nginx error log contains the following error message:

Steps to reproduce:

----------------------------
Download the attached .tar.gz files, join them using this command

Then run the import command:

NOTE: replace <port> with your port number

Workaround:
------------------
set "client_max_body_size 30m;" in /etc/nginx/conf.d/pmm.conf under "server" config and reload the nginx.

Suggested fix:
-------------------
Reduce the chunk size to prevent the issue or make the chunk size configurable.

Environment

None

Attachments

12

Smart Checklist

Activity

Sveta Smirnova October 6, 2021 at 2:22 PM

Works fine, thank you!

Oleksandr Levchenkov September 30, 2021 at 10:00 AM

We added max chunk size to meta file. Also, we added a few new commands: "show-meta" and "version".

"show-meta" prints dump meta in a human-readable format. Here is an example: 
./pmm-transferer show-meta --dump-path dump.tar.gz
Build: beb3c45
PMM Version: 2.18.0-57.2106010402.d3c4472.el7
Max Chunk Size: 476.8 kB (465.6 KiB)

"version" just shows binary version. That will help us in testing to know what exactly version is used

Sveta Smirnova September 18, 2021 at 5:48 PM

I tried with 5s chunk and still got the same error. Source PMM is monitoring one server with a generic SysBench workload. Actually, since this is an import error we can live with modifying the target PMM config. But we need maximum chunk size in the meta, so we do not have to perform try-and-error cycles when importing data. Workaround set "client_max_body_size 30m;" did not work for me, for example. And set "client_max_body_size 50m;" worked.

Oleksandr Levchenkov August 29, 2021 at 7:35 PM

We exposed chunk size params (chunk-time-range, chunk-rows) in this PR

Also, we reduced the default value so please, test it again

I think we should add some mechanism to determine chunk timerange/rows in runtime based on allowed chunks size (in terms of memory) if the issue persists for a lot of users

Done

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

Smart Checklist

Created August 17, 2021 at 12:01 PM
Updated March 7, 2024 at 9:38 AM
Resolved November 5, 2021 at 2:25 PM