Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

pmm-dump error importing data "failed to send HTTP request to victoria metrics: timeout goroutine_id=1"

Description

pmm-dump import failed with the following error

 

./pmm-dump import --pmm-url "http://admin:admin@localhost" --dump-path pmm-dump-xxx.tar.gz --verbose 2022-06-07T23:17:26-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/cmd/pmm-dump/main.go:322 > Got Victoria Metrics URL: http://admin:admin@localhost/prometheus goroutine_id=1 2022-06-07T23:18:01-04:00 INF ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/transferer.go:253 > Importing metrics... goroutine_id=1 2022-06-07T23:18:01-04:00 INF ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/transferer.go:262 > Opening dump file... goroutine_id=1 path=pmm-dump-1653880396.tar.gz 2022-06-07T23:18:01-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/transferer.go:282 > Reading file from dump... goroutine_id=1 2022-06-07T23:18:01-04:00 INF ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/transferer.go:303 > Processing chunk 'vm/1653863401-1653863701.bin'... goroutine_id=1 2022-06-07T23:18:03-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/victoriametrics/source.go:133 > Sending POST chunk request to Victoria Metrics endpoint goroutine_id=1 url=http://admin:admin@localhost/prometheus/api/v1/import/native 2022-06-07T23:18:33-04:00 FTL ../home/runner/work/pmm-dump/pmm-dump/cmd/pmm-dump/main.go:276 > Failed to import: failed to write chunk: failed to send HTTP request to victoria metrics: timeout goroutine_id=1

File size is 520MB 

-rw-r--r--  1 16000  513 520M xx 23:13 pmm-dump-xxx.tar.gz

Already added client_max_body_size 128m; in /etc/nginx/conf.d/pmm.conf under "server" config and reload the nginx

Max chunk size is 80.7 MB

./pmm-dump show-meta --dump-path pmm-dump-xxx.tar.gz --verbose 2022-06-07T23:26:49-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:49-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:50-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:51-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:52-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:53-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:55-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:56-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:39 > Reading files from dump... goroutine_id=1 2022-06-07T23:26:57-04:00 DBG ../home/runner/work/pmm-dump/pmm-dump/pkg/transferer/meta.go:58 > Found meta file goroutine_id=1 Build: 9cebba3 PMM Version: 2.23.0-64.2110061341.d2a5883.el7 Max Chunk Size: 80.7 MB (77.0 MiB)

Environment

None

AFFECTED CS IDs

CS0027250

Attachments

1
  • 11 Jun 2022, 03:17 AM

Smart Checklist

Activity

Show:

Andrii Dema April 24, 2023 at 9:28 PM

I can't reproduce this issue

Phong Dinh June 11, 2022 at 3:17 AM

Attaching more relevant log 

Next attempt show this error

2022-06-08T00:44:00-04:00 FTL ../../runner/work/pmm-dump/pmm-dump/cmd/pmm-dump/main.go:276 > Failed to import: failed to write chunk: non-OK response from victoria metrics: 500: <html>

<head><title>500 Internal Server Error</title></head>
<body>
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>
goroutine_id=1

nginx log

2022/06/08 04:41:51 [error] 89#89: *22 client intended to send too large body: 80702386 bytes, client: 172.17.0.1, server: _, request: "POST /prometheus/api/v1/import/native HTTP/1.1", host: "localhost:8280"
172.17.0.1 - admin [08/Jun/2022:04:41:51 +0000] "POST /prometheus/api/v1/import/native HTTP/1.1" 413 176 "" "fasthttp" ""
 

INFO[2022-06-08T07:12:06.169+00:00] time="2022-06-08T07:12:06Z" level=error msg="Error opening connection to database (postgres://***:PASSWORD_REMOVED@127.0.0.1:5432/postgres?connect_timeout=1&sslmode=disable): \"dial tcp 127.0.0.1:5432: connect: connection refused\": too many connection retries" source="postgres_exporter.go:1661"  agentID=/agent_id/a7bff3df-5d55-44e2-92c9-378a4ae5b497 component=agent-process type=postgres_exporter

Work around: disable auth request in /etc/nginx/conf.d/pmm.conf

Cannot Reproduce

Details

Assignee

Reporter

Components

Affects versions

Priority

Smart Checklist

Created June 8, 2022 at 3:28 AM
Updated March 7, 2024 at 9:36 AM
Resolved April 25, 2023 at 9:14 PM

Flag notifications