Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Issues

Select view

Select search mode

 
13 of 13

I/O error: Broken pipe, while writing to socket

Description

Observed the below errors in click-house server log file. I don't know how to reproduce the issue.


My env has mix of MariaDB and PostgreSQL servers.

MariaDB Version - 10.5.21

Postgresql Version - 16

PMM version - 2.41.2

PMM Agent Version - 2.41.2

2024.11.05 05:09:28.501028 [ 1591613 ] {2337e603-9011-4dea-a46f-d9521572f6be} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:50000). (NETWORK_ERROR) (version 23.8.2.7 (official build)) (from 127.0.0.1:50000) (in query: SELECT labels.key, labels.value FROM metrics WHERE (period_start >= 1728191248) AND (period_start <= 1730783248) ORDER BY labels.value ASC ), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001311785a in /usr/bin/clickhouse

  5. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  6. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  7. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  9. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  10. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  11. ? @ 0x00007f4caad9ac12 in ?

  12. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.529428 [ 3709299 ] {17ea0057-1672-4623-953c-b3e2aa0d00c5} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:48238). (NETWORK_ERROR) (version 23.8.2.7 (official build)) (from 127.0.0.1:48238) (in query: SELECT labels.key, labels.value FROM metrics WHERE (period_start >= 1728191188) AND (period_start <= 1730783188) ORDER BY labels.value ASC ), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000713cffd in /usr/bin/clickhouse

  5. DB::SerializationString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const @ 0x0000000011407b03 in /usr/bin/clickhouse

  6. DB::SerializationArray::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, unsigned long, unsigned long, DB::ISerialization::SerializeBinaryBulkSettings&, std::shared_ptrDB::ISerialization::SerializeBinaryBulkState&) const @ 0x00000000113d2afc in /usr/bin/clickhouse

  7. DB::NativeWriter::write(DB::Block const&) @ 0x000000001314351b in /usr/bin/clickhouse

  8. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000013117840 in /usr/bin/clickhouse

  9. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  10. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  11. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  12. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  13. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  14. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  15. ? @ 0x00007f4caad9ac12 in ?

  16. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.529428 [ 3709299 ] {17ea0057-1672-4623-953c-b3e2aa0d00c5} <Error> executeQuery: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:48238). (NETWORK_ERROR) (version 23.8.2.7 (official build)) (from 127.0.0.1:48238) (in query: SELECT labels.key, labels.value FROM metrics WHERE (period_start >= 1728191188) AND (period_start <= 1730783188) ORDER BY labels.value ASC ), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000713cffd in /usr/bin/clickhouse

  5. DB::SerializationString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const @ 0x0000000011407b03 in /usr/bin/clickhouse

  6. DB::SerializationArray::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, unsigned long, unsigned long, DB::ISerialization::SerializeBinaryBulkSettings&, std::shared_ptrDB::ISerialization::SerializeBinaryBulkState&) const @ 0x00000000113d2afc in /usr/bin/clickhouse

  7. DB::NativeWriter::write(DB::Block const&) @ 0x000000001314351b in /usr/bin/clickhouse

  8. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000013117840 in /usr/bin/clickhouse

  9. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  10. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  11. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  12. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  13. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  14. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  15. ? @ 0x00007f4caad9ac12 in ?

  16. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.630840 [ 3709299 ] {17ea0057-1672-4623-953c-b3e2aa0d00c5} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:48238). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000713cffd in /usr/bin/clickhouse

  5. DB::SerializationString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const @ 0x0000000011407b03 in /usr/bin/clickhouse

  6. DB::SerializationArray::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, unsigned long, unsigned long, DB::ISerialization::SerializeBinaryBulkSettings&, std::shared_ptrDB::ISerialization::SerializeBinaryBulkState&) const @ 0x00000000113d2afc in /usr/bin/clickhouse

  7. DB::NativeWriter::write(DB::Block const&) @ 0x000000001314351b in /usr/bin/clickhouse

  8. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000013117840 in /usr/bin/clickhouse

  9. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  10. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  11. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  12. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  13. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  14. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  15. ? @ 0x00007f4caad9ac12 in ?

  16. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.630840 [ 3709299 ] {17ea0057-1672-4623-953c-b3e2aa0d00c5} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:48238). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::WriteBuffer::write(char const*, unsigned long) @ 0x000000000713cffd in /usr/bin/clickhouse

  5. DB::SerializationString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const @ 0x0000000011407b03 in /usr/bin/clickhouse

  6. DB::SerializationArray::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, unsigned long, unsigned long, DB::ISerialization::SerializeBinaryBulkSettings&, std::shared_ptrDB::ISerialization::SerializeBinaryBulkState&) const @ 0x00000000113d2afc in /usr/bin/clickhouse

  7. DB::NativeWriter::write(DB::Block const&) @ 0x000000001314351b in /usr/bin/clickhouse

  8. DB::TCPHandler::sendData(DB::Block const&) @ 0x0000000013117840 in /usr/bin/clickhouse

  9. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  10. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  11. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  12. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  13. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  14. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  15. ? @ 0x00007f4caad9ac12 in ?

  16. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.631052 [ 3709299 ] {17ea0057-1672-4623-953c-b3e2aa0d00c5} <Warning> TCPHandler: Client has gone away.
2024.11.05 05:09:28.631052 [ 3709299 ] {17ea0057-1672-4623-953c-b3e2aa0d00c5} <Warning> TCPHandler: Client has gone away.
2024.11.05 05:09:28.633451 [ 1591613 ] {2337e603-9011-4dea-a46f-d9521572f6be} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:50000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001311785a in /usr/bin/clickhouse

  5. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  6. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  7. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  9. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  10. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  11. ? @ 0x00007f4caad9ac12 in ?

  12. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.633451 [ 1591613 ] {2337e603-9011-4dea-a46f-d9521572f6be} <Error> TCPHandler: Code: 210. DB::NetException: Connection reset by peer, while writing to socket (127.0.0.1:9000 -> 127.0.0.1:50000). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

  1. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c604bf7 in /usr/bin/clickhouse

  2. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000c87c410 in /usr/bin/clickhouse

  3. DB::WriteBufferFromPocoSocket::nextImpl() @ 0x000000000c87bf71 in /usr/bin/clickhouse

  4. DB::TCPHandler::sendData(DB::Block const&) @ 0x000000001311785a in /usr/bin/clickhouse

  5. DB::TCPHandler::runImpl() @ 0x000000001310e2d2 in /usr/bin/clickhouse

  6. DB::TCPHandler::run() @ 0x000000001311e839 in /usr/bin/clickhouse

  7. Poco::Net::TCPServerConnection::start() @ 0x0000000015b104d4 in /usr/bin/clickhouse

  8. Poco::Net::TCPServerDispatcher::run() @ 0x0000000015b116d1 in /usr/bin/clickhouse

  9. Poco::PooledThread::run() @ 0x0000000015c47f07 in /usr/bin/clickhouse

  10. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015c461dc in /usr/bin/clickhouse

  11. ? @ 0x00007f4caad9ac12 in ?

  12. ? @ 0x00007f4caae1ef54 in ?

2024.11.05 05:09:28.633646 [ 1591613 ] {2337e603-9011-4dea-a46f-d9521572f6be} <Warning> TCPHandler: Client has gone away.
2024.11.05 05:09:28.633646 [ 1591613 ] {2337e603-9011-4dea-a46f-d9521572f6be} <Warning> TCPHandler: Client has gone away.

How to test

None

How to document

None

Details

Assignee

Reporter

Priority

Labels

Needs QA

Yes

Needs Doc

Yes

Affects versions

Smart Checklist

Created November 6, 2024 at 10:55 AM
Updated November 18, 2024 at 2:17 PM

Activity

NareshNovember 18, 2024 at 2:12 PM

Even last time also, i have upgraded the PMM but no luck.

But i will try and let you know.

Aaditya DubeyNovember 18, 2024 at 2:09 PM

Hi

We have PMM 2.43.2, would it be possible to upgrade and see if your issue is fixed?

NareshNovember 18, 2024 at 1:59 PM

  1. How frequently it is happening? – > every minute.

  2. What is the PMM server/client version? → 2.41.1

Aaditya DubeyNovember 18, 2024 at 12:54 PM

Hi

  1. How frequently it is happening?

  2. What is the PMM server/client version?

NareshNovember 18, 2024 at 12:46 PM

I only raised that ticket, I don't have any reproducible steps at this moment.