Skip to:
We observed that `pmm_metrics_advisor_checks_downloaded` materialized view query failed due high memory usage. It seems that it makes sense to check if we can rewrite in a optimal way.https://github.com/percona/telemetryd/blob/main/migrations/012_create_pmm_metrics_advisor_checks_downloaded_view.up.sql
2023.12.01 09:52:40.051790 [ 372 ] {699f66bd-ba25-4774-8294-3158aa08aeba} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 5.50 GiB (attempt to allocate chunk of 0 bytes), maximum: 5.27 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: While executing AggregatingTransform: while pushing to view telemetryd.pmm_metrics_advisor_checks_executed (0b8e794f-a2f2-451f-8988-de0438d3a440). (MEMORY_LIMIT_EXCEEDED) (version 23.3.13.7.altinitystable (altinity build)) (from 10.0.0.5:54772) (in query: INSERT INTO pmm_metrics ( event_id, event_time, pmm_server_telemetry_id, portal_org_id, portal_org_name, percona_customer_tier, pmm_server_metrics) VALUES), Stack trace (when copying this message, always include the lines below):0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe2c3055 in /usr/bin/clickhouse1. ? @ 0xe2cf471 in /usr/bin/clickhouse2. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2cef24 in /usr/bin/clickhouse3. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2ce80a in /usr/bin/clickhouse4. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2ce80a in /usr/bin/clickhouse5. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2ce80a in /usr/bin/clickhouse6. DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x1306beb3 in /usr/bin/clickhouse7. DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x130e17b7 in /usr/bin/clickhouse8. DB::AggregatingTransform::consume(DB::Chunk) @ 0x14d9de48 in /usr/bin/clickhouse9. DB::AggregatingTransform::work() @ 0x14d99beb in /usr/bin/clickhouse10. DB::ExecutionThreadContext::executeTask() @ 0x14bd5c6a in /usr/bin/clickhouse11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x14bcacdb in /usr/bin/clickhouse12. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x14bca0c8 in /usr/bin/clickhouse13. DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x14bdacb7 in /usr/bin/clickhouse14. DB::ExecutingInnerQueryFromViewTransform::onGenerate() @ 0x14e5886b in /usr/bin/clickhouse15. ? @ 0x14dbc3d8 in /usr/bin/clickhouse16. ? @ 0x14dbbf99 in /usr/bin/clickhouse17. DB::ExceptionKeepingTransform::work() @ 0x14dbbabb in /usr/bin/clickhouse18. DB::ExecutionThreadContext::executeTask() @ 0x14bd5c6a in /usr/bin/clickhouse19. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x14bcacdb in /usr/bin/clickhouse20. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x14bca0c8 in /usr/bin/clickhouse21. DB::TCPHandler::processInsertQuery() @ 0x14b79c38 in /usr/bin/clickhouse22. DB::TCPHandler::runImpl() @ 0x14b71058 in /usr/bin/clickhouse23. DB::TCPHandler::run() @ 0x14b86579 in /usr/bin/clickhouse24. Poco::Net::TCPServerConnection::start() @ 0x17acf5b4 in /usr/bin/clickhouse25. Poco::Net::TCPServerDispatcher::run() @ 0x17ad07db in /usr/bin/clickhouse26. Poco::PooledThread::run() @ 0x17c58067 in /usr/bin/clickhouse27. Poco::ThreadImpl::runnableEntry(void*) @ 0x17c55a9d in /usr/bin/clickhouse28. ? @ 0x7fbe51996609 in ?29. clone @ 0x7fbe518bb133 in ?
We observed that `pmm_metrics_advisor_checks_downloaded` materialized view query failed due high memory usage. It seems that it makes sense to check if we can rewrite in a optimal way.
https://github.com/percona/telemetryd/blob/main/migrations/012_create_pmm_metrics_advisor_checks_downloaded_view.up.sql
2023.12.01 09:52:40.051790 [ 372 ] {699f66bd-ba25-4774-8294-3158aa08aeba} <Error> executeQuery: Code: 241. DB::Exception: Memory limit (total) exceeded: would use 5.50 GiB (attempt to allocate chunk of 0 bytes), maximum: 5.27 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker.: While executing AggregatingTransform: while pushing to view telemetryd.pmm_metrics_advisor_checks_executed (0b8e794f-a2f2-451f-8988-de0438d3a440). (MEMORY_LIMIT_EXCEEDED) (version 23.3.13.7.altinitystable (altinity build)) (from 10.0.0.5:54772) (in query: INSERT INTO pmm_metrics ( event_id, event_time, pmm_server_telemetry_id, portal_org_id, portal_org_name, percona_customer_tier, pmm_server_metrics) VALUES), Stack trace (when copying this message, always include the lines below):0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0xe2c3055 in /usr/bin/clickhouse1. ? @ 0xe2cf471 in /usr/bin/clickhouse2. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2cef24 in /usr/bin/clickhouse3. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2ce80a in /usr/bin/clickhouse4. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2ce80a in /usr/bin/clickhouse5. MemoryTracker::allocImpl(long, bool, MemoryTracker*) @ 0xe2ce80a in /usr/bin/clickhouse6. DB::Aggregator::checkLimits(unsigned long, bool&) const @ 0x1306beb3 in /usr/bin/clickhouse7. DB::Aggregator::executeOnBlock(std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>&, std::vector<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>, std::allocator<std::vector<DB::IColumn const*, std::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x130e17b7 in /usr/bin/clickhouse8. DB::AggregatingTransform::consume(DB::Chunk) @ 0x14d9de48 in /usr/bin/clickhouse9. DB::AggregatingTransform::work() @ 0x14d99beb in /usr/bin/clickhouse10. DB::ExecutionThreadContext::executeTask() @ 0x14bd5c6a in /usr/bin/clickhouse11. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x14bcacdb in /usr/bin/clickhouse12. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x14bca0c8 in /usr/bin/clickhouse13. DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x14bdacb7 in /usr/bin/clickhouse14. DB::ExecutingInnerQueryFromViewTransform::onGenerate() @ 0x14e5886b in /usr/bin/clickhouse15. ? @ 0x14dbc3d8 in /usr/bin/clickhouse16. ? @ 0x14dbbf99 in /usr/bin/clickhouse17. DB::ExceptionKeepingTransform::work() @ 0x14dbbabb in /usr/bin/clickhouse18. DB::ExecutionThreadContext::executeTask() @ 0x14bd5c6a in /usr/bin/clickhouse19. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x14bcacdb in /usr/bin/clickhouse20. DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x14bca0c8 in /usr/bin/clickhouse21. DB::TCPHandler::processInsertQuery() @ 0x14b79c38 in /usr/bin/clickhouse22. DB::TCPHandler::runImpl() @ 0x14b71058 in /usr/bin/clickhouse23. DB::TCPHandler::run() @ 0x14b86579 in /usr/bin/clickhouse24. Poco::Net::TCPServerConnection::start() @ 0x17acf5b4 in /usr/bin/clickhouse25. Poco::Net::TCPServerDispatcher::run() @ 0x17ad07db in /usr/bin/clickhouse26. Poco::PooledThread::run() @ 0x17c58067 in /usr/bin/clickhouse27. Poco::ThreadImpl::runnableEntry(void*) @ 0x17c55a9d in /usr/bin/clickhouse28. ? @ 0x7fbe51996609 in ?29. clone @ 0x7fbe518bb133 in ?