pmm-transfer export can not take QAN backup even with explicit use of dump-qan
General
Escalation
General
Escalation
Description
Environment
None
Smart Checklist
Activity
Lalit Choudhary February 22, 2022 at 3:30 PM
working with the following command,
$ ./pmm-transferer version
Build: 301a403
pmm-import-export-tool$ ./pmm-transferer export --pmm-url "http://admin:admin@172.17.0.2:80" --max-load="CPU=90,RAM=90" --dump-qan --ts-selector='{service_name="app1"}' --where=`service_name="app1"`
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=370.tsv source=ch
2022-02-22T20:58:41+05:30 INF Processing 427/427 chunk...
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=368.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=371.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=369.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=373.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=374.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=372.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=377.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=375.tsv source=ch
2022-02-22T20:58:41+05:30 INF Writing chunk to the dump... filename=376.tsv source=ch
2022-02-22T20:58:41+05:30 INF Successfully exported!Sveta Smirnova February 21, 2022 at 11:05 AM
@Lalit Choudhary could you please re-test? I think this is just a matter of how you use the option.
Andrii Dema October 4, 2021 at 1:22 PM
Lalit Choudhary, try to use where service_name value in single quotes, e.g. where=`service_name='ps57'` or where="service_name='ps57'".
We will make usage of -where and ts-selector flags more comfortable in next version.
Lalit Choudhary September 29, 2021 at 3:38 AM
@Oleksandr Levchenkov
--dump-qan broken with service_name filter, without service_name --dump-qan works fine
$ ./pmm-transferer export --pmm-url "http://admin:admin@172.17.0.2:80" --max-load="CPU=90,RAM=90" --dump-qan --ts-selector='{service_name="ps57"}' --where='service_name="ps57"'
2021-09-29T08:43:13+05:30 FTL Failed to create clickhouse chunks: failed to get amount of ClickHouse records: code: 47, message: Missing columns: 'ps57' while processing query: 'SELECT COUNT() FROM metrics WHERE service_name = ps57', required columns: 'service_name' 'ps57', source columns: 'm_plan_time_max' 'm_plan_time_sum' 'm_wal_bytes_sum' 'm_wal_bytes_cnt' 'm_wal_fpi_sum' 'm_wal_fpi_cnt' 'm_wal_records_sum' 'm_wal_records_cnt' 'm_plans_calls_cnt' 'm_cpu_sys_time_sum' 'm_cpu_sys_time_cnt' 'm_cpu_user_time_sum' 'm_cpu_user_time_cnt' 'm_lock_time_cnt' 'cluster' 'm_innodb_queue_wait_sum' 'm_innodb_io_r_wait_p99' 'm_tmp_table_sizes_min' 'm_plans_calls_sum' 'm_innodb_io_r_wait_cnt' 'container_name' 'm_rows_affected_min' 'm_innodb_io_r_bytes_cnt' 'm_innodb_io_r_ops_max' 'm_blk_read_time_cnt' 'm_innodb_io_r_ops_cnt' 'example_type' 'warnings.count' 'm_merge_passes_min' 'm_merge_passes_sum' 'm_query_time_max' 'm_rows_read_p99' 'm_rows_read_min' 'm_rows_read_sum' 'm_rows_affected_p99' 'm_innodb_rec_lock_wait_p99' 'schema' 'm_rows_examined_cnt' 'm_bytes_sent_max' 'm_rows_sent_p99' 'm_rows_sent_min' 'm_merge_passes_cnt' 'm_select_full_range_join_sum' 'm_rows_sent_sum' 'm_innodb_io_r_ops_sum' 'm_rows_sent_cnt' 'm_docs_scanned_p99' 'm_merge_passes_max' 'm_local_blks_hit_sum' 'm_innodb_queue_wait_cnt' 'm_rows_affected_max' 'm_tmp_disk_tables_min' 'm_lock_time_p99' 'node_id' 'replication_set' 'az' 'm_select_range_check_sum' 'm_innodb_rec_lock_wait_sum' 'm_qc_hit_sum' 'm_docs_returned_p99' 'm_rows_examined_min' 'm_query_time_p99' 'example_metrics' 'm_innodb_rec_lock_wait_max' 'm_merge_passes_p99' 'username' 'm_query_time_sum' 'service_id' 'm_shared_blks_written_cnt' 'queryid' 'm_shared_blks_dirtied_cnt' 'node_type' 'labels.value' 'm_rows_affected_sum' 'm_tmp_tables_max' 'service_type' 'm_query_length_sum' 'node_model' 'm_tmp_table_sizes_max' 'client_host' 'm_innodb_io_r_bytes_max' 'm_rows_examined_max' 'num_queries_with_errors' 'period_length' 'agent_id' 'm_innodb_io_r_ops_min' 'm_innodb_io_r_wait_sum' 'm_bytes_sent_p99' 'is_truncated' 'fingerprint' 'm_rows_affected_cnt' 'machine_id' 'num_queries' 'period_start' 'num_queries_with_warnings' 'm_bytes_sent_cnt' 'm_innodb_io_r_bytes_min' 'warnings.code' 'm_response_length_sum' 'region' 'container_id' 'm_innodb_rec_lock_wait_cnt' 'example_format' 'm_lock_time_min' 'environment' 'm_sort_rows_cnt' 'errors.code' 'm_rows_read_max' 'errors.count' 'm_filesort_cnt' 'm_filesort_sum' 'm_rows_examined_p99' 'm_no_good_index_used_sum' 'm_query_time_cnt' 'm_innodb_io_r_bytes_sum' 'm_innodb_io_r_wait_max' 'm_innodb_io_r_ops_p99' 'm_plan_time_cnt' 'm_lock_time_sum' 'm_innodb_io_r_wait_min' 'm_innodb_queue_wait_min' 'm_full_scan_cnt' 'm_query_time_min' 'm_innodb_queue_wait_max' 'm_temp_blks_written_sum' 'm_innodb_queue_wait_p99' 'm_tmp_table_sizes_p99' 'm_innodb_pages_distinct_cnt' 'tables' 'm_rows_read_cnt' 'm_no_index_used_cnt' 'm_innodb_pages_distinct_sum' 'm_innodb_pages_distinct_min' 'm_shared_blks_read_cnt' 'm_innodb_pages_distinct_max' 'm_rows_examined_sum' 'm_innodb_pages_distinct_p99' 'm_query_length_cnt' 'm_shared_blks_hit_sum' 'm_lock_time_max' 'm_query_length_max' 'm_innodb_rec_lock_wait_min' 'm_bytes_sent_sum' 'm_bytes_sent_min' 'm_tmp_tables_cnt' 'm_tmp_tables_sum' 'm_tmp_tables_min' 'm_tmp_tables_p99' 'm_full_join_sum' 'm_plan_time_min' 'm_innodb_io_r_bytes_p99' 'm_tmp_disk_tables_sum' 'example' 'm_select_range_cnt' 'm_tmp_table_on_disk_cnt' 'm_tmp_disk_tables_p99' 'm_tmp_table_sizes_cnt' 'm_tmp_table_sizes_sum' 'm_qc_hit_cnt' 'service_name' 'm_full_scan_sum' 'm_local_blks_written_cnt' 'm_local_blks_read_sum' 'm_tmp_table_cnt' 'm_tmp_table_sum' 'm_shared_blks_hit_cnt' 'm_full_join_cnt' 'm_tmp_table_on_disk_sum' 'm_blk_read_time_sum' 'm_filesort_on_disk_cnt' 'm_select_full_range_join_cnt' 'agent_type' 'm_select_range_sum' 'm_select_range_check_cnt' 'm_sort_range_cnt' 'm_blk_write_time_sum' 'labels.key' 'm_sort_rows_sum' 'm_docs_returned_cnt' 'm_sort_scan_cnt' 'm_query_length_min' 'm_temp_blks_written_cnt' 'm_sort_scan_sum' 'm_no_index_used_sum' 'm_filesort_on_disk_sum' 'm_no_good_index_used_cnt' 'm_docs_returned_sum' 'm_docs_returned_min' 'database' 'm_shared_blks_dirtied_sum' 'node_name' 'm_response_length_cnt' 'm_docs_scanned_sum' 'm_local_blks_dirtied_sum' 'm_response_length_min' 'm_response_length_max' 'm_response_length_p99' 'm_docs_scanned_cnt' 'm_sort_range_sum' 'm_docs_returned_max' 'm_docs_scanned_min' 'm_docs_scanned_max' 'm_local_blks_read_cnt' 'm_shared_blks_read_sum' 'm_shared_blks_written_sum' 'm_rows_sent_max' 'm_local_blks_hit_cnt' 'm_local_blks_dirtied_cnt' 'm_local_blks_written_sum' 'm_query_length_p99' 'm_tmp_disk_tables_cnt' 'm_temp_blks_read_cnt' 'm_tmp_disk_tables_max' 'm_temp_blks_read_sum' 'm_blk_write_time_cnt'Lalit Choudhary September 15, 2021 at 12:43 PMEdited
Thank @Oleksandr Levchenkov
it worked after using --dump-qan
in both export and import command.
QAN data available in pmm after import.
Not a Bug
Details
Details
Details
Smart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Created September 9, 2021 at 2:04 PM
Updated March 7, 2024 at 9:37 AM
Resolved February 22, 2022 at 3:34 PM
Issue : pmm-transfer export not collecting QAN data even with explicit use of dump-qan
Test:
--dump-qan
/pmm-import-export-tool$ ./pmm-transferer export --pmm-url "http://admin:admin@172.17.0.2:80" --max-load="CPU=90,RAM=90" --dump-qan/pmm-import-export-tool$ ./pmm-transferer import --pmm-url "http://admin:admin@172.17.0.2:80" --dump-path pmm-dump-1631194366.tar.gz 2021-09-09T19:06:11+05:30 INF Processing chunk 'vm/1631193166-1631193466.bin'... 2021-09-09T19:06:11+05:30 INF Successfully processed 'vm/1631193166-1631193466.bin' 2021-09-09T19:06:11+05:30 INF Processing chunk 'ch/0.tsv'... 2021-09-09T19:06:11+05:30 WRN Found dump data for ch, but it's not specified - skipped #############Warning 2021-09-09T19:06:11+05:30 INF Processing chunk 'vm/1631193766-1631194066.bin'... 2021-09-09T19:06:11+05:30 INF Successfully processed 'vm/1631193766-1631194066.bin' 2021-09-09T19:06:11+05:30 INF Processing chunk 'vm/1631194366-1631194666.bin'... 2021-09-09T19:06:11+05:30 INF Successfully processed 'vm/1631194366-1631194666.bin' 2021-09-09T19:06:11+05:30 INF Processing chunk 'vm/1631194066-1631194366.bin'... 2021-09-09T19:06:12+05:30 INF Successfully processed 'vm/1631194066-1631194366.bin' 2021-09-09T19:06:12+05:30 INF Successfully imported!Check PMM dashboards and GUI :
Result: Not showing QAN data, Other graphs data can be seen.