Define default retention for system log tables or disable them

General

Escalation

General

Escalation

Description

The internal log tables consumes a lot of space in clickhouse which causes disk issues at times. If trace_log, metric_log, query_log etc are not required for the PMM functioning we should either disable them or add a default TTL in the definition, say 30d.

<ttl>event_date + INTERVAL 30 DAY DELETE</ttl>

Ex:

<trace_log>
    <database>system</database>
    <table>trace_log</table>

    <partition_by>toYYYYMM(event_date)</partition_by>
    <ttl>event_date + INTERVAL 30 DAY DELETE</ttl>
    <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    <max_size_rows>1048576</max_size_rows>
    <reserved_size_rows>8192</reserved_size_rows>
    <buffer_size_rows_flush_threshold>524288</buffer_size_rows_flush_threshold>
    <!-- Indication whether logs should be dumped to the disk in case of a crash -->
    <flush_on_crash>false</flush_on_crash>
</trace_log>

Proposed solution (draft)

add configurable retention parameter to internal PMM clickhouse with 30d default

How to test

Test with short retention parameter (modify + restart CH) and check syslog table

How to document

Release notes + check if any mention in online help about CH params

FB test

None

AFFECTED CS IDs

PRB0041968, CS0052076, CS0054444

Activity

talha.rizwan

April 17, 2025 at 5:00 AM

Nailya Kutlubaeva

April 15, 2025 at 2:56 PM

talha.rizwan

April 9, 2025 at 7:12 AM

Nurlan Moldomurov

April 9, 2025 at 9:34 AM

Aaditya Dubey

January 2, 2025 at 11:09 AM

Resize work item view side panel

Done

Details

Security Level Help

None

Assignee

talha.rizwan

Reporter

Kedar Vaijanapurkar

Priority

Medium

Components

Packaging

QAN

Labels

AprilESMarchrefined

Needs QA

Yes

Needs Doc

Planned Version/s

3.2.0

Fix versions

3.2.0

Story Points

Sprint

Add sprint

Affects versions

2.41.2

Smart Checklist Progress

1/1 - Done

Start date

Apr 08, 2025

Due date

May 29, 2025

Created January 2, 2025 at 9:49 AM

Updated June 2, 2025 at 8:46 AM

Resolved May 5, 2025 at 8:59 AM