Issues

Select view

Select search mode

 

Cascading disk fill-up due to WAL file accumulation

Description

In the everest github repo, a user reported a WAL file accumulation on the repo host that eventually filled up the PVC and then led to the primary and replica pods to also fill up their respective PVCs and crashed the cluster.

https://github.com/percona/everest/issues/781

 

 

We require some investigation to understand the reason for this WAL file accumulation and assess possible solutions.

Environment

None

Attachments

1
  • 21 Nov 2024, 02:37 PM

Details

Assignee

Reporter

Needs QA

Yes

Affects versions

Priority

Smart Checklist

Created November 21, 2024 at 2:37 PM
Updated November 21, 2024 at 3:04 PM

Activity

Charly BatistaNovember 21, 2024 at 3:04 PM

Based on the symptoms, it seems the problem was related to “replication slot”. My theory is there was a replication slot used by one replica that may have crashed and was later replaced by the operator, or never replaced, and that replication slot prevented the primary from removing old WAL files. Another possibility is the WAL retention configuration wasn’t ideal.