As we know, observability is based on three pillars - metrics, logs and traces. While PMM is good at collecting and visualizing metrics, it also needs to offer the logging and tracing functionality to become a feature-complete observability solution.
When it comes to logging, most of PMM components output logs in the popular “logfmt” format, ensuring that all log events are structured as key/value pairs. However, nginx logs are simple text fields that need to be parsed before a log analysis tool can consume them.
To avoid the cost of parsing we propose formatting nginx logs in “logfmt”.
This change should not be a major breaking change unless the current Nginx log format is explicitly expected.
Note: nginx does not support applying custom formats to error messages, therefore the error messages will not be subject to this improvement.
How to test
Set up PMM
Tail nginx logs (tail -f /srv/logs/nginx.log) to confirm they are formatted in “logfmt”
skip error messages as per the note above
How to document
Mention this as an improvement that offers greater compatibility with major log analysis tools, such as Loki and Victorialogs (we tested both).
Attachments
1
Activity
Alex Demidoff March 16, 2025 at 6:28 PM
Test passes with Victorialogs 1.4.0
The sample output of /srv/logs/nginx.log is formatted in “logfmt”.
As we know, observability is based on three pillars - metrics, logs and traces. While PMM is good at collecting and visualizing metrics, it also needs to offer the logging and tracing functionality to become a feature-complete observability solution.
When it comes to logging, most of PMM components output logs in the popular “logfmt” format, ensuring that all log events are structured as key/value pairs. However, nginx logs are simple text fields that need to be parsed before a log analysis tool can consume them.
To avoid the cost of parsing we propose formatting nginx logs in “logfmt”.
This change should not be a major breaking change unless the current Nginx log format is explicitly expected.
Note: nginx does not support applying custom formats to error messages, therefore the error messages will not be subject to this improvement.