Using pmm helm chart pre-creating the pmm-secret results in a invalid user/password
Description
Environment
K8 1.28.11 current pmm helm chart
Activity
daniel.almeida April 17, 2025 at 1:25 PM
So far this is what I have had to do to get around this, although the error itself never goes away.
The workaround is to configure a new env var called `GF_SECURITY_ADMIN_PASSWORD`
.This is not a final solution or proper fix, the docker image should be fixed instead. Create a new pmm-secret like this, providing both vars, make sure to use the same password, example below:
apiVersion: v1
kind: Secret
metadata:
name: pmm-secret
labels:
app.kubernetes.io/name: pmm
type: Opaque
data:
# base64 encoded password
PMM_ADMIN_PASSWORD: dHJtWTl343434343435emxKeA== # encode some password, example: `echo -n "somethinghere" | base64`
GF_SECURITY_ADMIN_PASSWORD: dHJtWTl343434343435emxKeA== # encode the same password as above: `echo -n "somethinghere" | base64`
make sure the helm chart does not create a pmm-secret, so set this section to false:
secret:
## @param secret.name Defines the name of the k8s secret that holds passwords and other secrets
##
name: pmm-secret
## @param secret.annotations -- Secret annotations configuration
annotations: {}
## @param secret.create If true then secret will be generated by Helm chart. Otherwise it is expected to be created by user.
##
create: false
You are still going to get the error because this is a docker image problem, but it will at least let the password be properly configure, as the docker image looks for GF_SECURITY_ADMIN_PASSWORD
. It is also important to do this because if you don’t, a random admin password will be generated.
warning msg="Configuration warning: unknown environment variable \"PMM_ADMIN_PASSWORD=[REDACTED]\"."
daniel.almeida April 17, 2025 at 1:20 PM
This is a re-occurring issue, could we please investigate why this is happening and a permanent fix? It seems the docker image no longer recognizes that env var, so we either need to fix it so it is no longer referenced, or we need to add it back.
Customers are starting to see and report this: https://github.com/percona/percona-helm-charts/issues/464
Nurlan Moldomurov November 20, 2024 at 8:47 AM
Please use GF_SECURITY_ADMIN_PASSWORD
instead of PMM_ADMIN_PASSWORD
Aaditya Dubey November 19, 2024 at 2:47 PM
Hi @Lobo Lobo20919
Thank you for the report.
It is verified.
When using the percona pmm helm chart with pmm version 2.43.1 or 2.43.2 and a pre-created pmm-secret the password is not valid for a login to grafana. However if you let the helm chart create the secret it works. See below for test results, pre-created and then letting pmm create the secret. The only item of interest I see in the logs when creating the secret is: time="2024-11-12T14:49:08.305+00:00" level=warning msg="Configuration warning: unknown environment variable \"PMM_ADMIN_PASSWORD=summer1234\"."
That error is not in the logs when the helm chart creates the secret.