RDS API changes the data sent in the request
Description
How to test
How to document
Attachments
- 16 Jan 2025, 02:34 PM
- 16 Jan 2025, 02:34 PM
Activity
denis.subbota January 16, 2025 at 2:34 PM
Hello team,
I would like to add more visibility here by changing node_name, address and instance_id for RDS payload we use.
The payload for an API request:
# Create payload
$ cat <<EOS > /tmp/data.json
{
"cluster": "standalone",
"disable_collectors": [],
"disable_query_examples": true,
"environment": "production",
"log_level": "warn",
"metrics_mode": "PULL",
"node_name": "localhost_rds",
"password": "",
"replication_set": "",
"skip_connection_check": true,
"tls": false,
"tls_ca": "",
"tls_cert": "",
"tls_key": "",
"tls_skip_verify": true,
"username": "percona",
"address": "dummy.rds.instance.name",
"engine": "DISCOVER_RDS_MYSQL",
"instance_id": "API_test_rds_instance",
"port": "3306",
"qan_mysql_perfschema": false,
"qan_postgresql_perfschema": false,
"region": "1",
"custom_labels": {
"installer": "gascan"
}
}
EOS
# Send request to PMM API; netrc is used here to hide credentials
$ curl --netrc -k https://localhost:8443/v1/management/RDS/Add -X POST --data @/tmp/data.json
{
"node": {
"node_id": "/node_id/a1c867d1-c464-49a5-baf7-fb2366c6b64b",
"node_name": "localhost_rds",
"address": "API_test_rds_instance",
"region": "1",
"custom_labels": {
"installer": "gascan"
}
},
"mysql": {
"service_id": "/service_id/937db0a3-cace-4524-be4d-071e0d4f0e78",
"service_name": "API_test_rds_instance",
"node_id": "/node_id/a1c867d1-c464-49a5-baf7-fb2366c6b64b",
"address": "dummy.rds.instance.name",
"port": 3306,
"environment": "production",
"cluster": "standalone",
"custom_labels": {
"installer": "gascan"
}
},
"mysqld_exporter": {
"agent_id": "/agent_id/e90d394b-f314-484d-ad39-1157982019b9",
"pmm_agent_id": "pmm-server",
"service_id": "/service_id/937db0a3-cace-4524-be4d-071e0d4f0e78",
"username": "percona",
"tls_skip_verify": true,
"tablestats_group_table_limit": 1000,
"status": "UNKNOWN"
}
}I would also like to show the differences we get from the GUI of the pmm server to have more hands-on examples of the issue.
From Service we see that it’s showing us the actual endpoint provided via API call, which is ok.
But for the Nodes tab, we see that Address is duplicating the value from instance_id/ Node Name and showing now what we expect to see.
Why it’s a pain for us?
In the Managed Services department, we have a platform for a number of customers, and as our tools rely on the pmm-inventory, it becomes unusable for RDS nodes.
Example.
$ db_connect --list | pt-align
denis_replica2 address=172.31.43.67 port=3306 distro=linux environment=None service=mysql cluster=standalone
localhost_rds address=API_test_rds_instance port=3306 distro=None environment=None service=mysql cluster=standalone
We expect to see endpoints of RDS nodes in pmm-inventory for the Nodes-> Address column.
Additionally, it will give us more consistency for how MySQL and RDS nodes are represented PMM-inventory.
Please let us know if you need more information.
Regards,
Denis Subbota.
Managed Services, Percona.
C W June 7, 2024 at 11:23 AM
Note that "region": "1", is clearly invalid and seems to be permitted by the most minimal of validation (at least 1 rune is required). The same type of issue can be seen producing errors with addresses (see https://perconadev.atlassian.net/browse/PMM-13155 and https://perconadev.atlassian.net/browse/PMM-13156)
Details
Details
Assignee
Reporter
Priority
MediumComponents
Needs QA
Needs Doc
Planned Version/s
Smart Checklist
Open Smart Checklist
Smart Checklist
Impact on the user
The node data for RDS instances is different to that of a normal node and thus it is an inconsistency that the user needs to deal with when using the API to look up addresses. In addition, the user input for the request is changed by PMM
Steps to reproduce
No RDS instance required, the API can be used and disable the connection check
Add an RDS instance via the
v1/management/RDS/Addendpoint# Create payload ❯ cat <<EOS > /tmp/data.json { "cluster": "standalone", "disable_collectors": [], "disable_query_examples": true, "environment": "production", "log_level": "warn", "metrics_mode": "PULL", "node_name": "localhost", "password": "", "replication_set": "", "skip_connection_check": true, "tls": false, "tls_ca": "", "tls_cert": "", "tls_key": "", "tls_skip_verify": true, "username": "percona", "address": "127.0.0.1", "engine": "DISCOVER_RDS_MYSQL", "instance_id": "127", "port": "3306", "qan_mysql_perfschema": false, "qan_postgresql_perfschema": false, "region": "1", "custom_labels": { "installer": "gascan" } } EOS # Send request to PMM API; netrc is used here to hide credentials ❯ curl --netrc -k https://localhost:8443/v1/management/RDS/Add -X POST --data @/tmp/data.json { "node": { "node_id": "/node_id/da079927-80e2-421d-bae2-cdea280ac290", "node_name": "localhost", "address": "127", "region": "1", "custom_labels": { "installer": "gascan" } }, "mysql": { "service_id": "/service_id/38627319-f5d7-452c-9bf2-f49fe59ce21c", "service_name": "127", "node_id": "/node_id/da079927-80e2-421d-bae2-cdea280ac290", "address": "127.0.0.1", "port": 3306, "environment": "production", "cluster": "standalone", "custom_labels": { "installer": "gascan" } }, "mysqld_exporter": { "agent_id": "/agent_id/73adeb1f-7353-4acd-bbc7-640bebdbc4d7", "pmm_agent_id": "pmm-server", "service_id": "/service_id/38627319-f5d7-452c-9bf2-f49fe59ce21c", "username": "percona", "tls_skip_verify": true, "tablestats_group_table_limit": 1000, "status": "UNKNOWN" } }Actual result
The node address does not match the address that was sent and is changed by PMM
"node": { "node_id": "/node_id/da079927-80e2-421d-bae2-cdea280ac290", "node_name": "localhost", "address": "127", <--- this is not correct "region": "1", "custom_labels": { "installer": "gascan" } }Expected result
The node address matches the address that was sent
Workaround
N/A
Details
When creating an RDS instance via the API, the data sent by the user is changed by PMM and then becomes unusable by the user, unlike the address of a normal node.