Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Only single replica set visible in sharded cluster

Description

Hey guys,

 

thanks for the work on the mongodb-exporter. I was able to get everything set up but ran into an issue that I was not able to resolve. The ports are as follows:

  • mongos: 27017

  • mongod configserver: 27019

  • mongod replica set 1: 27018

  • mongod replica set 2: 27020

 

Environment:

We run a sharded 3 node cluster with a single mongos in front of it that accepts and distributes incoming application requests to the mongod nodes. We have two replica sets spread across the 3 cluster nodes. All mongo nodes run 3.2.21.

 

Steps:

Initially I only put the exporter on the mongos instance but soon realized that I was not getting mongod metrics - database and collection metrics were available though. I then rolled out the mongodb-exporter on each mongod node.

 

Since I did not want to have to distinguish between mongos and mongod nodes from a configuration perspective, I changed the mongodb uri to include all relevant ports of our setup:

/usr/local/bin/mongodb_exporter --mongodb.uri=mongodb://localhost:27017,localhost:27018,localhost:27019,localhost:27020 --collect.database --collect.collection --collect.topmetrics --collect.indexusage --web.listen-address=0.0.0.0:9216

The exporter runs and can be scraped successfully:

time="2019-04-04T22:38:24Z" level=info msg="Starting mongodb_exporter (version=0.7.0, branch=v0.7.0, revision=e9fda63e88dfa3ce8e851b061ca995557b5071d0)" source="mongodb_exporter.go:103" time="2019-04-04T22:38:24Z" level=info msg="Build context (go=go1.11.5, user=travis@build.travis-ci.org, date=20190311-17:55:22)" source="mongodb_exporter.go:104" time="2019-04-04T22:38:24Z" level=info msg="Starting HTTP server for http://0.0.0.0:9216/metrics ..." source="server.go:121"

If I run a Prometheus query to check the replica sets I only see one set (mgo_rs1 is missing):

Query: mongodb_mongod_replset_my_name{cluster="shared-development"} Result: mongodb_mongod_replset_my_name{cluster="shared-development",instance="inst-int-mgo1a",job="mongodb-exporter",name="172.16.44.136:27018",set="mgo_rs0"} mongodb_mongod_replset_my_name{cluster="shared-development",instance="inst-int-mgo2a",job="mongodb-exporter",name="172.16.44.139:27018",set="mgo_rs0"} mongodb_mongod_replset_my_name{cluster="shared-development",instance="inst-int-mgo3a",job="mongodb-exporter",name="172.16.44.140:27018",set="mgo_rs0"}

My assumption is that running a single exporter is fine for this use case since the cli help lists this schema:

[mongodb://][user:pass@]host1[:port1][,host2[:port2],...][/database][?options]

Please let me know what I can do to help you help me slightly smiling face

How to test

None

How to document

None

Attachments

2

Activity

Vasil Kosturski 
November 5, 2019 at 8:49 AM

Hey Lalit,

Following your suggestions above I'm configuring the exporter against a replica set. So for example I have the mongo uri set to mongodb://<firstserver:port>,<secondserver:port>. However the counters I get back from the exporter seem to be only for the first server. I manually inspected the response from the exporter by hitting the exporter metrics endpoint. To be more concrete, the "mongodb_op_counters_total" counters returned are just for "firstserver". I double-checked that by manually executing the top command in mongo against the server.

So the question is: If we want to get the metrics for each member of a replicaset, do we need to run a separate exporter for repl set member? This seems to be the case if I'm not missing something.

Please advice.

 

Lalit Choudhary 
April 23, 2019 at 12:58 PM

Marking this as documentation improvement, since having an example for MongoDB replica set sharded cluster case in the following link will be helpful for users.

https://github.com/percona/mongodb_exporter

Lalit Choudhary 
April 23, 2019 at 12:47 PM

Hi

Thank you for the report.

To make this work for each replicaSet run separate mongodb_exporter as follows, detail test case attached.

 

For example My Test setup:

PROCESS PORT STATUS PID () mongos 27017 running 18759 () config server 27027 running 18517 () shard01 secondary 27019 running 18247 secondary 27020 running 18280 mongod 27018 running 18213 () shard02 secondary 27023 running 18382 mongod 27021 running 18314 mongod 27022 running 18348 () shard03 mongod 27024 running 18415 mongod 27025 running 18449 mongod 27026 running 18483

 

For each replicaset with its member, run separate mongodb_exporter

 

# ./mongodb_exporter --mongodb.uri=mongodb://localhost:27018,localhost:27019,localhost:27020 --collect.database --collect.collection --collect.topmetrics --collect.indexusage --web.listen-address=:9216 # ./mongodb_exporter --mongodb.uri=mongodb://localhost:27021,localhost:27022,localhost:27023 --collect.database --collect.collection --collect.topmetrics --collect.indexusage --web.listen-address=:9217 # ./mongodb_exporter --mongodb.uri=mongodb://localhost:27024,localhost:27025,localhost:27026 --collect.database --collect.collection --collect.topmetrics --collect.indexusage --web.listen-address=:9218

 

Details

Assignee

Reporter

Priority

Components

Labels

Needs QA

Needs Doc

Environment

Amazon Linux instance

mongodb_exporter (version=0.7.0, branch=v0.7.0, revision=e9fda63e88dfa3ce8e851b061ca995557b5071d0)" source="mongodb_exporter.go:103"
time="2019-04-04T22:38:24Z" level=info msg="Build context (go=go1.11.5, user=travis@build.travis-ci.org, date=20190311-17:55:22)"

Created April 4, 2019 at 11:46 PM
Updated March 27, 2024 at 2:59 PM

Flag notifications