Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

operator tries to add shard repeatedly even when already added when arbiters used

Description

I started 3 shards with PSMDB 4.4 and in replicas using 2 data pods and 1 arbiter. The cluster starts and after some time the shards seem to be added, but the operator is still trying to re-add them.

The sharding without arbiter doesn't seem to reproduce this issue and also the arbiter in just replica set seems to work ok (fixed under K8SPSMDB-401).
For easier reproduction only 1 shard can be used (it just happens I have logs from 3 shards).

# pods NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 6m29s my-cluster-name-cfg-1 2/2 Running 2 6m4s my-cluster-name-cfg-2 2/2 Running 1 5m38s my-cluster-name-mongos-59c8877d77-c7t65 1/1 Running 0 6m20s my-cluster-name-mongos-59c8877d77-c8bbh 1/1 Running 0 6m20s my-cluster-name-mongos-59c8877d77-jjtqp 1/1 Running 0 6m20s my-cluster-name-rs0-0 2/2 Running 0 6m26s my-cluster-name-rs0-1 2/2 Running 0 6m1s my-cluster-name-rs0-arbiter-0 1/1 Running 2 6m26s my-cluster-name-rs1-0 2/2 Running 0 6m24s my-cluster-name-rs1-1 2/2 Running 0 5m58s my-cluster-name-rs1-arbiter-0 1/1 Running 2 6m24s my-cluster-name-rs2-0 2/2 Running 0 6m22s my-cluster-name-rs2-1 2/2 Running 0 5m57s my-cluster-name-rs2-arbiter-0 1/1 Running 2 6m22s percona-server-mongodb-operator-6f866d7857-452z7 1/1 Running 0 7m3s # psmdb object status mongoImage: percona/percona-server-mongodb:4.4.3-5 mongoVersion: 4.4.3-5 mongos: ready: 3 size: 3 status: ready observedGeneration: 1 replsets: cfg: initialized: true ready: 3 size: 3 status: ready rs0: added_as_shard: true initialized: true ready: 3 size: 3 status: ready rs1: added_as_shard: true initialized: true ready: 3 size: 3 status: ready rs2: added_as_shard: true initialized: true ready: 3 size: 3 status: ready state: ready # sharding status from mongos mongos> sh.status() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("60366bce88a1bca4d972c625") } shards: { "_id" : "rs0", "host" : "rs0/my-cluster-name-rs0-0.my-cluster-name-rs0.psmdb-test.svc.cluster.local:27017,my-cluster-name-rs0-1.my-cluster-name-rs0.psmdb-test.svc.cluster.local:27017", "state" : 1 } { "_id" : "rs1", "host" : "rs1/my-cluster-name-rs1-0.my-cluster-name-rs1.psmdb-test.svc.cluster.local:27017,my-cluster-name-rs1-1.my-cluster-name-rs1.psmdb-test.svc.cluster.local:27017", "state" : 1 } { "_id" : "rs2", "host" : "rs2/my-cluster-name-rs2-0.my-cluster-name-rs2.psmdb-test.svc.cluster.local:27017,my-cluster-name-rs2-1.my-cluster-name-rs2.psmdb-test.svc.cluster.local:27017", "state" : 1 } active mongoses: "4.4.3-5" : 3 autosplit: Currently enabled: yes balancer: Currently enabled: yes Currently running: no Failed balancer rounds in last 5 attempts: 0 Migration Results for the last 24 hours: 45 : Success databases: { "_id" : "config", "primary" : "config", "partitioned" : true } config.system.sessions shard key: { "_id" : 1 } unique: false balancing: true chunks: rs0 979 rs1 22 rs2 23 too many chunks to print, use verbose if you want to force print

operator is constantly trying to re-add shards and throwing errors like:

{"level":"info","ts":1614179760.3253045,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"error","ts":1614179760.3272545,"logger":"controller_psmdb","msg":"failed to reconcile cluster","Request.Namespace":"psmdb-test","Request.Name":"my-cluster-name","replset":"rs2","error":"add shard: rsPod is not ready","errorVerbose":"rsPod is not ready\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).handleRsAddToShard\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:297\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:103\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:423\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373\nadd shard\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileCluster\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/mgo.go:105\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:423\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:425\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"info","ts":1614179760.4064531,"logger":"controller_psmdb","msg":"cluster is not ready"}

you can notice that they were added in operator logs, but still they are tried to re-add after that:

$ k logs percona-server-mongodb-operator-6f866d7857-452z7|grep -E "adding|added" {"level":"info","ts":1614179313.9025528,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs0"} {"level":"info","ts":1614179314.7290761,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs0"} {"level":"info","ts":1614179315.6484187,"logger":"controller_psmdb","msg":"added to shard","rs":"rs0"} {"level":"info","ts":1614179316.4980485,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179317.270495,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179321.7546675,"logger":"controller_psmdb","msg":"added to shard","rs":"rs1"} {"level":"info","ts":1614179321.8961885,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179322.7032952,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179326.898448,"logger":"controller_psmdb","msg":"added to shard","rs":"rs2"} {"level":"info","ts":1614179327.7525988,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179328.457552,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179328.5680053,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179332.7441797,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179338.5770102,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179339.4158804,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179345.5875146,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs0"} {"level":"info","ts":1614179349.9265544,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179350.290953,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179350.4067173,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179355.3099682,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179355.4262846,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179360.8590903,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179360.9792159,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179367.1301944,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179367.6256638,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179372.035931,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179372.1778083,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179377.7036479,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179382.979291,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs0"} {"level":"info","ts":1614179383.2256422,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179388.7468698,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179394.2691803,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179399.8178644,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179399.927167,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179405.302914,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179405.8099694,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179416.2467613,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs0"} {"level":"info","ts":1614179416.3746886,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179422.0008743,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179422.3562002,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179422.479269,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179427.4321866,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179432.7990975,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179432.92684,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179434.245089,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs0"} {"level":"info","ts":1614179443.7987573,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179444.3927467,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"} {"level":"info","ts":1614179444.7783813,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs1"} {"level":"info","ts":1614179444.894543,"logger":"controller_psmdb","msg":"adding rs to shard","rs":"rs2"}

Environment

None

Smart Checklist

Activity

Tomislav Plavcic February 25, 2021 at 7:09 AM

Actually even in plain replica set the liveness probe is failing and arbiter pod is restarted, made a comment in https://perconadev.atlassian.net/browse/K8SPSMDB-401#icft=K8SPSMDB-401 - it might be related to this one.

Done

Details

Assignee

Reporter

Labels

Needs QA

Yes

Time tracking

2d 2h 15m logged

Fix versions

Affects versions

Priority

Smart Checklist

Created February 24, 2021 at 3:50 PM
Updated March 5, 2024 at 4:57 PM
Resolved March 8, 2021 at 4:49 PM

Flag notifications