Running Mongodb Replication Using Docker

For a proper mongodb replication, we are going to start 3 containers for this exercise.

First, start the first container, we will call it mongorep1. We need to set it so that it has hostname, configured to listen to all interfaces, and set a replSet for it called myrepl

docker run -dit --name mongorep1 --hostname mongorep1 mongo:6 --bind_ip_all --replSet myrepl

Once running, we need to get the ip address of mongorep1

docker inspect mongorep1 | grep -w IPAddress

            "IPAddress": "172.17.0.2",

Then, we will start the second container. We need to feed the ip address of the first container to the second container as hosts so that mongo will not have issue setting up the replication

docker run -dit --name mongorep2 --hostname mongorep2 --add-host mongorep1:172.17.0.2 mongo:6 --bind_ip_all --replSet myrepl

Start the third and final container, with command almost similar to the second container.

docker run -dit --name mongorep3 --hostname mongorep3 --add-host mongorep1:172.17.0.2 mongo:6 --bind_ip_all --replSet myrepl

Once we have all the nodes running, access mongosh on the first container, and initiate replica set

docker exec -it mongorep1 mongosh

test> rs.initiate()

Add the other node into the replicaset

myrepl [direct: secondary] test> rs.add("172.17.0.3")

myrepl [direct: primary] test> rs.add("172.17.0.4")

Check the status of the replica set, make sure the first node is the primary node, and the other 2 are the secondary nodes

myrepl [direct: primary] test> rs.status()

...

  members: [

    {

      _id: 0,

      name: 'mongorep1:27017',

      health: 1,

      state: 1,

      stateStr: 'PRIMARY',

...

    {

      _id: 1,

      name: '172.17.0.3:27017',

      health: 1,

      state: 2,

      stateStr: 'SECONDARY',

...

    {

      _id: 2,

      name: '172.17.0.4:27017',

      health: 1,

      state: 2,

      stateStr: 'SECONDARY',

... 

Check if the other node is lagged in terms of replicating data

myrepl [direct: primary] test> db.printSecondaryReplicationInfo()

source: 172.17.0.3:27017

{

  syncedTo: 'Mon Dec 19 2022 15:48:01 GMT+0000 (Coordinated Universal Time)',

  replLag: '0 secs (0 hrs) behind the primary '

}

---

source: 172.17.0.4:27017

{

  syncedTo: 'Mon Dec 19 2022 15:48:01 GMT+0000 (Coordinated Universal Time)',

  replLag: '0 secs (0 hrs) behind the primary '

}

We can test the replication, by adding data into the first node, and check if the data is being replicated into the second and third node. 

docker exec -it mongorep1 mongosh

myrepl [direct: primary] test> use mynewdb

myrepl [direct: primary] mynewdb> db.people.insertOne( { name: "John Rambo", occupation: "Soldier" } ) 

exit

Now access mongosh in the second node and view the data, The data should be similar to the mongorep1

docker exec -it mongorep2 mongosh

myrepl [direct: secondary] test> show dbs

myrepl [direct: secondary] test> use mynewdb

myrepl [direct: secondary] test> db.people.find()

[

  {

    _id: ObjectId("63a08880e1c97fba6959ec15"),

    name: 'John Rambo',

    occupation: 'Soldier'

  }

]

If you encounter this error:

MongoServerError: not primary and secondaryOk=false - consider using db.getMongo().setReadPref() or readPreference in the connection string

Run below command to enable read on the secondary nodes

myrepl [direct: secondary] test> db.getMongo().setReadPref("secondary")

Do the same for the third node, the data should also be the same.

docker exec -it mongorep3 mongosh

myrepl [direct: secondary] test> use mynewdb

myrepl [direct: secondary] test> db.people.find() 

[

  {

    _id: ObjectId("63a08880e1c97fba6959ec15"),

    name: 'John Rambo',

    occupation: 'Soldier'

  }

]

Force a mongodb in a replica set to be a primary

There was one situation, where our production mongo server suddenly becomes secondary, causing any write and read to the server to fail. Searching in the mongo documentation, we found an easy solution. Below are the steps.

First step:
Check that your replica is running fine
mongo> rs.status()

Second step:
Freeze all mongo node in the replica that you do not want to be primary for lets say, 120 seconds. Access the mongo shell, and run below command, do this for all nodes that you do not want to be primary
mongo> rs.freeze(120) 

Third step:
Demote the current primary, so that other node that has not been frozen, will take over as primary. Run this in mongo shell, to demote the node from being a primary, for 120 seconds
mongo> rs.stepdown(120)

That's it, run rs.status() again to make sure that your desired server is now a primary.

Reference: https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/

Setting up mongodb replication

Mongodb needs at least 2 servers, preferably 3, to setup a proper mongodb replication. In this article, we will use below hostname as our mongodb nodes:

192.168.0.10 mongo-1 (primary)
192.168.0.11 mongo-2
192.168.0.12 mongo-3


Make sure mongodb is installed in all servers.

Set mongodb repo:

mongo-1: $ cat > mongodb.repo << EOF >[mongodb]
>name=MongoDB Repository
>baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
>gpgcheck=0
>enabled=1
>EOF
mongo-1: $ sudo mv mongodb.repo /etc/yum.repos.d/

Install mongodb:
mongo-1: $ sudo yum install -y mongodb-org

Set /etc/hosts for each server as below:
mongo-1: $ cat >> hosts << EOF 

192.168.0.10 mongo-1

192.168.0.11 mongo-2

192.168.0.12 mongo-3

EOF 

mongo-1: $ sudo mv /etc/hosts /etc/hosts.original

mongo-1: $ sudo mv hosts /etc/


To ease up this installation, turn off firewall and set selinux to permissive mode, temporarily, in all servers.
mongo-1: $ sudo systemctl stop firewalld mongo-1: $ sudo setenforce 0

Edit /etc/mongod.conf in every server, to be similar as below (assuming we are using myreplica as our replSet)

mongo-1: $ sudo cat /etc/mongod.conf 

logpath=/var/log/mongodb/mongod.log 

logappend=true 

fork=true 

dbpath=/var/lib/mongo 

pidfilepath=/var/run/mongodb/mongod.pid 

replSet=myreplica

Once editing is done, restart mongodb in each server

mongo-1: $ sudo systemctl restart mongod

On the first server initiate mongo replica:

mongo-1: $ sudo mongo

MongoDB shell version: x.x.x

connecting to: test

Server has startup warnings: 

2018-05-28T04:39:22.580+0000 [initandlisten] 

2018-05-28T04:39:22.580+0000 [initandlisten] ** WARNING: Readahead for /var/lib/mongo is set to 4096KB

2018-05-28T04:39:22.580+0000 [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less

2018-05-28T04:39:22.580+0000 [initandlisten] **          http://dochub.mongodb.org/core/readahead

myreplica:PRIMARY> rs.initiate() 

Add the other server, namely mongo-2 and mongo-3 to the replicaset

myreplica:PRIMARY> rs.add("mongo-2")

myreplica:PRIMARY> rs.add("mongo-3")

Run rs.status() to see the status of our replica
myreplica:PRIMARY> rs.status() 

{

        "set" : "myreplica",

        "date" : ISODate("2018-05-28T05:32:10Z"),

        "myState" : 1,

        "members" : [

                {

                        "_id" : 0,

                        "name" : "mongodb01.novalocal.local:27017",

                        "health" : 1,

                        "state" : 1,

                        "stateStr" : "PRIMARY",

                        "uptime" : 11,

                        "optime" : Timestamp(1604978545, 5),

                        "optimeDate" : ISODate("2018-05-28T05:32:10Z"),

                        "electionTime" : Timestamp(1604626688, 1),

                        "electionDate" : ISODate("2018-05-28T05:32:10Z"),

                        "self" : true

                },

        ],

        "ok" : 1

}

In order to rectify the "stateStr: UNKNOWN" and "lastHeartbeatMessage: still initializing", simply add the name of the primary server, as given by mongodb in /etc/hosts of all secondary servers

mongo-2: $ cat /etc/hosts

192.168.0.10 mongo-1 mongodb-1.novalocal
192.168.0.11 mongo-2

192.168.0.12 mongo-3 

mongo-3: $ cat /etc/hosts

192.168.0.10 mongo-1 mongodb-1.novalocal
192.168.0.11 mongo-2

192.168.0.12 mongo-3 

You should be getting "syncingTo : mongodb-1.novalocal:27017", and "stateStr: SECONDARY" when you run rs.status() in primary server

myreplica:PRIMARY> rs.status()

...

{

                        "_id" : 2,

                        "name" : "mongo-3:27017",

                        "health" : 1,

                        "state" : 2,

                        "stateStr" : "SECONDARY",

                        "uptime" : 368,

                        "optime" : Timestamp(1527485519, 1),

                        "optimeDate" : ISODate("2018-05-28T05:31:59Z"),

                        "lastHeartbeat" : ISODate("2018-05-28T05:38:06Z"),

                        "lastHeartbeatRecv" : ISODate("2018-05-28T05:38:06Z"),

                        "pingMs" : 1,

                        "syncingTo" : "mongodb-1.novalocal:27017"

                }

...

Your replica is now complete. To test it out:

Create new database in primary server, and fill up with data

myreplica:PRIMARY> use mynewdb

myreplica:PRIMARY> db.stack.save(

... {

...     "name": "myreplica",

...     "description":  "this is my new mongodb replica",

...     "hosts" : [ "mongo-1", "mongo-2", "mongo-3" ],

... })

WriteResult({ "nInserted" : 1 })

myreplica:PRIMARY> show dbs

admin      (empty)

local      2.077GB

mynewdb    0.078GB

myreplica:PRIMARY> show collections;

stack

system.indexes

myreplica:PRIMARY> db.stack.find()

{ "_id" : ObjectId("5b0b97f9aca2dd0afb9d86a5"), "name" : "myreplica", "description" : "this is my new mongodb replica", "hosts" : [ "mongo-1", "mongo-2", "mongo-3" ] }

Login to secondary servers, sync (by running "rs.slaveOk()" ) and check whether the data gets replicated

myreplica:SECONDARY> use mynewdb

switched to db mynewdb

myreplica:SECONDARY> show collections

2018-05-28T05:51:42.601+0000 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:131

myreplica:SECONDARY> rs.slaveOk()

myreplica:SECONDARY> show collections

stack

system.indexes

myreplica:SECONDARY> db.stack.find()

{ "_id" : ObjectId("5b0b97f9aca2dd0afb9d86a5"), "name" : "myreplica", "description" : "this is my new mongodb replica", "hosts" : [ "mongo-1", "mongo-2", "mongo-3" ] }

Done :)