Monday, May 28, 2018

Setting up mongodb replication

Mongodb needs at least 2 servers, preferably 3, to setup a proper mongodb replication. In this article, we will use below hostname as our mongodb nodes:

192.168.0.10 mongo-1 (primary)
192.168.0.11 mongo-2
192.168.0.12 mongo-3



Make sure mongodb is installed in all servers.

Set mongodb repo:

mongo-1: $ cat <>> mongodb.repo
>[mongodb]
>name=MongoDB Repository
>baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
>gpgcheck=0
>enabled=1
>EOF
mongo-1: $ sudo mv mongodb.repo /etc/yum.repos.d/

Install mongodb:
mongo-1: $ sudo yum install -y mongodb-org


Set /etc/hosts for each server as below:
mongo-1: $ cat <> hosts
>192.168.0.10 mongo-1
>192.168.0.11 mongo-2
>192.168.0.12 mongo-3
mongo-1: $ sudo mv hosts /etc/



To ease up this installation, turn off firewall and set selinux to permissive mode, temporarily, in all servers.
mongo-1: $ sudo systemctl stop firewalld
mongo-1: $ sudo setenforce 0


Edit /etc/mongod.conf in every server, to be similar as below (assuming we are using myreplica as our replSet)

mongo-1: $ sudo cat /etc/mongod.conf
logpath=/var/log/mongodb/mongod.log
logappend=true
fork=true
dbpath=/var/lib/mongo
pidfilepath=/var/run/mongodb/mongod.pid
replSet=myreplica


Once editing is done, restart mongodb in each server
mongo-1: $ sudo systemctl restart mongod


On the first server initiate mongo replica:
mongo-1: $ sudo mongo
MongoDB shell version: x.x.x
connecting to: test
Server has startup warnings: 
2018-05-28T04:39:22.580+0000 [initandlisten] 
2018-05-28T04:39:22.580+0000 [initandlisten] ** WARNING: Readahead for /var/lib/mongo is set to 4096KB
2018-05-28T04:39:22.580+0000 [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
2018-05-28T04:39:22.580+0000 [initandlisten] **          http://dochub.mongodb.org/core/readahead
myreplica:PRIMARY> rs.initiate() 


Add the other server, namely mongo-2 and mongo-3 to the replicaset
myreplica:PRIMARY> rs.add("mongo-2")
myreplica:PRIMARY> rs.add("mongo-3")


Run rs.status() to see the status of our replica
myreplica:PRIMARY> rs.status()
{
        "set" : "myreplica",
        "date" : ISODate("2018-05-28T05:32:10Z"),
        "myState" : 1,
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongodb-1.novalocal:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 77,
                        "optime" : Timestamp(1527485519, 1),
                        "optimeDate" : ISODate("2018-05-28T05:31:59Z"),
                        "electionTime" : Timestamp(1527485474, 1),
                        "electionDate" : ISODate("2018-05-28T05:31:14Z"),
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "mongo-2:27017",
                        "health" : 1,
                        "state" : 6,
                        "stateStr" : "UNKNOWN",
                        "uptime" : 19,
                        "optime" : Timestamp(0, 0),
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2018-05-28T05:32:10Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : 0,
                        "lastHeartbeatMessage" : "still initializing"
                },
                {
                        "_id" : 2,
                        "name" : "mongo-3:27017",
                        "health" : 1,
                        "state" : 6,
                        "stateStr" : "UNKNOWN",
                        "uptime" : 11,
                        "optime" : Timestamp(0, 0),
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2018-05-28T05:32:09Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : 0,
                        "lastHeartbeatMessage" : "still initializing"
                }
        ],
        "ok" : 1
}

In order to rectify the "stateStr: UNKNOWN" and "lastHeartbeatMessage: still initializing", simply add the name of the primary server, as given by mongodb in /etc/hosts of all secondary servers

mongo-2: $ cat /etc/hosts
192.168.0.10 mongo-1 mongodb-1.novalocal
192.168.0.11 mongo-2
192.168.0.12 mongo-3 

mongo-3: $ cat /etc/hosts
192.168.0.10 mongo-1 mongodb-1.novalocal
192.168.0.11 mongo-2
192.168.0.12 mongo-3 


You should be getting "syncingTo : mongodb-1.novalocal:27017", and "stateStr: SECONDARY" when you run rs.status() in primary server

myreplica:PRIMARY> rs.status()
...
{
                        "_id" : 2,
                        "name" : "mongo-3:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 368,
                        "optime" : Timestamp(1527485519, 1),
                        "optimeDate" : ISODate("2018-05-28T05:31:59Z"),
                        "lastHeartbeat" : ISODate("2018-05-28T05:38:06Z"),
                        "lastHeartbeatRecv" : ISODate("2018-05-28T05:38:06Z"),
                        "pingMs" : 1,
                        "syncingTo" : "mongodb-1.novalocal:27017"
                }
...


Your replica is now complete. To test it out:

Create new database in primary server, and fill up with data
myreplica:PRIMARY> use mynewdb
myreplica:PRIMARY> db.stack.save(
... {
...     "name": "myreplica",
...     "description":  "this is my new mongodb replica",
...     "hosts" : [ "mongo-1", "mongo-2", "mongo-3" ],
... })
WriteResult({ "nInserted" : 1 })
myreplica:PRIMARY> show dbs
admin      (empty)
local      2.077GB
mynewdb    0.078GB
myreplica:PRIMARY> show collections;
stack
system.indexes
myreplica:PRIMARY> db.stack.find()
{ "_id" : ObjectId("5b0b97f9aca2dd0afb9d86a5"), "name" : "myreplica", "description" : "this is my new mongodb replica", "hosts" : [ "mongo-1", "mongo-2", "mongo-3" ] }

Login to secondary servers, sync (by running "rs.slaveOk()" ) and check whether the data gets replicated

myreplica:SECONDARY> use mynewdb
switched to db mynewdb
myreplica:SECONDARY> show collections
2018-05-28T05:51:42.601+0000 error: { "$err" : "not master and slaveOk=false", "code" : 13435 } at src/mongo/shell/query.js:131
myreplica:SECONDARY> rs.slaveOk()
myreplica:SECONDARY> show collections
stack
system.indexes
myreplica:SECONDARY> db.stack.find()
{ "_id" : ObjectId("5b0b97f9aca2dd0afb9d86a5"), "name" : "myreplica", "description" : "this is my new mongodb replica", "hosts" : [ "mongo-1", "mongo-2", "mongo-3" ] }


Done :)


No comments: