Saturday, December 29, 2018

Force a mongodb in a replica set to be a primary

There was one situation, where our production mongo server suddenly becomes secondary, causing any write and read to the server to fail. Searching in the mongo documentation, we found an easy solution. Below are the steps.

First step:
Check that your replica is running fine
mongo> rs.status()

Second step:
Freeze all mongo node in the replica that you do not want to be primary for lets say, 120 seconds. Access the mongo shell, and run below command, do this for all nodes that you do not want to be primary
mongo> rs.freeze(120) 

Third step:
Demote the current primary, so that other node that has not been frozen, will take over as primary. Run this in mongo shell, to demote the node from being a primary, for 120 seconds
mongo> rs.stepdown(120)

That's it, run rs.status() again to make sure that your desired server is now a primary.

Reference: https://docs.mongodb.com/manual/tutorial/force-member-to-be-primary/

Tuesday, December 18, 2018

Use centos repository in RHEL

Get the repository url in your country, by visiting http://mirror-status.centos.org/. In my case, since I am in Malaysia, I chose http://centos.ipserverone.com/centos/

Create a repo file in /etc/yum.repos.d
# cat > /etc/yum.repos.d/centos.repo
[centos]
name=CentOS-7
baseurl=http://centos.ipserverone.com/centos/7/os/x86_64/
enabled=1
gpgcheck=1
gpgkey=http://centos.ipserverone.com/centos/7/os/x86_64/RPM-GPG-KEY-CentOS-7


Press ctrl-d to exit and save

Check whether your new centos repo is listed and ready to be used
# yum repolist

Done

Backup RHEV/RHV/ovirt disk image to external drive

Login to RHV/RHEV/ovirt administrator interface



Search for your VM, lets say I want to search for windows VM



Right click on the VM name, and choose 'Create Snapshot'

Give a proper name, and tick "Save Memory" if you want to save current state of memory as well. Please be warned, that by choosing "Save Memory", the VM will be paused while the snapshooting is in progress. 

If you check the Snapshots tab, the status of the snapshot will be Locked, and the VM will be paused

Once completed, the status will be changed to Ok

Now, open a terminal inside the rhev machine, and search for the disk image snapshot by running below command:
# virsh -r dumpxml windows | grep "source file"

Your newly created snapshot is on the last line of the above command. Copy the file location, and put it into your external drive
# cp /rhev/data-center/00000002-0002-0002-0002-00000000035d/747b7d84-68d5-4436-98f5-baeec68381e1/images/8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/e42c8e85-9566-4953-b881-70c7ce97de0b /mnt/my_external_drive/windows-snapshot-20181219.raw


If you want a different format, use qemu-img command to change it. For example, you require this image to be vmware compatible
# qemu-img -f raw -O vmdk /mnt/my_external_drive/windows-snapshot-20181219.raw /mnt/my_external_drive/windows-snapshot-20181219.vmdk


Tuesday, December 11, 2018

Send postgresql log to syslog

Enable syslog to listen to udp port 514 (the port number can be changed to suit your need)

# vi /etc/rsyslog.conf
$ModLoad imudp
$UDPServerRun 514


Restart rsyslog
# systemctl restart rsyslog


Check if rsyslog is now listening to 514 udp port
# ss -tulpn | grep 514
udp    UNCONN     0      0                      *:514                   *:*      users:(("rsyslogd",15281,3))
udp    UNCONN     0      0                     :::514                  :::*      users:(("rsyslogd",15281,4))


Set postgres to log to syslog
# vi /var/lib/pgsql/9.5/data/postgresql.conf
log_destination = 'syslog'


Restart postgres
# systemctl restart postgresql-9.5


Check if your postgres log is now being logged by syslog
# tail -f /var/log/messages 
...
Dec 11 08:39:21 mypostgres postgres[21518]: [3-1] < 2018-12-11 08:39:21.932 UTC >LOG:  autovacuum launcher started

Tuesday, December 4, 2018

Install nvidia driver for Tesla card on ubuntu 16.04

Check what driver your card is currently using, in this case, the card is using the opensource nvidia drivers called nouveau
$ lspci -v | grep -A8 NVIDIA
3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)
        Subsystem: NVIDIA Corporation Device 11d8
        Flags: bus master, fast devsel, latency 0, IRQ 40
        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities:
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau



Download the driver for your card from nvidia download website


Choose the correct card model, and operating system, and click SEARCH


Click DOWNLOAD


Click AGREE & DOWNLOAD to download


If you want to download using wget, right click on the AGREE & DOWNLOAD button, and choose "Copy Link Address". Run wget against the link
$ wget http://us.download.nvidia.com/tesla/410.79/nvidia-diag-driver-local-repo-ubuntu1604-410.79_1.0-1_amd64.deb


Install the driver
$ sudo apt install ./nvidia-diag-driver-local-repo-ubuntu1604-410.79_1.0-1_amd64.deb


Make sure you have now nvidiafb in kernel modules 
$ lspci -v | grep -A15 NVIDIA
3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)
        Subsystem: NVIDIA Corporation Device 11d8
        Flags: bus master, fast devsel, latency 0, IRQ 40
        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [128] Power Budgeting
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
        Capabilities: [900] #19
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau


Update
$ sudo apt update


Install cuda-drivers
$ sudo apt install cuda-drivers


Reboot your machine
$ sudo reboot


Check whether your card is using nvidia driver, and not nouveau
$ lspci -v | grep -A8 NVIDIA
3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)
        Subsystem: NVIDIA Corporation Device 11d8
        Flags: bus master, fast devsel, latency 0, IRQ 88
        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities:
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_410_drm, nvidia_410

Monday, December 3, 2018

Using tsocks with apt to bypass proxy

Recently, I encountered an error while trying to update ubuntu server 16.04. The error is as below:

E: Failed to fetch http://my.archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)

It seems there is a proxy somewhere in the network, that I do not know. Asking around, even the owner of the server does not even know any proxy server inside their network. So, what I did was, create a socks proxy on my own, to a server outside of the network, and tunnel the apt connection using an application called tsocks.

First, you need to install tsocks. This is a bit tricky since we cannot use the apt command to download and install it from the internet. So I downloaded the deb file, using wget.
$ wget -c http://archive.ubuntu.com/ubuntu/pool/universe/t/tsocks/tsocks_1.8beta5-9.3_amd64.deb

Once downloaded, install using apt on the local file.
$ sudo apt install ./tsocks*.deb

Once installed, create a socks proxy to a server outside of the network, on localhost port 8888 (this is just my favorite number for socks proxy, you can use any number above 1000). You need to be able to ssh to the outside server for this to be possible.
$ ssh foo@server.outside -D 8888

Change the tsocks config file, to suite your new socks proxy
$ sudo cat /etc/tsocks.conf
# You just need to change these 3 lines
server = 127.0.0.1
server_type = 5
server_port = 8888

$ sudo -i 
# tsocks apt update
# tsocks apt upgrade



P/S: You can also set an environment variable called http_proxy, if you do not want to install tsocks, but this setting will be only active on your current bash session. This can be accomplished by:
# export http_proxy='socks5://localhost:8888'
# apt update && apt upgrade