Tuesday, May 7, 2019

Install openshift origin 3.11 cluster on a single virtualbox VM running CentOS 7

The minimum requirements for openshift origin (OKD) 3.11 is at least 16GB memory, but since my machine does not have that much capacity, I just use 8GB memory, and exclude all hardware checks in my inventory file

For openshift installation to run smoothly, you need a proper, separate DNS server. Refer to my previous post, on how to setup a very easy DNS server. The DNS can be installed in another VM with probably 512MB memory.

Prepare a VM, with:
- 8GB memory
- 50GB hardisk
- 1 vcpu
- bridged network

Install centos 7 on the VM

Since we are going to use ansible, passwordless ssh is necessary, even though it is just only one machine
# ssh-keygen
# ssh-copy-id localhost

Update the operating system, and install base packages
# yum update -y; yum install wget git net-tools bind-utils yum-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct -y; reboot

Install epel repository, and disable the repo by default
# yum install -y epel-release; sed -i 's/enabled=1/enabled=0/' /etc/yum/repos.d/epel.repo

Install ansible and pyOpenSSL
# yum install -y --enablerepo=epel ansible pyOpenSSL

Install docker
# yum install -y docker-1.13.1

Clone the openshift-origin repository in github. This repository will provide required playbooks and configuration files
# cd
# git clone https://github.com/openshift/openshift-ansible
# cd openshift-ansible
# git checkout release-3.11

Generate a hashed password for your first user
# openssl passwd -apr1 typeyourpasswordhere

Prepare your inventory file. You can refer here for the meaning of each options in below inventory file. Make sure that every hostname used in this file is DNS resolvable 
# cat > ~/openshift-ansible/inventory.ini <<EOF

'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
openshift_master_htpasswd_users={'admin': '$apr1$qpJB3Cls$PN7/HlUNqBXikBl.jnrHF.'}


osmaster.local.my openshift_schedulable=true 


osmaster.local.my openshift_schedulable=true openshift_node_group_name="node-config-all-in-one"

Run the prerequisites.yml playbook. This playbook will install required for openshift installation
# cd ~/openshift-ansible
# ansible-playbook -i inventory.ini playbook/prerequisites.yml

Run the deployment playbook to deploy your openshift cluster
# ansible-playbook -i inventory.ini playbook/deploy_cluster.yml

Once installation is complete, verify your installation by checking on the nodes
# oc get nodes

and logging in to openshift webconsole which in this case is https://console.local.my:8443, providing the username and password as per in your inventory.ini file

Monday, May 6, 2019

Postgresql replication on CentOS

I will use 2 machines to do this. For the sake of practicing, even 2 containers will do. The IP addresses are:

- master
- slave

Install postgresql repo on both machines
# yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm

Install postgresql on both machines, in this example, I am using postgres 9.6
# yum install -y postgresql96-server

Initialize both postgres
# su - postgres
$ /usr/pgsql-9.6/bin/initdb -D /var/lib/pgsql/9.6/data/

On master, put in below config
# su - postgres
$ cat >> /var/lib/pgsql/9.6/data/postgresql.conf <<EOF
wal_level = hot_standby
max_wal_senders = 1 # number of slave servers
wal_keep_segments = 100
synchronous_standby_names = 'pgslave'

On master, create a user for replication, called replica
$ psql -c "create user replica replication;"

Allow slave to access master as replica
$ cat >> /var/lib/pgsql/9.6/data/pg_hba.conf <<EOF
host    replication     replica         trust

Restart postgres on master server
# systemctl restart postgresql-9.6

On slave server, stop postgresql
# systemctl stop postgresql-9.6

Clear slave server postgresql data directory
# rm -rf /var/lib/pgsql/9.6/data/*

Copy data from master
# su - postgres
$ pg_basebackup -D /var/lib/pgsql/9.6/data -h -U replica --verbose

Create recovery.conf in master server
$ cat > /var/lib/pgsql/9.6/data/recovery.conf <<EOF
primary_conninfo='host= port=5432 user=replica application_name=pgslave'

Turn hot_standby on
$ sed -i 's/#hot_standby\ =\ off/ hot_standby\ =\ on/'/var/lib/pgsql/9.6/data/postgresql.conf

Start postgres on slave
# systemct restart postgresql-9.6

To check replication status, run below in master server
# su - postgres
$ psql -c "select client_addr, state, sent_location, write_location,flush_location, replay_location from pg_stat_replication;"

Test your replication by adding data/database into master server, and check whether the data/database is replicated to slave.

If a master is down, you need to promote the current slave to master, to allow it to be writable
# su - postgres
$ /usr/pgsql-9.6/bin/pg_ctl promote -D /var/lib/pgsql/9.6/data/

Friday, May 3, 2019

Setup easy DNS server using dnsmasq on CentOS 7

Install dnsmasq

# yum install dnsmasq -y

Put in upstream dns server in /etc/resolv.conf. In this case, I want to use opendns as my upstream dns server.
# cat >> /etc/resolv.conf <<EOF

For dns records, just use /etc/hosts
# cat >> /etc/hosts <<EOF mydns.local myportal.local myworkspace.local

With just these 2 settings, you are good to go. Start dnsmasq, and your dns server should be able to resolve those 3 domains.
# systemctl start dnsmasq

Allow on firewall
# firewall-cmd --add-service dns
# firewall-cmd --add-service dns --permanent

Test with dig
# dig +short @localhost myportal.local

Test from other machine
# dig +short @ myworkspace.local

It can even forward to upstream DNS
# dig +short @ www.google.com

Tuesday, April 30, 2019

How to install phpmyadmin for mysql 8 in Centos 7

To install mysql 8 on centos 7, please follow here.

Install epel-release
# yum install -y epel-release

Install phpmyadmin, httpd and php
# yum install -y phpmyadmin httpd php 

Change the ip allowed to access phpmyadmin by changing to your network's ip (in my case, I want to allow all machine with 192.168.0 ip to be able to access the phpmyadmin page)
# sed -i 's/'  /etc/httpd/phpMyAdmin.conf

Start httpd
# systemctl start httpd

Allow http on firewall
# firewall-cmd --add-service http
# firewall-cmd --add-service http --permanent

Change your mysql user's password to use mysql_native_password
# mysql -u root -p
mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'MyNewPassword!123';

You should be able to access phpmyadmin, by pointing your browser to the server's ip address, followed by /phpmyadmin

And the you can login into phpmyadmin by supplying mysql username and password

How to install mysql 8 in Centos 7

Install mysql yum repo

# yum install -y https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm

Install mysql-community-server
# yum install -y mysql-community-server

Start mysql server
# systemctl start mysqld

Get the default mysql root password from mysqld.log
# grep 'temporary password' /var/log/mysqld.log

Login to the mysql console, and change the root password
# mysql -u root -p
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewP@ssword!123';
mysql> exit

If you are using centos internal firewall, allow access to mysql port
# firewall-cmd --add-service mysql
# firewall-cmd --add-service mysql --permanent

Wednesday, April 10, 2019

Create a mini lab for practicing ansible using docker

To practice ansible, you need to have at least 2 machines. I suggest using containers rather than VM, since containers can be quickly spawned, and are light on the resources.

First, make sure you have docker Community Edition installed. If not, follow the install guide here.

Check your docker version
# docker version

Start the docker engine
# sudo systemctl start docker 

In this exercise, we will use ubuntu as our base operating system. So, run a container using the ubuntu image from docker hub. The options are -i for interactive, -t to allocate pseudo TTY and -d to run the container in the background
# docker run -it -d --name="ansible-master" ubuntu

The ubuntu image does not come with ssh, which is needed for ansible, so we need to install that, together with vim text editor 
# docker exec -it apt update; apt install vim openssh-server -y

Change the root password
# docker exec -it ansible-master passwd 

Permit root login for ssh
# docker exec -it ansible-master /bin/bash
ansible-master: # cat >> /etc/ssh/sshd_config <<EOF
PermitRootLogin yes

Start ssh
ansible-master: # service ssh start; exit

Create an image based on ansible-master. This image will be used later to create ansible-client1 container
# docker commit -m "ubuntu with vim and openssh-server" ansible-master myubuntu:2019041001

Run a container called ansible-client1 from the image created above
# docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
myubuntu            2019041001          17f43a3ef384        10 minutes ago      265MB
# docker run -d -it --name="ansible-client1" myubuntu:2019041001

Start ssh service on ansible-client1
# docker exec -it ansible-client1 service ssh start

Try to ssh into both machines. Get the ip address of the containers using "docker inspect" command
# docker inspect ansible-client1 | grep -w IPAddress
            "IPAddress": "",
                    "IPAddress": "",
# docker inspect ansible-master | grep -w IPAddress
            "IPAddress": "",
                    "IPAddress": "",
# ssh root@
# ssh root@

Install ansible on ansible-master
# docker exec -it ansible-master apt install ansible -y

Check ansible version
# docker exec -it ansible --version

Create ssh-key without password
# docker exec -it ssh-keygen

Transfer the key to ansible-client1
# docker exec -it ssh-copy-id

Edit /etc/ansible/hosts to include all nodes
# docker exec -it ansible-master /bin/bash
ansible-master: # cat >> /etc/ansible/hosts <<EOF
ansible-client1 ansible_host=


Test ansible using ping module
# docker exec -it ansible-master -m ping all
localhost | SUCCESS => {
    "changed": false, 
    "ping": "pong"
ansible-client1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"

Congratulations, now you have your own mini ansible lab, using docker. You can add more clients as you wish later.

Tuesday, April 9, 2019

Troubleshooting sshd fail to start

I encountered this issue, and journalctl just does not give enough information on what has happened to make ssh failed to start. So after searching around, I found that I can check the /etc/ssh/sshd_conf for any syntax error, just by running it with extended test (-T) flag. What this flag do is, check the validity of /etc/ssh/sshd_config, throw out error if any, and exit. So to check the issue in the configuration file, just run:

# /usr/sbin/sshd -T

Another way is, you can also start sshd manually with debug flag (-d), and it will throw out any error that stopping it from starting:

# /usr/sbin/sshd -d

Monday, April 8, 2019

/boot Keeps Filling Up On Kernel Update

This is an issue I encountered in one of my friend's ubuntu 16.04 box. He tried to do kernel update, but the /boot keeps filling up with old initramfs image files, making the update process failed. Then I found a post here, that says that if /var/lib/initramfs-tools is not being cleaned up from old kernel files, /boot will keep on being filled up with old initramfs images. So to clean it up:

# uname -r 
# cd /var/lib/initramfs-tools
# rm `ls | grep -v 4.15.0-46`

Once cleaned up, update your current initramfs, using:

# update-initramfs -u -k all

where -u is to generate initramfs for current kernel, and '-k all' to generate initramfs for kernel version newer than current kernel.

Once that done, you can safely reboot your machine. It will be rebooted using the latest kernel.

Friday, March 22, 2019

Rename and Remount LVM Logical Volume

A usual scenario where rename and remount of LVM logical volume are needed, is when you install a box with a CentOS, and use the default LVM based partitioning scheme. This scheme will take 50G for your / partition, and the rest will be allocated to your /home, which is not practical. In this example, we will be remounting logical volume originally mounted to /home, to /var/lib/elasticsearch, renaming it along the way.

The original partition scheme 
# df -Th
Filesystem                   Type      Size  Used Avail Use% Mounted on
/dev/mapper/cl-root          xfs        50G  3.0G   48G   6% /
/dev/sda1                    xfs      1014M  179M  836M  18% /boot
/dev/mapper/cl-home xfs       965G   42G  924G   5% /home

Stop the service that might be using the partition that we are going to change
# systemctl stop elasticsearch

Rename the original directory to other name
# mv /var/lib/elasticsearch /var/lib/elasticsearch-old

Create a new directory to replace the one we already renamed above
# mkdir /var/lib/elasticsearch

Unmount /home
# umount /home

Check the name of volume group and logical volume
# lvs

Rename the logical volume to the one we desired. This is totally optional. You can use -t flag to test out the renaming process before proceeding
# lvrename -t cl home elasticsearch
# lvrename cl home elasticsearch

Change /etc/fstab to reflect on the new logical volume name and mount point. Test it out without the  -i option, before permanently make the change using the -i option
# sed 's/cl-home/cl-elasticsearch/g;s/\/home/\/var\/lib\/elasticsearch/g' /etc/fstab
# sed -i 's/cl-home/cl-elasticsearch/g;s/\/home/\/var\/lib\/elasticsearch/g' /etc/fstab

Mount it
# mount -a

Check to see if your new directory and logical volume mounted properly
# df -Th
Filesystem                   Type      Size  Used Avail Use% Mounted on
/dev/mapper/cl-root          xfs        50G  3.0G   48G   6% /
/dev/sda1                    xfs      1014M  179M  836M  18% /boot
/dev/mapper/cl-elasticsearch xfs       965G   42G  924G   5% /var/lib/elasticsearch

Move the content of /var/lib/elasticsearch to /home
# mv /var/lib/elasticsearch/* /home

Move the content of /var/lib/elasticsearch-old to /var/lib/elasticsearch
# mv /var/lib/elasticsearch-old/* /var/lib/elasticsearch

Remove /var/lib/elasticsearch-old
# rmdir /var/lib/elasticsearch-old

Set proper permission for /var/lib/elasticsearch
# chmod 750 -R /var/lib/elasticsearch
# chown -R elasticsearch.elasticsearch /var/lib/elasticsearch

Start your service
# systemctl start elasticsearch


Thursday, March 7, 2019

Run an Application with GUI with Docker

There are a few ways to achieve this, like vnc and ssh X forwarding. But in this post, I will show how to run firefox by sharing the .Xauthority file with the docker instance.

First, create a Dockerfile in a directory called docker that use an ubuntu image from dockerhub, and install the firefox application

$ mkdir docker
$ cd docker
$ cat > Dockerfile
FROM ubuntu
RUN apt update && apt install -y firefox
CMD ["/usr/bin/firefox"]

Press ctrl-d to save the file

Next, build an image using the above Dockerfile

$ sudo docker build --tag=firefox-app docker/

Then, run the image, with additional options of --network and --env, and sharing our host .Xauthority file with the container that we are going to run 

$ sudo docker run --network=host --env="DISPLAY" --volume="$HOME/.Xauthority:/root/.Xauthority:rw" --name=firefox1 firefox-app

You should be able to get a working firefox, running from a docker container, separated from the current firefox that you have in your current machine

Credit: https://medium.com/@SaravSun/running-gui-applications-inside-docker-containers-83d65c0db110

Tuesday, February 26, 2019

Install vncserver on Ubuntu 18.04 desktop

In this guide, we will be using tigervnc server

Install the vncserver
$ sudo apt update; sudo apt install tigervnc-standalone-server tigervnc-xorg-extension -y

Setup a password for vncserver
$ vncpasswd

Once you have provided the password, make sure a passwd file is created
$ ls ~/.vnc/

Run below command to put some settings in ~/.vnc/xstartup, so that gnome will be started when vnc is used
$ cat > ~/.vnc/xstartup <<EOF
# Start Gnome 3 Desktop 
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
vncconfig -iconic &
dbus-launch --exit-with-session gnome-session &

Press ctrl-d to save the above file

Start a vncserver instance
$ vncserver

List the display to get the port number to connect to, in this example the port number would be 5901, since the display number is 1
$ vncserver -list 

TigerVNC server sessions:

:1 5375

In the client machine, connect with using vncviewer like below, replacing the x.x.x.x with your server's ip number
$ vncviewer x.x.x.x:5901

To terminate the vnc session, use below command to kill the first instance of the vncserver
$ vncserver -kill :1

Friday, February 15, 2019

Install xrdp on Ubuntu 18.04 desktop

Install xrdp
$ sudo apt-get -y install xrdp

Next, one may adjust the configuration file:
$ sudo nano /etc/xrdp/xrdp.ini

Set encryption level to high, and save xrdp.ini:

Allow just RDP through the local firewall:
$ sudo ufw allow 3389/tcp

Create a polkit configuration file:
$ sudo nano /etc/polkit-1/localauthority.conf.d/02-allow-colord.conf 

and put below settings into the file:

polkit.addRule(function(action, subject) {
if ((action.id == “org.freedesktop.color-manager.create-device” || action.id == “org.freedesktop.color-manager.create-profile” || action.id == “org.freedesktop.color-manager.delete-device” || action.id == “org.freedesktop.color-manager.delete-profile” || action.id == “org.freedesktop.color-manager.modify-device” || action.id == “org.freedesktop.color-manager.modify-profile”) && subject.isInGroup(“{group}”))
return polkit.Result.YES;

Restart xrdp
$ sudo systemctl restart xrdp

Logout all user from your desktop

Connect using remote desktop client. For linux, you can use remmina.

P/S: If you are still unable to login using remote desktop client, check whether you have xorgxrdp package installed. If not, install it and restart xrdp. Then try to connect again

$ sudo apt install xorgxrdp -y
$ sudo systemctl restart xrdp

Tuesday, January 15, 2019

Monitor the progress of mysql import

Mysql command does not have any means of monitoring the progress of sql file import. One method that I found useful in monitoring mysql import, is by using pipeviewer(pv) command. To install pipeviewer in a centos box:

Install epel repository
# yum install epel-release -y

Install pv
# yum install pv -y

To use the pv command to monitor mysql import progress

# pv mydatabase.sql | mysql -u myusername -p mydatabase

You will get progress bar showing how much data has been imported from the sql file to mysql.