Wednesday, April 10, 2019

Create a mini lab for practicing ansible using docker

To practice ansible, you need to have at least 2 machines. I suggest using containers rather than VM, since containers can be quickly spawned, and are light on the resources.

First, make sure you have docker Community Edition installed. If not, follow the install guide here.

Check your docker version
# docker version

Start the docker engine
# sudo systemctl start docker 

In this exercise, we will use ubuntu as our base operating system. So, run a container using the ubuntu image from docker hub. The options are -i for interactive, -t to allocate pseudo TTY and -d to run the container in the background
# docker run -it -d --name="ansible-master" ubuntu

The ubuntu image does not come with ssh, which is needed for ansible, so we need to install that, together with vim text editor 
# docker exec -it apt update; apt install vim openssh-server -y

Change the root password
# docker exec -it ansible-master passwd 

Permit root login for ssh
# docker exec -it ansible-master /bin/bash
ansible-master: # cat >> /etc/ssh/sshd_config <<EOF
PermitRootLogin yes

Start ssh
ansible-master: # service ssh start; exit

Create an image based on ansible-master. This image will be used later to create ansible-client1 container
# docker commit -m "ubuntu with vim and openssh-server" ansible-master myubuntu:2019041001

Run a container called ansible-client1 from the image created above
# docker image ls
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
myubuntu            2019041001          17f43a3ef384        10 minutes ago      265MB
# docker run -d -it --name="ansible-client1" myubuntu:2019041001

Start ssh service on ansible-client1
# docker exec -it ansible-client1 service ssh start

Try to ssh into both machines. Get the ip address of the containers using "docker inspect" command
# docker inspect ansible-client1 | grep -w IPAddress
            "IPAddress": "",
                    "IPAddress": "",
# docker inspect ansible-master | grep -w IPAddress
            "IPAddress": "",
                    "IPAddress": "",
# ssh root@
# ssh root@

Install ansible on ansible-master
# docker exec -it ansible-master apt install ansible -y

Check ansible version
# docker exec -it ansible --version

Create ssh-key without password
# docker exec -it ssh-keygen

Transfer the key to ansible-client1
# docker exec -it ssh-copy-id

Edit /etc/ansible/hosts to include all nodes
# docker exec -it ansible-master /bin/bash
ansible-master: # cat >> /etc/ansible/hosts <<EOF
ansible-client1 ansible_host=


Test ansible using ping module
# docker exec -it ansible-master -m ping all
localhost | SUCCESS => {
    "changed": false, 
    "ping": "pong"
ansible-client1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"

Congratulations, now you have your own mini ansible lab, using docker. You can add more clients as you wish later.

Tuesday, April 9, 2019

Troubleshooting sshd fail to start

I encountered this issue, and journalctl just does not give enough information on what has happened to make ssh failed to start. So after searching around, I found that I can check the /etc/ssh/sshd_conf for any syntax error, just by running it with extended test (-T) flag. What this flag do is, check the validity of /etc/ssh/sshd_config, throw out error if any, and exit. So to check the issue in the configuration file, just run:

# /usr/sbin/sshd -T

Another way is, you can also start sshd manually with debug flag (-d), and it will throw out any error that stopping it from starting:

# /usr/sbin/sshd -d

Monday, April 8, 2019

/boot Keeps Filling Up On Kernel Update

This is an issue I encountered in one of my friend's ubuntu 16.04 box. He tried to do kernel update, but the /boot keeps filling up with old initramfs image files, making the update process failed. Then I found a post here, that says that if /var/lib/initramfs-tools is not being cleaned up from old kernel files, /boot will keep on being filled up with old initramfs images. So to clean it up:

# uname -r 
# cd /var/lib/initramfs-tools
# rm `ls | grep -v 4.15.0-46`

Once cleaned up, update your current initramfs, using:

# update-initramfs -u -k all

where -u is to generate initramfs for current kernel, and '-k all' to generate initramfs for kernel version newer than current kernel.

Once that done, you can safely reboot your machine. It will be rebooted using the latest kernel.

Friday, March 22, 2019

Rename and Remount LVM Logical Volume

A usual scenario where rename and remount of LVM logical volume are needed, is when you install a box with a CentOS, and use the default LVM based partitioning scheme. This scheme will take 50G for your / partition, and the rest will be allocated to your /home, which is not practical. In this example, we will be remounting logical volume originally mounted to /home, to /var/lib/elasticsearch, renaming it along the way.

The original partition scheme 
# df -Th
Filesystem                   Type      Size  Used Avail Use% Mounted on
/dev/mapper/cl-root          xfs        50G  3.0G   48G   6% /
/dev/sda1                    xfs      1014M  179M  836M  18% /boot
/dev/mapper/cl-home xfs       965G   42G  924G   5% /home

Stop the service that might be using the partition that we are going to change
# systemctl stop elasticsearch

Rename the original directory to other name
# mv /var/lib/elasticsearch /var/lib/elasticsearch-old

Create a new directory to replace the one we already renamed above
# mkdir /var/lib/elasticsearch

Unmount /home
# umount /home

Check the name of volume group and logical volume
# lvs

Rename the logical volume to the one we desired. This is totally optional. You can use -t flag to test out the renaming process before proceeding
# lvrename -t cl home elasticsearch
# lvrename cl home elasticsearch

Change /etc/fstab to reflect on the new logical volume name and mount point. Test it out without the  -i option, before permanently make the change using the -i option
# sed 's/cl-home/cl-elasticsearch/g;s/\/home/\/var\/lib\/elasticsearch/g' /etc/fstab
# sed -i 's/cl-home/cl-elasticsearch/g;s/\/home/\/var\/lib\/elasticsearch/g' /etc/fstab

Mount it
# mount -a

Check to see if your new directory and logical volume mounted properly
# df -Th
Filesystem                   Type      Size  Used Avail Use% Mounted on
/dev/mapper/cl-root          xfs        50G  3.0G   48G   6% /
/dev/sda1                    xfs      1014M  179M  836M  18% /boot
/dev/mapper/cl-elasticsearch xfs       965G   42G  924G   5% /var/lib/elasticsearch

Move the content of /var/lib/elasticsearch to /home
# mv /var/lib/elasticsearch/* /home

Move the content of /var/lib/elasticsearch-old to /var/lib/elasticsearch
# mv /var/lib/elasticsearch-old/* /var/lib/elasticsearch

Remove /var/lib/elasticsearch-old
# rmdir /var/lib/elasticsearch-old

Set proper permission for /var/lib/elasticsearch
# chmod 750 -R /var/lib/elasticsearch
# chown -R elasticsearch.elasticsearch /var/lib/elasticsearch

Start your service
# systemctl start elasticsearch


Thursday, March 7, 2019

Run an Application with GUI with Docker

There are a few ways to achieve this, like vnc and ssh X forwarding. But in this post, I will show how to run firefox by sharing the .Xauthority file with the docker instance.

First, create a Dockerfile in a directory called docker that use an ubuntu image from dockerhub, and install the firefox application

$ mkdir docker
$ cd docker
$ cat > Dockerfile
FROM ubuntu
RUN apt update && apt install -y firefox
CMD ["/usr/bin/firefox"]

Press ctrl-d to save the file

Next, build an image using the above Dockerfile

$ sudo docker build --tag=firefox-app docker/

Then, run the image, with additional options of --network and --env, and sharing our host .Xauthority file with the container that we are going to run 

$ sudo docker run --network=host --env="DISPLAY" --volume="$HOME/.Xauthority:/root/.Xauthority:rw" --name=firefox1 firefox-app

You should be able to get a working firefox, running from a docker container, separated from the current firefox that you have in your current machine