Extending LVM By Adding New Disk To a Virtual Machine

In previous post, we have covered the way to make LVM and filesystem aware of the disk size increase that happen in the virtual machine layer. 

There is another way to increase LVM volume capacity, that is to add new disk into the system. 

1. First, shutdown the virtual machine

2. Add a new virtual disk size into the virtual machine manager

3. Start the virtual machine

4. Check the new disk availability for LVM

$ sudo lvmdiskscan

5. To be able to use the new disk, it needs to be converted into physical volume (PV)

$ sudo pvcreate <path to the new disk>

6. Check list of PVs. Memorize the PV name

$ sudo pvs

7. Extend the volume group using the new PV

$ sudo vgextend <VG name> <PV name>

We can get the name of the VG by using this command. This command also will show how much free space is now available in the VG.

$ sudo vgs

8. Increase the size of logical volume, to use 100% of the free space available inside VG, using this command

$ sudo lvextend -l +100%FREE <LV PATH>

We can get the full  path of LV using this command

$ sudo lvdisplay | grep Path

9. Now, make the filesystem aware that the partition size has been increased. The command to do this differ from filesystem to filesystem, but we include 2 of the most used (based on our experience) filesystem

For xfs:

$ sudo xfs_growfs <mountpoint>

For  ext4:

$ sudo resize2fs <mountpoint>

10. Verify that the mountpoint is now increase in size

$ df -Th

Troubleshooting proxmox pve-cluster down

Out of the blue, one of my proxmox node becomes offline in the web ui. All the VMs in that node is still working though, and the node is reachable via ssh, so network is not an issue. After searching around, it is because pve-cluster is down on that node, causing it to be invisible to the cluster altogether.

Looking around for clue, I found out that /etc/pve is empty, and "ls /etc/pve" returned "transport endpoint is not connected". journalctl command returned something more subtle, as below:

"host1 pmxcfs[29698]: [main] crit: fuse_mount error: Transport endpoint is not connected”

After searching around some more, I found out that the mountpoint is gone, and I need to remount /etc/pve back to get pve-cluster running.

Normal and forced unmount does not work. Thanks to this post, lazy unmount worked. So in order to restart pve-cluster, I have to lazy unmount /etc/pve first, the restart the pve-cluster.

# umount -l /etc/pve

# systemctl restart pve-cluster

# systemctl status pve-cluster

...

Active: active 

...

And my node is now back online in the proxmox webui.

Downloading ISO Directly to Proxmox Using Command Line

This is usually the faster way to get any iso to proxmox, especially if you are connecting to proxmox via some not so fast internet connection. But if you still prefer to use the web GUI, please refer to this post on how to do it via the web UI.

Go to the directory where iso's are being kept in proxmox
# cd /var/lib/vz/template/iso

If you have not downloaded the ISO, you can download it directly in this directory. For example I want to download a Centos 8 ISO
# wget http://centos.ipserverone.com/centos/8-stream/isos/x86_64/CentOS-Stream-8-x86_64-20191219-dvd1.iso

Once done, refresh the webUI

Proxmox VE No Subscription Repository

When you install Proxmox VE, by default the system will be configured to use Proxmox VE enterprise repository, which needs subscription in order to use it.

If you do not plan on purchasing the subscription, but still want to get updates on the latest package, you can use Proxmox VE no-subscription repository.

According to this document, this repository is recommended only for non production, as the packages are not heavily tested. Proceed at your own risk if you want to use it for production.

To enable the pve no-subscription repository

$ echo "deb http://download.proxmox.com/debian/pve buster pve-no-subscription" | sudo tee -a /etc/apt/sources.list

To disable the enterprise repository

$ cd /etc/apt/sources.list.d/

$ sudo mv pve-enterprise.list pve-enterprise.list.disabled

Update package information

$ sudo apt update

You should now be seeing the new no-subscription repository being checked, and the enterprise repository being left out

Windows in Proxmox Cannot Detect Virtio Network Adapter

I encountered this issue when installing Windows 2019 on proxmox Virtual Environment 6.1-7.

Below is my windows VM setting

In windows, I cannot see any network adapter to be configured.

After looking around in proxmox documentation, I found this gem, whereby you can download driver for the windows so that your windows can recognize the virtio interface.

You can download the iso for the driver from the above link, or using this direct link.

Once downloaded, upload the iso to proxmox using these steps.

Insert it into our windows vm cd drive, like below

You should now be able to see the iso mounted as D drive (depending on how many partition you have in your windows VM)

Now you have to install the driver. Press windows + s, type Device Manager and press Enter.

Expand "Network Adapters", you should see one adapter that is not recognized by windows.

Right click on it, and choose "Update driver"

Click on "Browse my computer for driver software"

Type in D:\ for the location, and tick "Include subfolders"

Click Next, and the wizard will search for a suitable driver for your device. Wait for it to finish.

Once done, it will show that your windows now has recognized the network adapter.

Reboot, if the wizard asks for it.

Done
     

Uploading iso to Proxmox

To upload iso to proxmox, so that the iso can be used to create VM, please follow below steps

1. Login to proxmox

2. On the left hand side, expand Datacenter, until you see node/s

3. Expand the node until you see local. Click on "local"

4. On the right hand side, click "Content"

5. Click "Upload"

6. Select the iso file that you want to upload, and click "Upload"

7. Once uploaded, you will see the iso listed on the right hand side. Please refer below picture for clearer view

Install and Use Vagrant on Linux Mint 19 with Virtualbox

Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past.

We can always install vagrant from the repository, but the one in the repository is quite an old version, which is 2.0.2. So we will download the latest version from https://www.vagrantup.com/downloads.html. Since they do not provide a package for ubuntu/mint, we will use debian package.

To download the installer

$ wget -c https://releases.hashicorp.com/vagrant/2.2.7/vagrant_2.2.7_x86_64.deb 

To install vagrant

$ sudo apt install ./vagrant_2.2.7_x86_64.deb -y

Download virtualbox, if you still have not install it

$ wget -c https://download.virtualbox.org/virtualbox/6.1.4/virtualbox-6.1_6.1.4-136177~Ubuntu~bionic_amd64.deb

Install  virtualbox

$ sudo apt install ./virtualbox-6.1_6.1.4-136177~Ubuntu~bionic_amd64.deb -y

Let's say we want to create an ubuntu bionic virtual machine (VM) using vagrant. First we have to run init

$ mkdir -p  vagrant/ubuntu-bionic

$ cd vagrant/ubuntu-bionic

$ vagrant init hashicorp/bionic64

$ vagrant up

The init command will create a standard Vagrantfile, which can be used to define the specs of virtual machine we want to make.

The up command will use the Vagrantfile, and create a VM out of it.

Once all the downloading is done, the VM will be created, and you can login into the box using ssh command

$ vagrant ssh

And within just a few commands, you already have a working virtual machine without having to do the standard operating system installation just like when you create a virtual machine using the traditional way.

Backup RHEV/RHV/ovirt disk image to external drive

Login to RHV/RHEV/ovirt administrator interface

Search for your VM, lets say I want to search for windows VM

Right click on the VM name, and choose 'Create Snapshot'

Give a proper name, and tick "Save Memory" if you want to save current state of memory as well. Please be warned, that by choosing "Save Memory", the VM will be paused while the snapshooting is in progress. 

If you check the Snapshots tab, the status of the snapshot will be Locked, and the VM will be paused

Once completed, the status will be changed to Ok

Now, open a terminal inside the rhev machine, and search for the disk image snapshot by running below command:

# virsh -r dumpxml windows | grep "source file"

Your newly created snapshot is on the last line of the above command. Copy the file location, and put it into your external drive

# cp /rhev/data-center/00000002-0002-0002-0002-00000000035d/747b7d84-68d5-4436-98f5-baeec68381e1/images/8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/../8af6c761-94ac-4adb-bf33-f009e3dd3dc7/e42c8e85-9566-4953-b881-70c7ce97de0b /mnt/my_external_drive/windows-snapshot-20181219.raw

If you want a different format, use qemu-img command to change it. For example, you require this image to be vmware compatible

# qemu-img -f raw -O vmdk /mnt/my_external_drive/windows-snapshot-20181219.raw /mnt/my_external_drive/windows-snapshot-20181219.vmdk

How to test that your usb live cd is working

To do this, you need qemu-kvm. Install it:
$ sudo apt install qemu-kvm


Check the address of your usb, by using dmesg
$ dmesg | grep sd


Once you have the address, you can start it using below command (make sure you allocate at least 1024M memory, else it will return kernel panic)
$ sudo qemu-system-x86_64 -m 1024 -hda /dev/sdb


To use a cdrom, you can use below command
$ sudo qemu-system-x86_64 -m 1024 -cdrom /dev/cdrom


You can also test an iso file, using below command
$ sudo qemu-system-x86_64 -m 1024 -cdrom centos.iso

Adding new virtual hard disk to existing RHV virtual machine

Login to your RHV/RHEV/ovirt console

Enter your administrator username and password



Once inside, search for your VM name

Scroll down, and choose Disk tab

Add details of your new virtual hard disk and click OK once done.



Make sure that your newly created disk is listed in the Disk tab

Login to your vm, and run lsblk to check your newly created disk
# lsblk 
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
...
vdc               252:32   0   200G  0 disk 

Create partition on the new disk

# cfdisk /dev/vdc

Choose New --> Primary --> Set size --> Write --> answer 'yes' --> Quit

Run lsblk again to check if your partition is successfully created 

# lsblk 
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
...
vdc               252:32   0   200G  0 disk 
└─vdc1            252:33   0   200G  0 part 



Make filesystem for your partition, in this case I want an ext4 partition
# mkfs.ext4 /dev/vdc1 
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
...
Writing superblocks and filesystem accounting information: done   

Once done, you can mount your partition wherever you want
# mount /dev/vdc1 /mnt
# df -Th /mnt/
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vdc1      ext4  197G   61M  187G   1% /mnt

Score!!

Hot attach and hot detach network interface to kvm guest

To hot attach a network interface to a kvm guest, please follow below steps. The command we'll be using is virsh:

1. Get to know the name of the guest, run below command on the kvm host: 
   

   foo@host:~$ sudo virsh list
   
    Id Name                 State
   ----------------------------------
     1 kvm-guest running

2. Check whether module acpiphp is loaded on the guest: 
   

   foo@guest:~$ sudo lsmod | grep -i acpiphp
   

3. If yes, proceed to step 4. If no, run below command:
   

   foo@guest:~$ sudo modprobe acpiphp

4. Hot attach the network interface:
   

   foo@host:~$ sudo virsh attach-interface kvm-guest network --model virtio --persistent
   Interface attached successfully

5. Run dmesg on guest to verify that the interface has been attached successfully:
   

   foo@guest:~$ dmesg | tail 
   [38613567.591261] virtio-pci 0000:00:04.0: using default PCI settings
   [38613567.591283] pci 0000:00:05.0: no hotplug settings from platform
   [38613567.591285] pci 0000:00:05.0: using default PCI settings
   [38613567.591741] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)
   [38613567.593361] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
   [38613567.601486] virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10
   [38613567.601524] virtio-pci 0000:00:05.0: setting latency timer to 64
   [38613567.602328] virtio-pci 0000:00:05.0: irq 43 for MSI/MSI-X
   [38613567.602343] virtio-pci 0000:00:05.0: irq 44 for MSI/MSI-X
   [38613567.602357] virtio-pci 0000:00:05.0: irq 45 for MSI/MSI-X

6. Set ipaddress for the new interface:
   

   foo@guest:~$ sudo touch /etc/sysconfig/network-scripts/ifcfg-eth1; sudo echo -e "DEVICE=eth1\nONBOOT=yes\nTYPE=Ethernet\nBOOTPROTO=static\nIPADDR=10.0.0.8\nNETMASK=255.255.255.0" > /etc/sysconfig/ifcfg-eth1

7. Bring up the interface:

   foo@guest:~$ sudo ifup eth1

8. Check the interface:
   

   foo@guest:~$ ifconfig eth1
   eth1      Link encap:Ethernet  HWaddr 52:54:00:D7:10:04
             inet addr:10.0.0.8  Bcast:10.0.0.255  Mask:255.255.255.0
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:178767965 errors:0 dropped:0 overruns:0 frame:0
             TX packets:58477452 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:1000
             RX bytes:11949338417 (11.1 GiB)  TX bytes:498944480375 (464.6 GiB)

Done.

For hot detaching, the command in virsh is detach-interface, please follow below steps to detach the newly added interface in the above instruction:

1. Bring down the interface in the guest:

   foo@guest:~$ sudo ifdown eth1

2. Delete the interface config file:

   foo@guest:~$ sudo rm /etc/sysconfig/network-scripts/ifcfg-eth1

3. Detach the network interface in host:

   foo@host:~$ sudo detach-interface kvm-guest type network --mac 52:54:00:D7:10:04

4. Verify that the network has been removed, by running the dumpxml command, pipe to less, and search for interface:

   foo@host:~$ sudo virsh dumpxml kvm-guest | less

Done.

Hope you all will gain benefit from this post.

How to know if the machine you are working on are a VM

There are a few ways you can detect if the machine you are working on is a VM. Below are commands to use and the output if your machine is a VM:

1. Use dmidecode command, dmidecode is a tool for dumping a computer DMI (some say SMBIOS) table contents in a human-readable format. So if your machine is a vm, you should not get any output:

   [somebody@theserver ~]$ sudo dmidecode
    # dmidecode 2.10
    /dev/mem: mmap: Bad address

2. Try check on /proc/scsi/scsi, if it is a vm, you would not get any attached device:

   [somebody@theserver ~]$ cat /proc/scsi/scsi 
   Attached devices:

   
   
3. Use lspci and see if there is any virtual related devices:

   [somebody@theserver ~]$ /sbin/lspci 
   00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
   00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
   00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
   00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
   00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
   00:02.0 VGA compatible controller: Cirrus Logic GD 5446
   00:03.0 Ethernet controller: Red Hat, Inc Virtio network device
   00:04.0 Ethernet controller: Red Hat, Inc Virtio network device
   00:05.0 SCSI storage controller: Red Hat, Inc Virtio block device
   00:06.0 RAM memory: Red Hat, Inc Virtio memory balloon
   00:07.0 SCSI storage controller: Red Hat, Inc Virtio block device

4. Check on your network interface:

   [somebody@theserver ~]$ /sbin/ethtool -i eth0
   driver: virtio_net
   version: 
   firmware-version: 
   bus-info: virtio0

5. Use dmesg and search for word virt, KVM, vmware, xen:

   [somebody@theserver ~]$ dmesg | grep -i vir
   Booting paravirtualized kernel on KVM
   CPU0: Intel QEMU Virtual CPU version (cpu64-rhel6) stepping 03
   input: Macintosh mouse button emulation as /devices/virtual/input/input1
   virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
   virtio-pci 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11
   virtio-pci 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ
   

   [somebody@theserver2:~]$ dmesg | grep -i -e virt
   [    0.000000] Booting paravirtualized kernel on Xen
   [    0.102602] input: Macintosh mouse button emulation as /devices/virtual/input/input0
   [    3.577404] Initialising Xen virtual ethernet driver

   
   

Cloning virtual machine on vmware esx using vmware-cmd

Below are the steps:

1. List down all the virtual machines that you have

# /usr/bin/vmware-cmd -l
/vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-1/server-1.vmx
/vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-2/server-2.vmx
/vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-3/server-3.vmx

2. Shutdown the machine that you are going to clone. Let say the machine is server-1

# /usr/bin/vmware-cmd /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-1/server-1.vmx stop

3. Use hard stop if you cannot stop it using above command

# /usr/bin/vmware-cmd /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-1/server-1.vmx stop hard

4. Make sure the vm is shutdown by viewing the state

# /usr/bin/vmware-cmd /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-1/server-1.vmx getstate

5. Copy the whole directory

# cp /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/server-1/ /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/newserver/

6. Once done, rename all the files to new name

# rename server-1 newserver /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/newserver/*

7. Change all occurrence of "server-1" to "newserver" in .vmx, .vmdk and .vmxf

# cd /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/newserver/
# sed -i 's/server-1/newserver/g' {newserver.vmdk,newserver.vmx,newserver.vmxf}

8. Edit newserver.vmx file and remove below lines

ethernet0.generatedAddress = "....."
ethernet0.generatedAddressOffset = "....."
uuid.location = "....."
uuid.bios = "....."

9. Add below line to newserver.vmx

uuid.action = "keep"

10. Register your newly cloned virtual machine. Please use the full path to register as below

# vmware-cmd -s register /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/newserver/newserver.vmx

11. Start your new vm after registration.

# /usr/bin/vmware-cmd /vmfs/volumes/47971de9-34657d96-8ac4-001d09068c0c/newserver/newserver.vmx start

Xen's domU always depend on dom0's time and date

By default, the time and date in domU is following dom0. To reset that, below are just what you have to do to decouple the domU's date from dom0

On dom0:

Append xen.independent_wallclock=1 to /etc/sysctl.conf

# echo "xen.independent_wallclock=1"  >> /etc/sysctl.conf

Activate the change

# sysctl -p

Append extra="clocksource=jiffies" to the domU's configuration file.

# echo "extra=\"clocksource=jiffies\"" >> /etc/xen/<domU's name>.cfg 

On domU:

Append xen.independent_wallclock=1

# echo "xen.independent_wallclock=1"  >> /etc/sysctl.conf 

Activate the change

# sysctl -p 

Append jiffies to /sys/devices/system/clocksource/clocksource0/current_clocksource

# echo "jiffies" >> /sys/devices/system/clocksource/clocksource0/current_clocksource

Now you can set your domU's date using "date" command or "ntpdate" to get update from ntp servers.

That's all folks

Installing virtualbox 2.2.0 on ubuntu 8.10

To install virtualbox is fairly easy. Just follow the instructions below:

1. Open /etc/apt/sources.list and add this line deb http://download.virtualbox.org/virtualbox/debian intrepid non-free

2. Import sun public-key and register it using apt-key (wget is for downloading, apt-key is for registering the key to your machine)
$ wget -q http://download.virtualbox.org/virtualbox/debian/sun_vbox.asc -O - | sudo apt-key add -
OK

3. Update your system's package index file
$ sudo apt-get update

4. Install virtualbox 2.2
$ sudo apt-get install virtualbox-2.2

Done, that's all :)

Resizing your xen DomU using LVM

To resize the disk space of a xen DomU that is using Logical Volume Manager(LVM) is very easy. Below is step by step on how to do the resizing process.

1. Create a new image with the size that you require. Just give any meaningful name to the image. In this case I will use the name extended.img
   
• # dd if=/dev/zero of=extended.img bs=1 count=1 seek=20G conv=notrunc
2. Add the new image to the configuration file of your DomU. In this example, the name of the domU is xen0
   
• # vi /etc/xen/xen0
• Add these line to it
• disk = [ 'tap:aio:/path/to/xen/xen0.img,xvda,w','file:/path/to/xen/extended.img,xvdb,w' ]
• Save
3. Start your domU
• # xm create xen0
4. Access your domU
• # xm console xen0
5. Once inside, check whether the new image is detected
• # fdisk -lu
6. After confirm that your new hard disk image is detected, it is time we have to work on the lvm
• Create new physical volume (PV) using the new hard disk image
  
• # pvcreate -v /dev/xvdb
• Check that you have successfully added the PV
• # pvdisplay
• Extend your existing volume group (VG) to include the new PV
• # vgextend -v VolGroup00 /dev/xvdb
• Check that you have successfully add the PV into the VG
• # vgdisplay
• Extend your logical volume (LV)
  
• # lvextend -L +20G -v /dev/VolGroup00/LogVol00
• Check that the extension has been added
• # lvdisplay
• If all the steps are successfully done, you have to resize the / partition
• # resize2fs /dev/mapper/VolGroup00-LogVol00
7. You are done. Check your new hard disk space :)
• # df -lh
  

Resizing qemu image

Qemu uses its own type of file which is qcow that stands for Qemu Copy On Write. Sometimes after you have setup the image, you suddenly find that you need more space. Below are the proven steps on how you can add more space to your image.

1. Convert qemu image to raw
• # qemu-img convert -f qcow imagename -O raw imagename.raw
2. Extend the image size using dd (In this example, to the size of 10G)
   
• # dd bs=1 if=/dev/zero of=/path/to/imagename.raw seek=10G count=1 conv=notrunc
3. Convert back to qemu image
• # qemu-img convert -f raw imagename.raw -O qcow imagename
4. Partition the extra disk space you have created using parted

That's all folk

Accessing data on xen lvm guest image

Accessing xen guest image is very easy if the image is not lvm partitioned. But the main problem arise when the image is of lvm format and normal mount command cannot be used. Here I will show both the way. The first is when ext filesystem is used, and the second is when lvm is used.

To mount xen guest image (without lvm)

1. check the partition on the image
# fdisk -lu
The result will be something like this:



2. Mount using offset option
# mount -o loop,offset=106929152 /path/to/image /mnt
where 106929152=208846*512, 208846 is the start of the partition. Using this way, you only mount the second partition and not the whole image

3. You can now access your image at /mnt

To mount xen guest image (with lvm)

1. Check the partition on the image
# fdisk -lu /path/to/image

2. You have to install kpartx to handle lvm partiton
# yum install kpartx

3. Run kpartx on the image
# kpartx -av /path/to/image

4. Run vgscan to scan volume group available
# vgscan

5. Run vgchange to activate the volume group in the image
# vgchange -ay VolGroup00

6. Use lvs to see what is the name of your low volume group
# lvs

7. Mount the low volume group
# mount /dev/VolGroup00/LogVol01 /mnt

8. You can access your lvm image at the mounted directory which is /mnt

9. To unmount it, a few commands have to be executed (umount for unmounting, vgchange -an to deactivate volume group, kpartx -d to delete device map and losetup -d to delete loop device used)
# umount /mnt/
# vgchange -an VolGroup00
# kpartx -d /path/to/image
# losetup -d /dev/loop0

Hope this will be useful