Expediting Files Copy to USB

Copying to usb is quite troublesome in linux, whereby the "cp" command or file manager reports that the file has been copied successfully, but if your usb drive has LED, you can see that the LED is flashing frantically as if it is doing some hard work in the background. 

This is due to the default ubuntu linux setting for kernel parameter called vm.dirty_bytes is being set 0 (unlimited), which means that after copying process has been started, the file will be copied to buffer as a whole, the process that started the copying (file manager or "cp" command) will be notified that copy has completed, but in the background, the file is actually sitting in the buffer, waiting to be written to the usb driver.

This will results in the file manager showing 100% copied, but the file is not actually being written to the usb.

One of the way to expedite the copy process is, by limiting the size of the buffer, so the file will get written to the usb faster. 

You can check what is the current buffer (vm.dirty_bytes) value by running:

$ sudo sysctl vm.dirty_bytes

vm_dirty_bytes = 0

Change the current value to something small (like 15MB)

$ sudo sysctl vm.dirty_bytes=15000000

Start the copying process

$ cp some-big-file /mnt

The copy process will complete faster.

To make the change permanent:

$ echo "vm.dirty_bytes=15000000" | sudo tee -a /etc/sysctl.conf 

Reboot your machine

$ sudo reboot

Once rebooted, check whether the value stays

$ sudo sysctl vm.dirty_bytes

vm_dirty_bytes = 15000000

Done.

Credit to:

- https://askubuntu.com/questions/1117832/copying-files-from-pc-to-pendrive-gets-stuck-at-the-end-on-ubuntu-16-04

- https://unix.stackexchange.com/questions/180818/gnome-nautilus-copy-files-to-usb-stops-at-100-or-near

Replacing Single Line Spacing with Double Line Spacing in VIM

Thanks to this post, I learned on how to replace blank lines in my text document into 2 blank lines, to make it neat and clear.

The original document, with single line spacing

My objective is to replace all single blank lines representing new lines, into 2 new lines instead. To do that in vim, I just need to press "escape" to enter command mode and type the command below, which is ":%s/^$/\r/g", where
%s is to replace all occurrence of the selection in a line
^$ for single blank lines
\r carriage return, which will print double blank lines

The result should be like below 

Save and quit as usual, and you get yourself a neater document with double line spacing. Enjoy :)

Upgrading ubuntu server 16.04 to 18.04

The process is very simple, but quite time consuming, and a fast internet will help speed up the process.

To start, upgrade all packages to latest, and reboot if necessary

$ sudo apt update -y

$ sudo apt upgrade -y

$ sudo reboot

Once rebooted, and all packages are updated to the latest version, issue a 

$ sudo do-release-upgrade

Answer yes(y) to all questions, and answer "Keep the local version currently installed" for all questions asking to change any configuration to avoid any issue with currently installed applications

Once completed, press y to restart

Login and check your current version

$ cat /etc/os-release

SSH Too many authentication failures error

I encountered this error while trying to ssh into one of my client's machine, one day.

$ ssh web
Received disconnect from 192.168.0.36 port 22:2: Too many authentication failures
Disconnected from 192.168.0.36 port 22

After searching around, I found an article that showed that, I can solve this issue just by adding one flag to my ssh command

$ ssh web -o IdentitiesOnly=yes 

sam@192.168.25.36's password: 

Now we're talking. I found out that the reason of this behavior is, I accidentally offered too many private keys to the server, causing the server to reach its maximum MaxAuthRetries, and terminate the connection. You can see this by using -v (for verbose) while doing ssh.

To solve this issue for each session, just add IdentitiesOnly=yes option to your ssh command

$ ssh -o IdentitiesOnly=yes web

To make it permanent, edit ~/.ssh/config file, and add below lines

$ cat >> ~/.ssh/config <<EOF

Host web

  IdentitiesOnly yes

EOF

And you are good to go :)

Adding GPU as Resource for Slurm

To make a GPU part of resource that can be managed by Slurm, create /etc/slurm-llnl/gres.conf file with definitions of GPUs available on the node. GRES stands for generic resources, and need to be declared so that slurm can manage it.


Below example is for a node with nvidia tesla v100 gpu. Name - name of the resource, can be gpu, nic or mic
Type - arbitrary string identifying the type of device
File - Fully  qualified pathname of the device files associated with a resource
Cores - specific cpu core numbers, which can use this resource
$ sudo cat /etc/slurm-llnl/gres.conf
Name=gpu Type=v100 File=/dev/nvidia0 Cores=0,1

Add GresTypes and gres resources in slurm.conf.
The format for gres resources is grestype:optional-type:number-of-resource
$ sudo cat /etc/slurm-llnl/slurm.conf
...
GresTypes=gpu
NodeName=mynode CPUs=12 RealMemory=64091 Sockets=1 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:v100:1
...


Restart slurm services to have the changes take effect
$ sudo systemctl restart slurmd


Check the availability of the gres
$ scontrol show node