Thursday, October 17, 2019

SSH Too many authentication failures error

I encountered this error while trying to ssh into one of my client's machine, one day.

$ ssh web
Received disconnect from port 22:2: Too many authentication failures
Disconnected from port 22

After searching around, I found an article that showed that, I can solve this issue just by adding one flag to my ssh command
$ ssh web -o IdentitiesOnly=yes 
sam@'s password: 

Now we're talking. I found out that the reason of this behavior is, I accidentally offered too many private keys to the server, causing the server to reach its maximum MaxAuthRetries, and terminate the connection. You can see this by using -v (for verbose) while doing ssh.

To solve this issue for each session, just add IdentitiesOnly=yes option to your ssh command
$ ssh -o IdentitiesOnly=yes web

To make it permanent, edit ~/.ssh/config file, and add below lines
$ cat >> ~/.ssh/config <<EOF
Host web
  IdentitiesOnly yes

And you are good to go :)

Tuesday, October 1, 2019

Adding GPU as Resource for Slurm

To make a GPU part of resource that can be managed by Slurm, create /etc/slurm-llnl/gres.conf file with definitions of GPUs available on the node. GRES stands for generic resources, and need to be declared so that slurm can manage it.

Below example is for a node with nvidia tesla v100 gpu. Name - name of the resource, can be gpu, nic or mic
Type - arbitrary string identifying the type of device
File - Fully  qualified pathname of the device files associated with a resource
Cores - specific cpu core numbers, which can use this resource
$ sudo cat /etc/slurm-llnl/gres.conf
Name=gpu Type=v100 File=/dev/nvidia0 Cores=0,1

Add GresTypes and gres resources in slurm.conf.
The format for gres resources is grestype:optional-type:number-of-resource
$ sudo cat /etc/slurm-llnl/slurm.conf
NodeName=mynode CPUs=12 RealMemory=64091 Sockets=1 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:v100:1

Restart slurm services to have the changes take effect
$ sudo systemctl restart slurmd

Check the availability of the gres
$ scontrol show node

Installing Slurm Workload Manager & Job Scheduler on Ubuntu 18.04

Enable universe repository
$ echo "deb bionic universe" | sudo tee -a /etc/apt/sources.list

Update package list
$ sudo apt update

Install slurm-wlm
$ sudo apt install slurm-wlm -y

Install slurm documentation. This is useful to generate slurm.conf using configurator.easy.html page
$ sudo apt install slurm-wlm-doc -y

Get a machine with a web browser, and open /usr/share/doc/slurm-wlm-doc/html/configurator.easy.html to easily generate slurm.conf.

You can also access the configurator online at, but depending on your slurm version, the online version might not be suitable.

Fill up the form, some of the information can be retrieved using command
$ slurmd -C

Some of the configuration that I changed from the default
- Make sure the hostname of the system is ControlMachine and NodeName
- State Preservation: set StateSaveLocation to /var/spool/slurm-llnl
- Process tracking: use Pgid instead of Cgroup
- Process ID logging: set this to /var/run/slurm-llnl/ and /var/run/slurm-llnl/

Once done, click submit, and copy the generated config file to /etc/slurm-llnl/slurm.conf. Below is my sample config, with only one node
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
NodeName=myserver CPUs=1 State=UNKNOWN
PartitionName=debug Nodes=myserver Default=YES MaxTime=INFINITE State=UP

Create slurm spool directory
$ sudo mkdir /var/spool/slurm-llnl
$ sudo chown -R slurm.slurm /var/spool/slurm-llnl

Create slurm pid directory
$ sudo mkdir /var/run/slurm-llnl/
$ sudo chown -R slurm.slurm /var/run/slurm-llnl

Start and enable the slurm manager on boot
$ sudo systemctl start slurmctld
$ sudo systemctl enable slurmctld

Start slurmd and enable on boot
$ sudo systemctl start slurmd
$ sudo systemctl enable slurmd

If somehow slurmcrld or slurmd failed to start, run the applications interactively with debug options, to check for any errors. If there is any error, adjust slurm.conf accordingly.
$ sudo -u slurm slurmctld -Dcvvv
$ sudo slurmd -Dcvvv

Check slurm ndoes using scontrol command
$ scontrol show node

Wednesday, September 25, 2019

3 Ways To Locate Your php.ini File

Sometimes php installation comes with a bunch of php.ini files, that we do not know which one is actually being used by our currently running php. Below are 3 ways to locate your php.ini file

1. Using "php --ini" command
$ php --ini
Configuration File (php.ini) Path: /etc
Loaded Configuration File:         /etc/php.ini

2. Running phpinfo()
$ php -r 'phpinfo();' | grep ini
Configuration File (php.ini) Path => /etc
Loaded Configuration File => /etc/php.ini

3. Using php -i command
$ php -i | grep ini
Configuration File (php.ini) Path => /etc
Loaded Configuration File => /etc/php.ini

Sunday, September 15, 2019

Quickly testing live usb using qemu-system

You will need qemu for this. Install qemu-system for x86 machine (intel based)

$ sudo apt install qemu-system-x86

Attach your usb, check the device name using lsblk
$ lsblk

Test run your live usb using qemu-system-x86_64 command. Let's say your usb device is /dev/sdb
$ sudo qemu-system-x86_64 -hda /dev/sdb

If you get kernel panic in qemu when you run the above command, assign higher memory to the command, because qemu by default only assign 128MiB
$ sudo qemu-system-x86_64 -m 512 -hda /dev/sdb