Adding GPU as Resource for Slurm

To make a GPU part of resource that can be managed by Slurm, create /etc/slurm-llnl/gres.conf file with definitions of GPUs available on the node. GRES stands for generic resources, and need to be declared so that slurm can manage it.


Below example is for a node with nvidia tesla v100 gpu. Name - name of the resource, can be gpu, nic or mic
Type - arbitrary string identifying the type of device
File - Fully  qualified pathname of the device files associated with a resource
Cores - specific cpu core numbers, which can use this resource
$ sudo cat /etc/slurm-llnl/gres.conf
Name=gpu Type=v100 File=/dev/nvidia0 Cores=0,1

Add GresTypes and gres resources in slurm.conf.
The format for gres resources is grestype:optional-type:number-of-resource
$ sudo cat /etc/slurm-llnl/slurm.conf
...
GresTypes=gpu
NodeName=mynode CPUs=12 RealMemory=64091 Sockets=1 CoresPerSocket=6 ThreadsPerCore=2 State=UNKNOWN Gres=gpu:v100:1
...


Restart slurm services to have the changes take effect
$ sudo systemctl restart slurmd


Check the availability of the gres
$ scontrol show node

Install nvidia driver for Tesla card on ubuntu 16.04

Check what driver your card is currently using, in this case, the card is using the opensource nvidia drivers called nouveau
$ lspci -v | grep -A8 NVIDIA
3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)
        Subsystem: NVIDIA Corporation Device 11d8
        Flags: bus master, fast devsel, latency 0, IRQ 40
        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]
        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities:
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau

Download the driver for your card from nvidia download website

Choose the correct card model, and operating system, and click SEARCH

Click DOWNLOAD

Click AGREE & DOWNLOAD to download

If you want to download using wget, right click on the AGREE & DOWNLOAD button, and choose "Copy Link Address". Run wget against the link

$ wget http://us.download.nvidia.com/tesla/410.79/nvidia-diag-driver-local-repo-ubuntu1604-410.79_1.0-1_amd64.deb

Install the driver

$ sudo apt install ./nvidia-diag-driver-local-repo-ubuntu1604-410.79_1.0-1_amd64.deb

Make sure you have now nvidiafb in kernel modules 

$ lspci -v | grep -A15 NVIDIA

3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)

        Subsystem: NVIDIA Corporation Device 11d8

        Flags: bus master, fast devsel, latency 0, IRQ 40

        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]

        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]

        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]

        Capabilities: [60] Power Management version 3

        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+

        Capabilities: [78] Express Endpoint, MSI 00

        Capabilities: [100] Virtual Channel

        Capabilities: [250] Latency Tolerance Reporting

        Capabilities: [128] Power Budgeting

        Capabilities: [420] Advanced Error Reporting

        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024

        Capabilities: [900] #19

        Kernel driver in use: nouveau

        Kernel modules: nvidiafb, nouveau

Update

$ sudo apt update

Install cuda-drivers

$ sudo apt install cuda-drivers

Reboot your machine

$ sudo reboot

Check whether your card is using nvidia driver, and not nouveau

$ lspci -v | grep -A8 NVIDIA

3d:00.0 3D controller: NVIDIA Corporation Device 1bb3 (rev a1)

        Subsystem: NVIDIA Corporation Device 11d8

        Flags: bus master, fast devsel, latency 0, IRQ 88

        Memory at b7000000 (32-bit, non-prefetchable) [size=16M]

        Memory at 4bfe0000000 (64-bit, prefetchable) [size=256M]

        Memory at 4bff0000000 (64-bit, prefetchable) [size=32M]

        Capabilities:

        Kernel driver in use: nvidia

        Kernel modules: nvidiafb, nouveau, nvidia_410_drm, nvidia_410