Requirements

If you wish to take advantage of GPU Acceleration, the following are required:

Installation

Certain pre-requisites must be installed:

Pre-requisites

  • GPU Acceleration requires CUDA 8.0 and cuDNN 6.0 for CUDA 8.0.

    • It is often easier to CUDA install from the NVIDIA site on Linux systems.
    • You will need to sign up for the NVIDIA Developer Program to download cudNN.

    During the installation process, Montblanc will inspect your CUDA installation to determine if a GPU-supported installation can proceed. If your CUDA installation does not live in /usr, it helps to set a number of environment variables for this to proceed smoothly. For example, if CUDA is installed in /usr/local/cuda-8.0 and cuDNN is unzipped into /usr/local/cudnn-6.0-cuda-8.0, run the following on the command line or place it in your .bashrc

    # CUDA 8
$ export CUDA_PATH=/usr/local/cuda-8.0
$ export PATH=$CUDA_PATH/bin:$PATH
$ export LD_LIBRARY_PATH=$CUDA_PATH/lib64:$LD_LIBRARY_PATH
$ export LD_LIBRARY_PATH=$CUDA_PATH/extras/CUPTI/lib64/:$LD_LIBRARY_PATH

# CUDNN 6.0 (CUDA 8.0)
$ export CUDNN_HOME=/usr/local/cudnn-6.0-cuda-8.0
$ export C_INCLUDE_PATH=$CUDNN_HOME/include:$C_INCLUDE_PATH
$ export CPLUS_INCLUDE_PATH=$CUDNN_HOME/include:$CPLUS_INCLUDE_PATH
$ export LD_LIBRARY_PATH=$CUDNN_HOME/lib64:$LD_LIBRARY_PATH

# Latest NVIDIA drivers
$ export LD_LIBRARY_PATH=/usr/lib/nvidia-375:$LD_LIBRARY_PATH

    If the installer cannot find CUDA it will only install the CPU version of tensorflow and only compile multi-threaded CPU operators.

  • casacore and the measures found in casacore-data. Gijs Molenaar has kindly packaged this on Ubuntu/Debian style systems.

    On Ubuntu 14.04, these packages can be added from the radio astronomy PPA :

    $ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:radio-astro/main
$ sudo apt-get update
$ sudo apt-get install libcasacore2-dev casacore-data

    On Ubuntu 16.04 these packages can be added from the kernsuite PPA:

    $ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:kernsuite/kern-1
$ sudo apt-get update
$ sudo apt-get install casacore-dev casacore-data

    Otherwise, casacore and the measures tables will need to be manually installed.

  • Check that the python-casacore and casacore dependencies are installed. By default python-casacore builds from pip and therefore from source. To succeed, library dependencies such as libboost-python must be installed beforehand. Additionally, python-casacore depends on casacore. Even though kernsuite installs casacore, it may not install the development package dependencies (headers) that python-casacore needs to compile.

Installing the package

Set the CUDA_PATH so that the setup script can find CUDA:

$ export CUDA_PATH=/usr/local/cuda-8.0

If nvcc is installed in /usr/bin/nvcc (as in a standard Ubuntu installation) or somewhere on your PATH, you can leave CUDA_PATH unset. In this case setup will infer the CUDA_PATH as /usr

It is strongly recommended that you perform the install within a Virtual Environment. If not, consider adding the --user flag to the following pip and python commands to install within your home directory.

$ virtualenv $HOME/mb
$ source virtualenv $HOME/mb/bin/activate
(mb) $ pip install -U pip setuptools wheel

Then, run:

(mb) $ pip install --log=mb.log git+git://github.com/ska-sa/montblanc.git@master

Installing the package in development mode

Clone the repository, checkout the master branch and pip install montblanc in development mode.

(mb) $ git clone git://github.com/ska-sa/montblanc.git
(mb) $ pip install --log=mb.log -e $HOME/montblanc

Possible Issues

  • Montblanc doesn’t use your GPU, install the GPU version of tensorflow or compile GPU tensorflow operators. The installation process attempts to find your CUDA install location. It will log information about where it thinks this is and which GPU devices you have installed. Check the install log generated by the pip commands given above to see why this fails, searching for “Montblanc Install” entries.

    It is possible to see if the GPU version of tensorflow is installed by running the following code in a python interpreter:

    import tensorflow as tf
with tf.Session() as S: pass

    If tensorflow knows about your GPU it will log some information about it:

    2017-05-16 14:24:38.571320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX 960M
major: 5 minor: 0 memoryClockRate (GHz) 1.176
pciBusID 0000:01:00.0
Total memory: 3.95GiB
Free memory: 3.92GiB
2017-05-16 14:24:38.571352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-05-16 14:24:38.571372: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y
2017-05-16 14:24:38.571403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0)

  • cub 1.6.4. The setup script will attempt to download this from github and install to the correct directory during install. If this fails do the following:

    $ wget -c https://codeload.github.com/NVlabs/cub/zip/1.6.4
$ mv 1.6.4 cub.zip
$ pip install -e .

  • python-casacore is specified as a dependency in setup.py. If installation fails here:

    1. Check that the python-casacore dependencies are installed.
    2. You will need to manually install it and point it at your casacore libraries.