GPU TensorFlow Installation Guide for Windows

Before we actually start the installation process of the GPU-accelerated Python API of TensorFlow on a Windows platform, we shortly outline what TensorFlow actually is.

What is TensorFlow?

  • TensorFlow is an open-source software library for machine learning that was originally invented by Google’s Brain Team but the code was open sourced in November 2015. It comprises a collection of resources for flexibly defining machine learning models, train them with data, and exporting them for further use. In particular, TensorFlow is useful  for building and training the different types of neural networks.
  • TensorFlow is available on several desktop platforms such as Windows, Linux and macOS as well as on mobile computing platforms.
  • TensorFlow can run on multiple CPUs and GPUs (with optional CUDA extensions for general-purpose computing on graphics processing units).
  • TensorFlow provides APIs for Python, C++, Haskell, Java, Go, and Rust. Third party packages are available for C#, Julia, R, and Scala.

How to install TensorFlow on Windows?

In this section we sketch how to install the Python API of TensorFlow in Windows using the GPU-acceleration capabilities. We assume that a Nvidia GPU is already installed in the Windows system:

Windows 10 Device Manager listing several Nvidia GPUs

The installation of a GPU is usually straightforward in Windows. We therefore skip this part completely.

TensorFlow’s official installation guideline provides a not that comprehensive description of the installation process. We therefore provide a more elaborated and detailed description, which should be understand as a supplementary to the official guidance.

Microsoft’s Visual C++, Nvidia’s CUDA Development Tools & Nvidia’s cuDNN

well as cuDNN Installation Guide. To this end, I recommend to install the  version 8.0 of the CUDA toolkit and version 5.1 of cuDNN as listed below.

  1. First, we need to get a C++ compiler and an IDE up and running since this is a prerequisite for a working CUDA environment. In this post we are going to install the free Visual Studio 2015 Community edition that comprises a powerful IDE as well as a suitable compiler.According to Table 2 of the official Nvidia CUDA Installation Guide for Microsoft Windows you need to have installed specific compiler versions for specific Visual Studio versions. In our case we need to install the Microsoft Visual C++ 14.0 compiler as we are going to install Visual Studio 2015. If the compiler is already installed it should look like the following:

    Apps and Features in Windows 10: Microsoft Visual C++ 14.0 Compiler as well as the Visual Studio Community 2015 edition
  2. Second, the CUDA Development Tools version 8.0 should be installed. I recommend to follow the official Nvidia CUDA Installation Guide for Microsoft Windows and to chose the express full installation including the CUDA samples (which is the default setting). Provided that the installation of the Visual Studio incl. the compiler  has been successful, the CUDA Toolkit installation procedure is basically just clicking through.  If you have installed the wrong version of the Visual Studio and corresponding compiler you will realize it as a message will pop up in the course of the installation process;
  3. Third, Nvidia’s cuDNN, version 5.1 (for CUDA toolkit version 8.0) needs to be installed. To this end, you need to sign up and log in to Nvidia’s website.The installation process of cuDNN is to extract the downloaded files, to copy those files into the right folder and to make sure that the corresponding path is known to the operating system;
  4. Fourth, ensure that the system environment variables such as CUDA_HOME, CUDA_PATH and CUDA_PATH_V8_0 are set correctly:
    Windows 10 – System Environment Variables for CUDA
    Windows 10 – Path System Environment Variables for CUDA and cuDNN

    Please ensure that the path to the file cudnn64_5.dll (for the cuDNN version 5.1) is set as displayed in the selected variable of exhibit “Path System Environment Variables for CUDA and cuDNN“. In addition, make also sure that “.DLL” files are included in the PATHEX variable.

    Windows 10 – PATHEXT Environment Variables
  5. Finally, I highly recommend to validate your C++/CUDA installation.
    Click the Windows start symbol and browse through the programs to find the Nvidia Corporation folder. Open the entry that is called Browse CUDA sample. The standard installation path of the CUDA samples is given in the following exhibit.

    Windows 10: CUDA Samples v8.0

    From there enter the folder 1_Utilities/deviceQuery and double-click the 2015 VC++ Project file named deviceQuery_vs2015.vcxproj. Since we have already installed Microsoft Visual 2015 in the first step of this post your operating system should be able to open the file. Then build the corresponding exe file and open the file deviceQuery.exe in the corresponding Debug folder which is in our case C:\ProgramData\NVIDIA Corporation\CUDA Samples\v8.0\bin\win64\Debug\. Open a command box and run this exe file. The output should look like the following:

    Validating the CUDA installation using deviceQuery.exe

    BTW, you can use this little program to request detailed information about your GPU. For instance, the exact number of CUDA cores, the memory clock rate and the total amount of global memory are listed.

    If you would like to chose another suitable program to check your CUDA system feel free to do so. There is, for instance, a folder 0_Simple\simpleMultiGPU that you can also build and execute to double-check the CUDA installation.

    You should also call nvcc -V command in a prompt. This command provides you with the installed version of the toolkit:

    Validate CUDA installation on Windows

    One possibility how to validate the cuDNN installation is described in this stack-overflow post.

Python

After the long preparation phase we can finally install Python on Windows. There are several ways to install Python. I recommend to install the latest version of Anaconda Python framework.

Even though the installation of Anaconda via the installation routine may be straightforward, I suggest to double-check whether the PATH variable (please see above) is set correctly. You can double-check this by executing python in a command prompt.

Anaconda Navigator – creating and activating new python environments

After the installation of Anaconda, we first create a new Python environment to encapsulate the installation from other Python projects. This can be either done via Conda commands (conda create -n tensorflow-gpu python=X.Y.Z) or via the graphical interface of Anaconda. Second, we need to install the package  tensorflow-gpu.

In order to create a new Python environment using Anaconda Navigator we simply need to click the “Create” button and name the new environment. The respective button is highlighted in the exhibit above. Alternatively, we could also use the command box executing conda install tensorflow-gpu.

We can activate an environment by clicking the symbol on the right of the environment.

Then we select the tensorflow-gpu package on the right-hand side of the Anaconda Navigator and install it as sketched in the following:

Using Anaconda to install the package tensorflow-gpu

After pressing the “Apply” button the respective packages will be installed. Alternatively, we could also use the command prompt executing  conda install tensorflow-gpu.

Now, your TensorFlow should be up and running. In order to validate this, use the command prompt and enter the following:

import tensorflow as tf
# Creates a graph.
with tf.device('/gpu:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
  b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))

You should be good to go!