Trouble with hardware-assisted encoding/decoding via FFmpeg on Azure GPU vm's (ubuntu 16.04)

I am trying to use NVIDIA hardware acceleration with FFmpeg/libav, but can’t get it to work correctly on Azure vm’s running Ubuntu 16.04. For a sample case, I am trying to do a simple decoding of an h264 video into a raw YUV file.

So far, I’ve tried it on NC-6, NC-12, and NV-6 machines (in different regions). In each of these instances, it would take about 30-45 seconds to process a single video frame. As a comparison, I also tried it on a P2.xlarge vm on AWS (which has very similar specs to the NC-6), which was able to process about 3000 frames in about 5 seconds. Has anyone else run into this issue with Azure machines, or has any idea why this would be the case?

Here are the commands I used to install the necessary drivers/libraries/etc (I also verified that each machine as the same NVIDIA driver version installed - 375.51):
CUDA_REPO_PKG=cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
get -O /tmp/{CUDA_REPO_PKG} http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/{CUDA_REPO_PKG}
sudo dpkg -i /tmp/${CUDA_REPO_PKG}
sudo apt-get update
sudo apt-get install -y cuda-drivers
sudo apt-get install -y cuda
sudo apt-get install -y nvidia-cuda-toolkit
[reboot]
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
[reboot]
git clone https://github.com/FFmpeg/FFmpeg.git
[download the latest video codec SDK from NVIDIA at: https://developer.nvidia.com/designworks/video_codec_sdk/downloads/v7.1]
[unzipped codec, and copy header files from /Video_Codec_SDK_7.1.9/Samples/common/inc/ into /usr/include/]
cd ~/FFmpeg
./configure --enable-nonfree --disable-shared --enable-nvenc --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags=-Ilocal/include --extra-cflags=-I…/nv_sdk --extra-ldflags=-L…/nv_sdk
sudo make && sudo make install

For the FFmpeg command that I used to decode a sample movie file, I used the following:
sudo ffmpeg -vsync 0 -c:v h264_cuvid -i sample_vid.mp4 -f rawvideo outputvid.yuv

Doing this:

sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y

after this:

sudo apt-get install -y cuda-drivers
sudo apt-get install -y cuda

may very well break your cuda install

Follow the instructions in the linux install guide to install CUDA. Thereafter, if you upgrade packages that are involved with CUDA or packages that are involved with your kernel, you may break your CUDA install. To check whether anything bad happened during your install, verify that your CUDA install is not broken. The verification steps are include din the linux install guide.

Also see this note about using the GPUs in Azure:

https://devtalk.nvidia.com/default/topic/1006244/cuda-setup-and-installation/which-step-leads-me-fail-to-find-my-nvidia-card-during-installing-cuda-and-cudnn-/post/5139798/#5139798

Thanks for your reply txbob. I tried it while making sure the upgrade/dist-upgrate happened before the cuda installs, but it didn’t help. Regardless, I was also able to verify a successful cuda installation using either process. I also looked into the link you provided about GPUs in Azure, and verified that I’m not running into the same problem as that user (I have no trouble locating my gpu device across all my machines).

Could it be a difference in the kernels? The Azure machines is using version 4.4.0-77-generic, while the AWS machine uses 4.4.0-1013-aws.