Nvidia Drivers - Docker

Hey, I just wanted to circle back around on this ✅ Hardware-accelerated encoding/transcoding - #14 by adamf

I was doing some testing with Photostructure alpha1 alongside Frigate NVR and Plex. I compared some outputs of ffmpeg and nvidia-smi.

First - if I run nvidia-smi on my host, I can see the containers using various processes. In this case, I have five security cameras and two instances of Plex transcoding a video with my P2000 card.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P2000        Off  | 00000000:01:00.0 Off |                  N/A |
| 61%   64C    P0    24W /  75W |    912MiB /  5059MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     20521      C   ffmpeg                            111MiB |
|    0   N/A  N/A     20523      C   ffmpeg                            111MiB |
|    0   N/A  N/A     20572      C   ffmpeg                            111MiB |
|    0   N/A  N/A     20574      C   ffmpeg                            111MiB |
|    0   N/A  N/A   2405990      C   ...diaserver/Plex Transcoder      177MiB |
|    0   N/A  N/A   2409847      C   ffmpeg                            111MiB |
|    0   N/A  N/A   2473381      C   ...diaserver/Plex Transcoder      177MiB |
+-----------------------------------------------------------------------------+

If I run the same incantation inside of Plex or Frigate, they show the RAM is in use but (and I’ve never figured this out) they do not display the processes that each container itself is using.

Frigate, for example:

docker exec -ti frigate sh -c 'nvidia-smi'
Mon Jun  6 20:26:07 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.129.06   Driver Version: 470.129.06   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P2000        Off  | 00000000:01:00.0 Off |                  N/A |
| 61%   63C    P0    24W /  75W |    912MiB /  5059MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Now, the way I pass through the GPU to both of those containers (and attempt to with Photostructure) is via a runtime and an environment flag. I’m snipping the rest for brevity.

  frigate:
    image: blakeblackshear/frigate:stable-amd64nvidia
    privileged: true
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video

Similarly, Plex:

  plex:
    image: lscr.io/linuxserver/plex
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all

Note that Plex functions just fine with regard to hardware transcoding without the capabilities being specified and without privilege so I don’t believe that is the problem.

When I add the same flags (visible devices and runtime) to Photostructure, I partly get what I want. Nvidia devices show up inside of /dev.

Issue: nvidia-smi is missing from Photostructure. It returns an error when attempting to run. This would indicate the drivers do not exist. I’m sure you already know they are not included as we discussed it on the prior thread, but I’m just going through my thought process.

Next I tested hwaccel. Plex’s container does not include it but here is Frigate and Photostructure.

Frigate

docker exec -ti frigate sh -c 'ffmpeg -hwaccels'
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-libsrt --enable-libaribb24 --enable-nvenc --enable-cuda --enable-cuvid --enable-libnpp --extra-cflags='-I/opt/ffmpeg/include -I/opt/ffmpeg/include/ffnvcodec -I/usr/local/cuda/include/' --extra-ldflags='-L/opt/ffmpeg/lib -L/usr/local/cuda/lib64 -L/usr/local/cuda/lib32/'
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Hardware acceleration methods:
cuda

Photostructure

docker exec -ti photostructure sh -c 'ffmpeg -hwaccels'
ffmpeg version 4.4.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 10.3.1 (Alpine 10.3.1_git20211027) 20211027
  configuration: --prefix=/usr --enable-avresample --enable-avfilter --enable-gnutls --enable-gpl --enable-libass --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-libdav1d --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-libxcb --enable-libsrt --enable-libssh --enable-libvidstab --disable-stripping --disable-static --disable-librtmp --enable-libaom --enable-libopus --enable-libsoxr --enable-libwebp --enable-vaapi --enable-vdpau --enable-vulkan --disable-debug
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Hardware acceleration methods:
vdpau
vaapi
vulkan

Issue: cuda does not appear. Again I’m sure this is driver related.

All that being said, Frigate’s solution to this is to host a separate dockerfile image for different use cases. He has an arch image and an arm image, but most importantly he has both an amd64 and an amd64 with nvidia support.

I’m wondering simply: How much of a lift would it be to do something similar here, either with a secondary image tag or the ability to trigger the drivers based on an environmental variable. (I’m not sure the latter is even a thing, but spitballing.)

Else, how would you recommend or what is the simplest way I go about adding them without having to fork and maintain the PS image myself? Currently to keep CPU load down I’ve disabled transcoding, and I could simply use CPU cycles to transcode, but I’d really prefer to leverage the P2000 if possible as it’s already serving me very well overall.

Thanks!

Dude, nice research!

So, I absolutely could publish another docker image that has cuda support.

I went through the (pretty big!) hassle of making PhotoStructure work nicely with Alpine so the docker image could be ~200MB instead of ~1.5GB.

The frigate-base image is Ubuntu. I just scanned all codepaths that are isDocker()-specific, and other than a couple minor bits (that will be fixed in alpha.2), things should work just fine.

FWIW, PhotoStructure’s Dockerfile is source-available: here’s the alpha version.

To make this happen, I’d need to convert base-tools and that Dockerfile to be Ubuntu-flavored. That shouldn’t be too painful.

I suspect the image, along with the CUDA driver, may push 1.5-2GB. That certainly isn’t a deal killer, but it’s certainly something to consider.

And holy crap do I suddenly want one of these, but yikes frigate-base install is scary.

1 Like

That would be great if it’s possible. (I figured you were super busy with everything else we were discussing around Alpha 2, and that this was the ideal time to post this :joy:)

I found one UK company with a US office that allegedly is shipping me a Coral accelerator this month - been waiting for 2 months already. On ebay they’re currently going (due to Frigate and Covid combined) for hundreds of dollars above MSRP.

What makes frigate base scary? I don’t know enough about docker files.