Jetson Xavier AGX and Docker Checkpoint issue

fffonseca · December 13, 2022, 6:45pm

Hi there!

I been working in docker for a while and i found a problem running the example from docker checkpoint documentation: Link to Docker Checkpoint

Running the command i get this error:

 $ docker checkpoint create cr checkpoint1
Error response from daemon: Cannot checkpoint container cr: nvidia-container-runtime did not terminate successfully: exit status 1: criu failed: type NOTIFY errno 0 path= /run/containerd/io.containerd.runtime.v2.task/moby/702011d32f4b26bcc119793feb376169ca2bb287ba1f466caa6bdf68f4a608f9/criu-dump.log: unknown

After looking the criu-dump.log file i found this at the end:

(00.003611) Error (criu/proc_parse.c:2571): Can't open 12081/task/12081/children on procfs: No such file or directory
(00.003645) Collected (0 attempts, -1 in_progress)
(00.003714) Unlock network
(00.003802) Unfreezing tasks into 1
(00.003843) Error (criu/cr-dump.c:1709): Dumping FAILED.

My Jetson characteristics are:

Jetson Xavier AGX
Linux ubuntu 4.9.299-tegra #1 SMP PREEMPT Tue Nov 22 09:24:30 PST 2022 aarch64 GNU/Linux
R32 (release), REVISION: 7.3, GCID: 31982016, BOARD: t186ref, EABI: aarch64, DATE: Tue Nov 22 17:32:54 UTC 2022
docker version 20.10.12
CRIU Version: 3.6

I want to know why this error is happening and how to fully run the checkpoint flow work.

Regards.

AastaLLL · December 14, 2022, 4:28am

Hi,

Could you try if it works with sudo?

$ sudo docker ...

Thanks.

fffonseca · December 14, 2022, 4:33am

Thanks @AastaLLL.

The same message pops out:

Error response from daemon: Cannot checkpoint container cr: nvidia-container-runtime did not terminate
 successfully: exit status 1: criu failed: type NOTIFY errno 0 path= /run/containerd/io.containerd.runtime.v2.task
/moby/609c20a3ca67f7d1c4d1affb7ed0f21f5f9c7ba04b380506d1b88c7702d50c00/criu-dump.log: unknown

AastaLLL · December 21, 2022, 7:54am

Hi,

Based on the below ticket in NV container, checkpoint is not supported.

github.com/NVIDIA/nvidia-container-runtime

NVIDIA_VISIBLE_DEVICES makes checkpointing impossible

opened 10:25AM - 20 Nov 18 UTC

closed 08:21PM - 06 Jan 19 UTC

cbachhuber

## Issue Description First of all, thank you for this useful container! Now, …my problem: I want to perform a live migration of a running docker container. My container is using the [Nvidia runtime](https://github.com/NVIDIA/nvidia-docker) and [GPU passthrough to container](https://github.com/NVIDIA/nvidia-container-runtime) using `ENV NVIDIA_VISIBLE_DEVICES all`. Live migration is done through docker's experimental [checkpoint and restore](https://github.com/docker/cli/blob/master/experimental/checkpoint-restore.md) using [CRIU](https://criu.org/Main_Page). So I start my container ``` nvidia-docker run --name myApp -i app ``` which runs fine, and also exits normally if not checkpointed. However, if I create a checkpoint using ``` docker checkpoint create --leave-running=true myApp checkpoint1 ``` I get following error response: ``` Error response from daemon: Cannot checkpoint container myApp: nvidia-container-runtime did not terminate sucessfully: criu failed: type NOTIFY errno 0 path= /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/10eb4bf688652c5d8f612fca192c12c80cc59bce605cf3ff0a0e8a0e07ce17da/criu-dump.log: unknown ``` Inspecting file `criu-dump.log` leads me to the following error: ``` Error (criu/mount.c:925): mnt: Mount 448 ./proc/driver/nvidia/gpus/0000:01:00.0 (master_id: 13 shared_id: 0) has unreachable sharing. Try --enable-external-masters. ``` So it seems to be an issue with CRUI not being able to create a checkpoint of the GPU. The `--enable-external-masters` command is [not suitable for GPUs](https://criu.org/External_resources). Including `ENV NVIDIA_DRIVER_CAPABILITIES all` in the Dockerfile does not resolve the issue either. So my main question is if there is a way of integrating a GPU dump from the nvidia-docker container checkpoint. Proper support from Nvidia seems to be [not given](https://github.com/checkpoint-restore/criu/issues/534) yet, but there is Nvidia software for [live migrating between GPUs](https://www.nvidia.com/en-us/design-visualization/solutions/vgpu-migration/). So can we expect support for this kind of application, too? ## Steps to Reproduce Set docker to [experimental mode](https://stackoverflow.com/questions/44346322/how-to-run-docker-with-experimental-functions-on-ubuntu-16-04), [install criu](https://github.com/docker/cli/blob/master/experimental/checkpoint-restore.md#installing-criu) for docker, install nvidia driver, install [nvidia-docker](https://github.com/NVIDIA/nvidia-docker), install [nvidia-container-runtime](https://github.com/NVIDIA/nvidia-container-runtime#installation). Create your Dockerfile with the following two lines: ``` FROM ubuntu:16.04 ENV NVIDIA_VISIBLE_DEVICES all ``` Build the container ``` sudo docker build -t app . ``` When the container is built, run it ``` nvidia-docker run --name myApp -i app ``` where you have to waive the `-t` flag to [avoid issues](https://github.com/checkpoint-restore/criu/issues/76) with CRIU. Next, we try to create the checkpoint ``` docker checkpoint create --leave-running=true myApp checkpoint1 ``` upon which you will receive the error response from the issue description. ## System Information I'm attaching hopefully relevant system information, [as suggested in the nvidia-docker issues](https://github.com/NVIDIA/nvidia-docker/issues/748). Kernel version from `uname -a`: ``` Linux ECS 4.15.0-39-generic #42~16.04.1-Ubuntu SMP Wed Oct 24 17:09:54 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ``` Driver information from `nvidia-smi -a`: ``` ==============NVSMI LOG============== Timestamp : Tue Nov 20 09:37:42 2018 Driver Version : 384.130 Attached GPUs : 1 GPU 00000000:01:00.0 Product Name : GeForce GTX 1080 Product Brand : GeForce ``` Docker version from `docker version`: ``` Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:24:56 2018 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:23:21 2018 OS/Arch: linux/amd64 Experimental: true ``` Nvidia packages version from `dpkg -l '*nvidia*'` _or_ `rpm -qa '*nvidia*'`: ``` _or_: command not found Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-======================-================-================-================================================== ii libnvidia-container-to 1.0.0-1 amd64 NVIDIA container runtime library (command-line too ii libnvidia-container1:a 1.0.0-1 amd64 NVIDIA container runtime library ii nvidia-384 384.130-0ubuntu0 amd64 NVIDIA binary driver - version 384.130 ii nvidia-384-dev 384.130-0ubuntu0 amd64 NVIDIA binary Xorg driver development files un nvidia-common <none> <none> (no description available) ii nvidia-container-runti 2.0.0+docker18.0 amd64 NVIDIA container runtime ii nvidia-container-runti 1.4.0-1 amd64 NVIDIA container runtime hook un nvidia-docker <none> <none> (no description available) ii nvidia-docker2 2.0.3+docker18.0 all nvidia-docker CLI wrapper un nvidia-driver-binary <none> <none> (no description available) un nvidia-legacy-340xx-vd <none> <none> (no description available) un nvidia-libopencl1-384 <none> <none> (no description available) un nvidia-libopencl1-dev <none> <none> (no description available) ii nvidia-modprobe 384.81-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device fi un nvidia-opencl-icd <none> <none> (no description available) ii nvidia-opencl-icd-384 384.130-0ubuntu0 amd64 NVIDIA OpenCL ICD un nvidia-persistenced <none> <none> (no description available) ii nvidia-prime 0.8.2 amd64 Tools to enable NVIDIA's Prime ii nvidia-settings 384.81-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver un nvidia-settings-binary <none> <none> (no description available) un nvidia-smi <none> <none> (no description available) un nvidia-vdpau-driver <none> <none> (no description available) dpkg-query: no packages found matching *nvidia*rpm dpkg-query: no packages found matching -qa ``` NVIDIA container library version from `nvidia-container-cli -V`: ``` version: 1.0.0 build date: 2018-09-20T20:18+00:00 build revision: 881c88e2e5bb682c9bb14e68bd165cfb64563bb1 build compiler: gcc-5 5.4.0 20160609 build platform: x86_64 build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections ``` Nvidia-docker version from `nvidia-docker --version`: ``` Docker version 18.06.1-ce, build e68fc7a ``` The [debugging logs](https://github.com/NVIDIA/nvidia-docker/wiki/Troubleshooting#generating-debugging-logs) from nvidia-container-runtime do not contain relevant information.

Thanks.

system · January 11, 2023, 5:56am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.