Skip to content
This repository has been archived by the owner on Jan 22, 2024. It is now read-only.

ERRO[0004] error waiting for container: context canceled #1379

Closed
5 tasks
lshiwjx opened this issue Sep 2, 2020 · 1 comment
Closed
5 tasks

ERRO[0004] error waiting for container: context canceled #1379

lshiwjx opened this issue Sep 2, 2020 · 1 comment

Comments

@lshiwjx
Copy link

lshiwjx commented Sep 2, 2020

1. Issue or feature description

docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused "process_linux.go:432: running prestart hook 0 caused \"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: ldcache error: process /sbin/ldconfig failed with error code: 1\\n\""": unknown.
ERRO[0004] error waiting for container: context canceled

2. Steps to reproduce the issue

docker run --gpus all nvidia/cuda:10.0-base nvidia-smi

3. Information to attach (optional if deemed irrelevant)

  • Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0902 07:34:40.343972 37421 nvc.c:282] initializing library context (version=1.2.0, build=d22237acaea94aa5ad5de70aac903534ed598819)
I0902 07:34:40.344395 37421 nvc.c:256] using root /
I0902 07:34:40.344429 37421 nvc.c:257] using ldcache /etc/ld.so.cache
I0902 07:34:40.344448 37421 nvc.c:258] using unprivileged user 1002:1002
I0902 07:34:40.344680 37421 nvc.c:299] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0902 07:34:40.346301 37421 nvc.c:301] dxcore initialization failed, continuing assuming a non-WSL environment
W0902 07:34:40.358114 37422 nvc.c:187] failed to set inheritable capabilities
W0902 07:34:40.358277 37422 nvc.c:188] skipping kernel modules load due to failure
I0902 07:34:40.358746 37423 driver.c:101] starting driver service
I0902 07:34:44.885543 37421 nvc_info.c:679] requesting driver information with ''
I0902 07:34:44.888641 37421 nvc_info.c:168] selecting /usr/lib64/vdpau/libvdpau_nvidia.so.430.64
I0902 07:34:44.889105 37421 nvc_info.c:168] selecting /usr/lib64/libnvoptix.so.430.64
I0902 07:34:44.889207 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-tls.so.430.64
I0902 07:34:44.889269 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-rtcore.so.430.64
I0902 07:34:44.889337 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-ptxjitcompiler.so.430.64
I0902 07:34:44.889423 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-opticalflow.so.430.64
I0902 07:34:44.889507 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-opencl.so.430.64
I0902 07:34:44.889563 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-ml.so.430.64
I0902 07:34:44.889644 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-ifr.so.430.64
I0902 07:34:44.889727 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-glvkspirv.so.430.64
I0902 07:34:44.889785 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-glsi.so.430.64
I0902 07:34:44.889840 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-glcore.so.430.64
I0902 07:34:44.889902 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-fbc.so.430.64
I0902 07:34:44.889995 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-fatbinaryloader.so.430.64
I0902 07:34:44.890053 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-encode.so.430.64
I0902 07:34:44.890134 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-eglcore.so.430.64
I0902 07:34:44.890196 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-compiler.so.430.64
I0902 07:34:44.890255 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-cfg.so.430.64
I0902 07:34:44.890333 37421 nvc_info.c:168] selecting /usr/lib64/libnvidia-cbl.so.430.64
I0902 07:34:44.890391 37421 nvc_info.c:168] selecting /usr/lib64/libnvcuvid.so.430.64
I0902 07:34:44.891054 37421 nvc_info.c:168] selecting /usr/lib64/libcuda.so.430.64
I0902 07:34:44.891503 37421 nvc_info.c:168] selecting /usr/lib64/libGLX_nvidia.so.430.64
I0902 07:34:44.891567 37421 nvc_info.c:168] selecting /usr/lib64/libGLESv2_nvidia.so.430.64
I0902 07:34:44.891625 37421 nvc_info.c:168] selecting /usr/lib64/libGLESv1_CM_nvidia.so.430.64
I0902 07:34:44.891690 37421 nvc_info.c:168] selecting /usr/lib64/libEGL_nvidia.so.430.64
I0902 07:34:44.892186 37421 nvc_info.c:168] selecting /usr/lib/vdpau/libvdpau_nvidia.so.430.64
I0902 07:34:44.892489 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-tls.so.430.64
I0902 07:34:44.892556 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-ptxjitcompiler.so.430.64
I0902 07:34:44.892643 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-opticalflow.so.430.64
I0902 07:34:44.892724 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-opencl.so.430.64
I0902 07:34:44.892781 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-ml.so.430.64
I0902 07:34:44.892863 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-ifr.so.430.64
I0902 07:34:44.892950 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-glvkspirv.so.430.64
I0902 07:34:44.893007 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-glsi.so.430.64
I0902 07:34:44.893060 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-glcore.so.430.64
I0902 07:34:44.893118 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-fbc.so.430.64
I0902 07:34:44.893194 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-fatbinaryloader.so.430.64
I0902 07:34:44.893247 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-encode.so.430.64
I0902 07:34:44.893324 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-eglcore.so.430.64
I0902 07:34:44.893377 37421 nvc_info.c:168] selecting /usr/lib/libnvidia-compiler.so.430.64
I0902 07:34:44.893434 37421 nvc_info.c:168] selecting /usr/lib/libnvcuvid.so.430.64
I0902 07:34:44.893550 37421 nvc_info.c:168] selecting /usr/lib/libcuda.so.430.64
I0902 07:34:44.893664 37421 nvc_info.c:168] selecting /usr/lib/libGLX_nvidia.so.430.64
I0902 07:34:44.893721 37421 nvc_info.c:168] selecting /usr/lib/libGLESv2_nvidia.so.430.64
I0902 07:34:44.893777 37421 nvc_info.c:168] selecting /usr/lib/libGLESv1_CM_nvidia.so.430.64
I0902 07:34:44.893832 37421 nvc_info.c:168] selecting /usr/lib/libEGL_nvidia.so.430.64
W0902 07:34:44.893861 37421 nvc_info.c:349] missing library libnvidia-allocator.so
W0902 07:34:44.893872 37421 nvc_info.c:349] missing library libnvidia-ngx.so
W0902 07:34:44.893881 37421 nvc_info.c:353] missing compat32 library libnvidia-cfg.so
W0902 07:34:44.893890 37421 nvc_info.c:353] missing compat32 library libnvidia-allocator.so
W0902 07:34:44.893899 37421 nvc_info.c:353] missing compat32 library libnvidia-ngx.so
W0902 07:34:44.893908 37421 nvc_info.c:353] missing compat32 library libnvidia-rtcore.so
W0902 07:34:44.893916 37421 nvc_info.c:353] missing compat32 library libnvoptix.so
W0902 07:34:44.893925 37421 nvc_info.c:353] missing compat32 library libnvidia-cbl.so
I0902 07:34:44.896151 37421 nvc_info.c:275] selecting /usr/bin/nvidia-smi
I0902 07:34:44.896218 37421 nvc_info.c:275] selecting /usr/bin/nvidia-debugdump
I0902 07:34:44.896271 37421 nvc_info.c:275] selecting /usr/bin/nvidia-persistenced
I0902 07:34:44.896316 37421 nvc_info.c:275] selecting /usr/bin/nvidia-cuda-mps-control
I0902 07:34:44.896362 37421 nvc_info.c:275] selecting /usr/bin/nvidia-cuda-mps-server
I0902 07:34:44.896844 37421 nvc_info.c:437] listing device /dev/nvidiactl
I0902 07:34:44.896868 37421 nvc_info.c:437] listing device /dev/nvidia-uvm
I0902 07:34:44.896884 37421 nvc_info.c:437] listing device /dev/nvidia-uvm-tools
I0902 07:34:44.896902 37421 nvc_info.c:437] listing device /dev/nvidia-modeset
W0902 07:34:44.897756 37421 nvc_info.c:320] missing ipc /var/run/nvidia-persistenced/socket
W0902 07:34:44.898123 37421 nvc_info.c:320] missing ipc /tmp/nvidia-mps
I0902 07:34:44.898172 37421 nvc_info.c:744] requesting device information with ''
I0902 07:34:44.908343 37421 nvc_info.c:627] listing device /dev/nvidia0 (GPU-528e1551-5586-83e2-6aba-8e20def6df84 at 00000000:04:00.0)
I0902 07:34:44.918682 37421 nvc_info.c:627] listing device /dev/nvidia1 (GPU-3ce09e07-9691-a098-c3c5-38601d077e35 at 00000000:06:00.0)
I0902 07:34:44.928814 37421 nvc_info.c:627] listing device /dev/nvidia2 (GPU-30e41e89-f7f2-466e-8945-55e1ba28ddf0 at 00000000:07:00.0)
I0902 07:34:44.943143 37421 nvc_info.c:627] listing device /dev/nvidia3 (GPU-670b353d-735f-137a-1516-768debde93ce at 00000000:08:00.0)
I0902 07:34:44.957814 37421 nvc_info.c:627] listing device /dev/nvidia4 (GPU-1d7a7faa-98f9-f945-b540-6354b49df954 at 00000000:0c:00.0)
I0902 07:34:44.968733 37421 nvc_info.c:627] listing device /dev/nvidia5 (GPU-ac450966-dd6b-4b25-c163-6377b27c73f3 at 00000000:0d:00.0)
I0902 07:34:44.983736 37421 nvc_info.c:627] listing device /dev/nvidia6 (GPU-a05ccb67-47fc-bc5e-0741-7432e7688117 at 00000000:0e:00.0)
I0902 07:34:44.999721 37421 nvc_info.c:627] listing device /dev/nvidia7 (GPU-c9b0a29e-7512-305b-7dd4-9f01f4b33652 at 00000000:0f:00.0)
NVRM version: 430.64
CUDA version: 10.1

Device Index: 0
Device Minor: 0
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-528e1551-5586-83e2-6aba-8e20def6df84
Bus Location: 00000000:04:00.0
Architecture: 6.1

Device Index: 1
Device Minor: 1
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-3ce09e07-9691-a098-c3c5-38601d077e35
Bus Location: 00000000:06:00.0
Architecture: 6.1

Device Index: 2
Device Minor: 2
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-30e41e89-f7f2-466e-8945-55e1ba28ddf0
Bus Location: 00000000:07:00.0
Architecture: 6.1

Device Index: 3
Device Minor: 3
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-670b353d-735f-137a-1516-768debde93ce
Bus Location: 00000000:08:00.0
Architecture: 6.1

Device Index: 4
Device Minor: 4
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-1d7a7faa-98f9-f945-b540-6354b49df954
Bus Location: 00000000:0c:00.0
Architecture: 6.1

Device Index: 5
Device Minor: 5
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-ac450966-dd6b-4b25-c163-6377b27c73f3
Bus Location: 00000000:0d:00.0
Architecture: 6.1

Device Index: 6
Device Minor: 6
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-a05ccb67-47fc-bc5e-0741-7432e7688117
Bus Location: 00000000:0e:00.0
Architecture: 6.1

Device Index: 7
Device Minor: 7
Model: TITAN Xp
Brand: GeForce
GPU UUID: GPU-c9b0a29e-7512-305b-7dd4-9f01f4b33652
Bus Location: 00000000:0f:00.0
Architecture: 6.1
I0902 07:34:45.000302 37421 nvc.c:337] shutting down library context
I0902 07:34:45.009564 37423 driver.c:156] terminating driver service
I0902 07:34:45.010138 37421 driver.c:196] driver service terminated successfully

  • Kernel version from uname -a

Linux GPU 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • Docker version from docker version

Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:46:54 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:28 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

  • NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'

nvidia-container-toolkit-1.2.1-2.x86_64
libnvidia-container1-1.2.0-1.x86_64
libnvidia-container-tools-1.2.0-1.x86_64

  • NVIDIA container library version from nvidia-container-cli -V

version: 1.2.0
build date: 2020-07-08T19:33+0000
build revision: d22237acaea94aa5ad5de70aac903534ed598819
build compiler: gcc 4.8.5 20150623 (Red Hat 4.8.5-39)
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
@klueska
Copy link
Contributor

klueska commented Sep 2, 2020

The actual underlying error from your output is:

ldcache error: process /sbin/ldconfig failed with error code: 1

Which seems related to:
#552

@elezar elezar closed this as completed Oct 30, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@klueska @elezar @lshiwjx