Failed to start results service after running deploy.sh

Hi,

I am trying to install deploy SDK by running deploy.sh. But I always got stucked at starting results service.

Loaded image: clara/argo-controller:0.2.0
2019-09-03 23:27:24 [INFO]: Clara Containers loaded successfully
2019-09-03 23:27:24 [INFO]: Copying the included AI models to directory: /clara/common...
2019-09-03 23:27:24 [INFO]: AI models copied successfully.
2019-09-03 23:27:24 [INFO]: Installing Clara CLI
2019-09-03 23:27:24 [INFO]: Clara CLI installed successfully
2019-09-03 23:27:24 [INFO]: Starting Clara...
 
NAME:   clara
2019-09-03 23:27:41 [INFO]: Starting Result Service...

I’ve checked the status of the results service pod, and I got the following output:

Events:
  Type     Reason             Age                   From                         Message
  ----     ------             ----                  ----                         -------
  Normal   Scheduled          2m23s                 default-scheduler            Successfully assigned default/clara-results-service-58d7cdc45-cvv9j to haonan-xps-15-9560
  Normal   Pulling            103s (x3 over 2m22s)  kubelet, haonan-xps-15-9560  Pulling image "clara/resultsservice:0.2.0"
  Warning  Failed             103s (x3 over 2m22s)  kubelet, haonan-xps-15-9560  Failed to pull image "clara/resultsservice:0.2.0": rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed             103s (x3 over 2m22s)  kubelet, haonan-xps-15-9560  Error: ErrImagePull
  Warning  MissingClusterDNS  79s (x8 over 2m22s)   kubelet, haonan-xps-15-9560  pod: "clara-results-service-58d7cdc45-cvv9j_default(63a34d02-8bd6-4857-a3ee-682cc9eab537)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  Normal   BackOff            79s (x4 over 2m21s)   kubelet, haonan-xps-15-9560  Back-off pulling image "clara/resultsservice:0.2.0"
  Warning  Failed             79s (x4 over 2m21s)   kubelet, haonan-xps-15-9560  Error: ImagePullBackOff

This is the output I got from df -h

haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  2.1M  1.6G   1% /run
/dev/nvme0n1p5  201G   52G  140G  27% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop0       10M   10M     0 100% /snap/helm/124
/dev/loop1      1.0M  1.0M     0 100% /snap/gnome-logs/61
/dev/loop2      150M  150M     0 100% /snap/gnome-3-28-1804/71
/dev/loop3      184M  184M     0 100% /snap/microk8s/743
/dev/loop5       89M   89M     0 100% /snap/core/7396
/dev/loop6      150M  150M     0 100% /snap/gnome-3-28-1804/67
/dev/loop7      4.2M  4.2M     0 100% /snap/gnome-calculator/406
/dev/loop4       55M   55M     0 100% /snap/core18/1098
/dev/loop8       11M   11M     0 100% /snap/kubectl/1189
/dev/loop10      15M   15M     0 100% /snap/gnome-characters/296
/dev/loop9       89M   89M     0 100% /snap/core/7270
/dev/loop12      43M   43M     0 100% /snap/gtk-common-themes/1313
/dev/loop11      15M   15M     0 100% /snap/gnome-characters/317
/dev/loop13     3.8M  3.8M     0 100% /snap/gnome-system-monitor/100
/dev/loop14      55M   55M     0 100% /snap/core18/1074
/dev/nvme0n1p2   96M   31M   66M  33% /boot/efi
tmpfs           1.6G   36K  1.6G   1% /run/user/1000
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b26f7594d378afdf75f2116c0205a3ecf756e96bc9570b9277d0e11654face2e/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/b26f7594d378afdf75f2116c0205a3ecf756e96bc9570b9277d0e11654face2e/rootfs
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/076685ef50314982d6f220722d2192c20dfeb37892fc2364ab283f59c652657b/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/9a77f11cc42857552520f183b02ce99f81dea040931d97363a5ce14851e28588/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/9a77f11cc42857552520f183b02ce99f81dea040931d97363a5ce14851e28588/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/7419e302f001ec704e5912ee7ad5f811c5f2f109070d460c46f78bbad4086fc2/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/7419e302f001ec704e5912ee7ad5f811c5f2f109070d460c46f78bbad4086fc2/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/85c3f43dc97e241f65c894e4d5a4098779ae33fd54e3d1f6454acab11e51a382/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/85c3f43dc97e241f65c894e4d5a4098779ae33fd54e3d1f6454acab11e51a382/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/fe24de69e988a7facecd4f5db8c7bae87b407f8a341921d10dc3c99dfdcdd5e6/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/fe24de69e988a7facecd4f5db8c7bae87b407f8a341921d10dc3c99dfdcdd5e6/rootfs

And after I ran docker images:

haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
clara/ai-livertumor          0.2.0               7562ad4da7f5        2 months ago        1.62GB
clara/app_base_inference     0.2.0               d3acb028127e        2 months ago        1.62GB
clara/clara-dashboard        0.2.0               f8fb31aec950        2 months ago        293MB
clara/clara-datasetservice   0.2.0               db79de506bad        2 months ago        314MB
clara/renderserver_ng        0.2.0               6d2b1ff7fa61        2 months ago        149MB
clara/dicom-writer           0.2.0               603830ba77cb        2 months ago        525MB
clara/ai-vnet                0.2.0               42c1a1db77e8        2 months ago        668MB
clara/dicom-reader           0.2.0               a2df0db91524        2 months ago        525MB
clara/recon-operator         0.2.0               4d17471bd145        2 months ago        620MB
clara/dicomadapter           0.2.0               e58e4ec8a8fc        2 months ago        226MB
clara/platformapiserver      0.2.0               2ee93cbfdbc3        2 months ago        156MB
clara/resultsservice         0.2.0               843eb1163421        2 months ago        206MB
clara/controller             0.2.0               c43ca37ac1d4        2 months ago        99.1MB
clara/register-results       0.2.0               f8b4825c27f8        2 months ago        132MB
clara/argoui                 0.2.0               069f1955f3fc        4 months ago        184MB
clara/trtis                  0.2.0               80a21438f599        6 months ago        4.42GB
clara/argoexec               0.2.0               0857e3f49e65        10 months ago       328MB
clara/argo-controller        0.2.0               abcb0c0ba87c        10 months ago       140MB

Does anybody know how to solve the problem? By the way, I still have 70% disk space so I think the kubelet garbage collection might not be the issue. Thanks!

Issue resolved here:

https://devtalk.nvidia.com/default/topic/1058401/clara-deploy-sdk-new-/get-stuck-while-running-deploy-sh/post/5381648/#5381648