Get stuck while running deploy.sh

Healthcare Clara Deploy SDK
#1

Hi,

I am now trying to install deploy sdk on my Ubuntu. I first run install-prereqs.sh and it works fine with the final output “2019-07-31 13:43:05 [INFO]: Clara Deploy SDK Prerequisites installed successfully!”

However, when I continue running the deploy.sh, it is stuck like this for hours:

2019-07-31 13:45:51 [INFO]: Clara Containers loaded successfully
2019-07-31 13:45:51 [INFO]: Copying the included AI models to directory: /clara/common...
2019-07-31 13:45:52 [INFO]: AI models copied successfully.
2019-07-31 13:45:52 [INFO]: Installing Clara CLI
2019-07-31 13:45:53 [INFO]: Clara CLI installed successfully
2019-07-31 13:45:53 [INFO]: Starting Clara...
 
NAME:   clara
2019-07-31 13:46:10 [INFO]: Starting Result Service...

Has anyone met the same situation? And how did you solve this? Thanks!

#2

Hi
Thanks for your interest in Clara Deploy.
This is interesting as it looks like every thing is fine until Clara is starting the other services. To help you further could you run

kubectl get pods
kubectls get svc

Also check you have a ./clara folder in your home and have the correct permissions

Also try manually starting the services by following https://docs.nvidia.com/clara/deploy/ClaraAdministration.html#start-stop-restart-the-platform

clara dicom stop
clara results stop
clara stop

wait till kubectl get pods gives no pods
then run

clara start
clara results start
clara dicom start

Hope this helps.

#3

Stop and re-start does not work. Actually, if I run

helm ls --all

, I get

E0731 16:20:39.222009    5308 portforward.go:385] error copying from local connection to remote stream: read tcp4 127.0.0.1:37109->127.0.0.1:60162: read: connection reset by peer
NAME                 	REVISION	UPDATED                 	STATUS         	CHART                	APP VERSION	NAMESPACE
clara                	1       	Wed Jul 31 16:06:23 2019	DEPLOYED       	clara-0.2.0          	1.0        	default  
clara-results-service	1       	Wed Jul 31 16:06:31 2019	PENDING_INSTALL	results-service-0.2.0	1.0        	default

Seems like the “PENDING INSTALL” status is the reason, but I don’t know how to fix it. I tried to uninstall the prereqs and re-install them. But it was still stuck at that point.

#4

Hi,

Can you do a kubectl get pods?
You will find a pod with the prefix clara-results-service in the name.
Can you please check the status of the pod using

kubectl describe pods/clara-results-service-<suffix>

and check the logs using

kubectl logs pods/clara-results-service-<suffix>

Please post the result of the commands here and I would be happy to look further into the issue.

#5

Thanks for your reply. Think I have found the reason of causing this problem.
The free space on the disk must be bigger than 30GB, or the docker would automatically kick the loaded images out or evicted the tiller pod during the installation. So when I ran “clara results start” command, the “clara-resultsservice” image was no longer loaded.

After cleaning my disk and re-install the toolkit, everything starts working.

Thanks again for your help!

#6

Hi,

You can find documentation related to the issue here :

https://docs.nvidia.com/clara/deploy/ReleaseNotes.html#installation-error-due-insufficient-disk-space-to-deploy-clara-container-images

#7

Hi,

Yes, this is what I found after doing several days research. CRY…
It would save me a lot of time if I was able to notice this earlier…
Thanks for your reply!

#8

Hi,

I’ve ran into the same error with you. I also get stucked at starting result service part. But my disk space is sufficient, still got 150GB left. Do you know how I can solve the problem? Thanks!

#9

Hi,

Can you please run the command

df -h

on your system. I would like to have a look at the disk space for the different mounts. Also, can you please please print the output of

docker images | grep clara
#10

Hi pnahar,

Thanks for your reply! This is the output I got from df -h

haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.8G     0  7.8G   0% /dev
tmpfs           1.6G  2.1M  1.6G   1% /run
/dev/nvme0n1p5  201G   52G  140G  27% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/loop0       10M   10M     0 100% /snap/helm/124
/dev/loop1      1.0M  1.0M     0 100% /snap/gnome-logs/61
/dev/loop2      150M  150M     0 100% /snap/gnome-3-28-1804/71
/dev/loop3      184M  184M     0 100% /snap/microk8s/743
/dev/loop5       89M   89M     0 100% /snap/core/7396
/dev/loop6      150M  150M     0 100% /snap/gnome-3-28-1804/67
/dev/loop7      4.2M  4.2M     0 100% /snap/gnome-calculator/406
/dev/loop4       55M   55M     0 100% /snap/core18/1098
/dev/loop8       11M   11M     0 100% /snap/kubectl/1189
/dev/loop10      15M   15M     0 100% /snap/gnome-characters/296
/dev/loop9       89M   89M     0 100% /snap/core/7270
/dev/loop12      43M   43M     0 100% /snap/gtk-common-themes/1313
/dev/loop11      15M   15M     0 100% /snap/gnome-characters/317
/dev/loop13     3.8M  3.8M     0 100% /snap/gnome-system-monitor/100
/dev/loop14      55M   55M     0 100% /snap/core18/1074
/dev/nvme0n1p2   96M   31M   66M  33% /boot/efi
tmpfs           1.6G   36K  1.6G   1% /run/user/1000
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b26f7594d378afdf75f2116c0205a3ecf756e96bc9570b9277d0e11654face2e/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/b26f7594d378afdf75f2116c0205a3ecf756e96bc9570b9277d0e11654face2e/rootfs
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/076685ef50314982d6f220722d2192c20dfeb37892fc2364ab283f59c652657b/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/9a77f11cc42857552520f183b02ce99f81dea040931d97363a5ce14851e28588/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/9a77f11cc42857552520f183b02ce99f81dea040931d97363a5ce14851e28588/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/7419e302f001ec704e5912ee7ad5f811c5f2f109070d460c46f78bbad4086fc2/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/7419e302f001ec704e5912ee7ad5f811c5f2f109070d460c46f78bbad4086fc2/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/85c3f43dc97e241f65c894e4d5a4098779ae33fd54e3d1f6454acab11e51a382/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/85c3f43dc97e241f65c894e4d5a4098779ae33fd54e3d1f6454acab11e51a382/rootfs
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/fe24de69e988a7facecd4f5db8c7bae87b407f8a341921d10dc3c99dfdcdd5e6/shm
overlay         201G   52G  140G  27% /var/snap/microk8s/common/run/containerd/io.containerd.runtime.v1.linux/k8s.io/fe24de69e988a7facecd4f5db8c7bae87b407f8a341921d10dc3c99dfdcdd5e6/rootfs

And after I ran docker images:

haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ docker images
REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
clara/ai-livertumor          0.2.0               7562ad4da7f5        2 months ago        1.62GB
clara/app_base_inference     0.2.0               d3acb028127e        2 months ago        1.62GB
clara/clara-dashboard        0.2.0               f8fb31aec950        2 months ago        293MB
clara/clara-datasetservice   0.2.0               db79de506bad        2 months ago        314MB
clara/renderserver_ng        0.2.0               6d2b1ff7fa61        2 months ago        149MB
clara/dicom-writer           0.2.0               603830ba77cb        2 months ago        525MB
clara/ai-vnet                0.2.0               42c1a1db77e8        2 months ago        668MB
clara/dicom-reader           0.2.0               a2df0db91524        2 months ago        525MB
clara/recon-operator         0.2.0               4d17471bd145        2 months ago        620MB
clara/dicomadapter           0.2.0               e58e4ec8a8fc        2 months ago        226MB
clara/platformapiserver      0.2.0               2ee93cbfdbc3        2 months ago        156MB
clara/resultsservice         0.2.0               843eb1163421        2 months ago        206MB
clara/controller             0.2.0               c43ca37ac1d4        2 months ago        99.1MB
clara/register-results       0.2.0               f8b4825c27f8        2 months ago        132MB
clara/argoui                 0.2.0               069f1955f3fc        4 months ago        184MB
clara/trtis                  0.2.0               80a21438f599        6 months ago        4.42GB
clara/argoexec               0.2.0               0857e3f49e65        10 months ago       328MB
clara/argo-controller        0.2.0               abcb0c0ba87c        10 months ago       140MB
#11

Do you get this error after running deploy.sh?

rpc error: code = Unknown desc = release clara-results-service failed: etcdserver: request timed out

Or do you just get blocked after running the deploy script?

#12

I just got blocked. No errors.

Loaded image: clara/argo-controller:0.2.0
2019-09-03 23:27:24 [INFO]: Clara Containers loaded successfully
2019-09-03 23:27:24 [INFO]: Copying the included AI models to directory: /clara/common...
2019-09-03 23:27:24 [INFO]: AI models copied successfully.
2019-09-03 23:27:24 [INFO]: Installing Clara CLI
2019-09-03 23:27:24 [INFO]: Clara CLI installed successfully
2019-09-03 23:27:24 [INFO]: Starting Clara...
 
NAME:   clara
2019-09-03 23:27:41 [INFO]: Starting Result Service...

But after I ran kubectl describe pods, I can see the error in the pod.

Events:
  Type     Reason             Age                   From                         Message
  ----     ------             ----                  ----                         -------
  Normal   Scheduled          2m23s                 default-scheduler            Successfully assigned default/clara-results-service-58d7cdc45-cvv9j to haonan-xps-15-9560
  Normal   Pulling            103s (x3 over 2m22s)  kubelet, haonan-xps-15-9560  Pulling image "clara/resultsservice:0.2.0"
  Warning  Failed             103s (x3 over 2m22s)  kubelet, haonan-xps-15-9560  Failed to pull image "clara/resultsservice:0.2.0": rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed             103s (x3 over 2m22s)  kubelet, haonan-xps-15-9560  Error: ErrImagePull
  Warning  MissingClusterDNS  79s (x8 over 2m22s)   kubelet, haonan-xps-15-9560  pod: "clara-results-service-58d7cdc45-cvv9j_default(63a34d02-8bd6-4857-a3ee-682cc9eab537)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
  Normal   BackOff            79s (x4 over 2m21s)   kubelet, haonan-xps-15-9560  Back-off pulling image "clara/resultsservice:0.2.0"
  Warning  Failed             79s (x4 over 2m21s)   kubelet, haonan-xps-15-9560  Error: ImagePullBackOff
#13

Hi haonanz,

What kind of kubernetes setup are you using? Did you install kubernetes using install-prereqs.sh or are you using something like microk8s?

#14

Hi pnahar,

I ran install-prereqs.sh to install all the dependencies.

#15

Just delete my comment. Thought you may fail to load resultservice docker image. Later I saw it in your earlier posts…

#16

Hi haonanz,

You seem to have the correct results service image of size 206 MB. Can you post the output of these commands, so that I can have a look at your kubernetes setup?

# Check status
printenv | grep DOCKER
sudo kubectl config get-contexts
snap list | grep microk8s -C 2
sudo docker version
journalctl -xe | grep "PullImage" -C 5
#17

Hi haohis66,

Thanks for your reply! I think I unzipped both s1 and s2 since all the images are loaded properly. I just cannot figure out why kube cannot pull the loaded images.

#18

Yes, you do have all images loaded correctly. I verified that the results service image size should be 206MB.

#19

Hi pnahar,

Thanks for the reply! Followings are the output after I ran the commands.

  1. printenv | grep DOCKER

output is empty

  1. sudo kubectl config get-contexts
haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ sudo kubectl config get-contexts
CURRENT   NAME   CLUSTER   AUTHINFO   NAMESPACE
  1. snap list | grep microk8s -C 2
haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ snap list | grep microk8s -C 2
helm                  2.14.3                      124   stable    snapcrafters  classic
kubectl               1.15.3                      1189  stable    canonical*    classic
microk8s              v1.15.2                     743   stable    canonical*    classic
  1. sudo docker version
haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ sudo docker version
Client: Docker Engine - Community
 Version:           19.03.2
 API version:       1.40
 Go version:        go1.12.8
 Git commit:        6a30dfc
 Built:             Thu Aug 29 05:29:11 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.2
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.8
  Git commit:       6a30dfc
  Built:            Thu Aug 29 05:27:45 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 nvidia:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
  1. journalctl -xe | grep “PullImage” -C 5
haonan@haonan-XPS-15-9560:~/Downloads/ClaraSDK/scripts$ journalctl -xe | grep "PullImage" -C 5
-- Logs begin at Tue 2019-08-27 07:51:42 PDT, end at Fri 2019-09-06 14:29:36 PDT. --
Sep 06 14:16:57 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:16:57.709264430-07:00" level=info msg="PullImage "clara/argoui:0.2.0""
Sep 06 14:16:57 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:57.708620    4070 remote_image.go:113] PullImage "clara/argo-controller:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:16:57 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:57.708737    4070 kuberuntime_image.go:51] Pull image "clara/argo-controller:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:16:57 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:57.708986    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:16:57 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:57.709097    4070 pod_workers.go:190] Error syncing pod e2bc13cd-3f2e-477d-afed-66617af06f33 ("clara-workflow-controller-796b77b8d-bhfxz_default(e2bc13cd-3f2e-477d-afed-66617af06f33)"), skipping: failed to "StartContainer" for "controller" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/argo-controller:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:16:57 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 06 14:16:57 haonan-XPS-15-9560 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 66.
--
-- 
-- The start-up result is RESULT.
Sep 06 14:16:57 haonan-XPS-15-9560 kubelet[13834]: F0906 14:16:57.922972   13834 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Sep 06 14:16:57 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:16:57 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:16:58 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:16:58.264748211-07:00" level=error msg="PullImage "clara/argoui:0.2.0" failed" error="failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:16:58 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:58.265111    4070 remote_image.go:113] PullImage "clara/argoui:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:16:58 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:58.265191    4070 kuberuntime_image.go:51] Pull image "clara/argoui:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:16:58 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:58.265324    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:16:58 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:16:58.265421    4070 pod_workers.go:190] Error syncing pod 20eebe37-f431-4f68-80e4-ba33c3cf6aae ("clara-ui-58b97bc76b-pglz2_default(20eebe37-f431-4f68-80e4-ba33c3cf6aae)"), skipping: failed to "StartContainer" for "ui" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/argoui:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:17:01 haonan-XPS-15-9560 CRON[13891]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 06 14:17:01 haonan-XPS-15-9560 CRON[13898]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 06 14:17:01 haonan-XPS-15-9560 CRON[13891]: pam_unix(cron:session): session closed for user root
Sep 06 14:17:07 haonan-XPS-15-9560 systemd-timesyncd[644]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Sep 06 14:17:07 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:17:07.606638287-07:00" level=info msg="PullImage "clara/resultsservice:0.2.0""
Sep 06 14:17:08 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 06 14:17:08 haonan-XPS-15-9560 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 67.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Support: http://www.ubuntu.com/support
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is RESULT.
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:17:08.176515877-07:00" level=error msg="PullImage "clara/resultsservice:0.2.0" failed" error="failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.176689    4070 remote_image.go:113] PullImage "clara/resultsservice:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.176735    4070 kuberuntime_image.go:51] Pull image "clara/resultsservice:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.176911    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.176949    4070 pod_workers.go:190] Error syncing pod 060bcc1a-d357-44bf-ab83-5fddb9a8c80a ("clara-results-service-58d7cdc45-mkc2j_default(060bcc1a-d357-44bf-ab83-5fddb9a8c80a)"), skipping: failed to "StartContainer" for "results-service" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/resultsservice:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:17:08.176969199-07:00" level=info msg="PullImage "clara/platformapiserver:0.2.0""
Sep 06 14:17:08 haonan-XPS-15-9560 kubelet[13969]: F0906 14:17:08.193147   13969 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Sep 06 14:17:08 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:17:08 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.606382    4070 pod_workers.go:190] Error syncing pod 20eebe37-f431-4f68-80e4-ba33c3cf6aae ("clara-ui-58b97bc76b-pglz2_default(20eebe37-f431-4f68-80e4-ba33c3cf6aae)"), skipping: failed to "StartContainer" for "ui" with ImagePullBackOff: "Back-off pulling image \"clara/argoui:0.2.0\""
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:17:08.759572881-07:00" level=error msg="PullImage "clara/platformapiserver:0.2.0" failed" error="failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.759839    4070 remote_image.go:113] PullImage "clara/platformapiserver:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.759899    4070 kuberuntime_image.go:51] Pull image "clara/platformapiserver:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.760001    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:17:08 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:08.760057    4070 pod_workers.go:190] Error syncing pod 7fffe677-d713-4b94-a059-88309e26a417 ("clara-clara-platformapiserver-59dfbcd6b6-rcvtk_default(7fffe677-d713-4b94-a059-88309e26a417)"), skipping: failed to "StartContainer" for "platformapiserver" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/platformapiserver:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:17:09 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:17:09.605795    4070 pod_workers.go:190] Error syncing pod e2bc13cd-3f2e-477d-afed-66617af06f33 ("clara-workflow-controller-796b77b8d-bhfxz_default(e2bc13cd-3f2e-477d-afed-66617af06f33)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image \"clara/argo-controller:0.2.0\""
Sep 06 14:17:13 haonan-XPS-15-9560 kernel: pcieport 0000:00:1d.0: AER: Corrected error received: 0000:00:1d.0
--
Sep 06 14:21:55 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:21:55 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:21:56 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:21:56.606376    4070 pod_workers.go:190] Error syncing pod e2bc13cd-3f2e-477d-afed-66617af06f33 ("clara-workflow-controller-796b77b8d-bhfxz_default(e2bc13cd-3f2e-477d-afed-66617af06f33)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image \"clara/argo-controller:0.2.0\""
Sep 06 14:22:02 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:02.605859    4070 pod_workers.go:190] Error syncing pod 060bcc1a-d357-44bf-ab83-5fddb9a8c80a ("clara-results-service-58d7cdc45-mkc2j_default(060bcc1a-d357-44bf-ab83-5fddb9a8c80a)"), skipping: failed to "StartContainer" for "results-service" with ImagePullBackOff: "Back-off pulling image \"clara/resultsservice:0.2.0\""
Sep 06 14:22:02 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:02.605884    4070 pod_workers.go:190] Error syncing pod 7fffe677-d713-4b94-a059-88309e26a417 ("clara-clara-platformapiserver-59dfbcd6b6-rcvtk_default(7fffe677-d713-4b94-a059-88309e26a417)"), skipping: failed to "StartContainer" for "platformapiserver" with ImagePullBackOff: "Back-off pulling image \"clara/platformapiserver:0.2.0\""
Sep 06 14:22:03 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:03.606335918-07:00" level=info msg="PullImage "clara/argoui:0.2.0""
Sep 06 14:22:04 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:04.674374892-07:00" level=error msg="PullImage "clara/argoui:0.2.0" failed" error="failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:04 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:04.674842    4070 remote_image.go:113] PullImage "clara/argoui:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:04 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:04.674942    4070 kuberuntime_image.go:51] Pull image "clara/argoui:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:04 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:04.675063    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:04 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:04.675138    4070 pod_workers.go:190] Error syncing pod 20eebe37-f431-4f68-80e4-ba33c3cf6aae ("clara-ui-58b97bc76b-pglz2_default(20eebe37-f431-4f68-80e4-ba33c3cf6aae)"), skipping: failed to "StartContainer" for "ui" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/argoui:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:05 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 06 14:22:05 haonan-XPS-15-9560 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 96.
--
-- 
-- The start-up result is RESULT.
Sep 06 14:22:05 haonan-XPS-15-9560 kubelet[17489]: F0906 14:22:05.431556   17489 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Sep 06 14:22:05 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:22:05 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:22:09 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:09.606523884-07:00" level=info msg="PullImage "clara/argo-controller:0.2.0""
Sep 06 14:22:10 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:10.155148376-07:00" level=error msg="PullImage "clara/argo-controller:0.2.0" failed" error="failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:10 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:10.156824    4070 remote_image.go:113] PullImage "clara/argo-controller:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:10 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:10.156935    4070 kuberuntime_image.go:51] Pull image "clara/argo-controller:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:10 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:10.157092    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:10 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:10.157195    4070 pod_workers.go:190] Error syncing pod e2bc13cd-3f2e-477d-afed-66617af06f33 ("clara-workflow-controller-796b77b8d-bhfxz_default(e2bc13cd-3f2e-477d-afed-66617af06f33)"), skipping: failed to "StartContainer" for "controller" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/argo-controller:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:15 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 06 14:22:15 haonan-XPS-15-9560 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 97.
--
-- 
-- The start-up result is RESULT.
Sep 06 14:22:15 haonan-XPS-15-9560 kubelet[17607]: F0906 14:22:15.508632   17607 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Sep 06 14:22:15 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:22:15 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:22:15 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:15.606365380-07:00" level=info msg="PullImage "clara/platformapiserver:0.2.0""
Sep 06 14:22:16 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:16.138465418-07:00" level=error msg="PullImage "clara/platformapiserver:0.2.0" failed" error="failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:16 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:16.138898    4070 remote_image.go:113] PullImage "clara/platformapiserver:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:16 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:16.139093    4070 kuberuntime_image.go:51] Pull image "clara/platformapiserver:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:16 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:16.139208    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:16 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:16.139276    4070 pod_workers.go:190] Error syncing pod 7fffe677-d713-4b94-a059-88309e26a417 ("clara-clara-platformapiserver-59dfbcd6b6-rcvtk_default(7fffe677-d713-4b94-a059-88309e26a417)"), skipping: failed to "StartContainer" for "platformapiserver" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/platformapiserver:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:16 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:16.606666653-07:00" level=info msg="PullImage "clara/resultsservice:0.2.0""
Sep 06 14:22:17 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:22:17.391359032-07:00" level=error msg="PullImage "clara/resultsservice:0.2.0" failed" error="failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:17 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:17.391858    4070 remote_image.go:113] PullImage "clara/resultsservice:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:17 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:17.391964    4070 kuberuntime_image.go:51] Pull image "clara/resultsservice:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:17 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:17.392119    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:22:17 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:17.392269    4070 pod_workers.go:190] Error syncing pod 060bcc1a-d357-44bf-ab83-5fddb9a8c80a ("clara-results-service-58d7cdc45-mkc2j_default(060bcc1a-d357-44bf-ab83-5fddb9a8c80a)"), skipping: failed to "StartContainer" for "results-service" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/resultsservice:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:22:17 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:22:17.606387    4070 pod_workers.go:190] Error syncing pod 20eebe37-f431-4f68-80e4-ba33c3cf6aae ("clara-ui-58b97bc76b-pglz2_default(20eebe37-f431-4f68-80e4-ba33c3cf6aae)"), skipping: failed to "StartContainer" for "ui" with ImagePullBackOff: "Back-off pulling image \"clara/argoui:0.2.0\""
Sep 06 14:22:24 haonan-XPS-15-9560 systemd[1]: Starting Message of the Day...
--
-- The start-up result is RESULT.
Sep 06 14:27:01 haonan-XPS-15-9560 kubelet[21020]: F0906 14:27:01.921839   21020 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Sep 06 14:27:01 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:27:01 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:27:02 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:02.605835    4070 pod_workers.go:190] Error syncing pod 060bcc1a-d357-44bf-ab83-5fddb9a8c80a ("clara-results-service-58d7cdc45-mkc2j_default(060bcc1a-d357-44bf-ab83-5fddb9a8c80a)"), skipping: failed to "StartContainer" for "results-service" with ImagePullBackOff: "Back-off pulling image \"clara/resultsservice:0.2.0\""
Sep 06 14:27:10 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:10.605849088-07:00" level=info msg="PullImage "clara/argoui:0.2.0""
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:11.605912    4070 pod_workers.go:190] Error syncing pod 7fffe677-d713-4b94-a059-88309e26a417 ("clara-clara-platformapiserver-59dfbcd6b6-rcvtk_default(7fffe677-d713-4b94-a059-88309e26a417)"), skipping: failed to "StartContainer" for "platformapiserver" with ImagePullBackOff: "Back-off pulling image \"clara/platformapiserver:0.2.0\""
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:11.970590519-07:00" level=error msg="PullImage "clara/argoui:0.2.0" failed" error="failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:11.970947    4070 remote_image.go:113] PullImage "clara/argoui:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:11.971068    4070 kuberuntime_image.go:51] Pull image "clara/argoui:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:11.971341    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argoui:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:11.971462    4070 pod_workers.go:190] Error syncing pod 20eebe37-f431-4f68-80e4-ba33c3cf6aae ("clara-ui-58b97bc76b-pglz2_default(20eebe37-f431-4f68-80e4-ba33c3cf6aae)"), skipping: failed to "StartContainer" for "ui" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/argoui:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:11 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:11.971725003-07:00" level=info msg="PullImage "clara/argo-controller:0.2.0""
Sep 06 14:27:12 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 06 14:27:12 haonan-XPS-15-9560 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 126.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- 
-- The start-up result is RESULT.
Sep 06 14:27:12 haonan-XPS-15-9560 kubelet[21153]: F0906 14:27:12.192884   21153 server.go:198] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
Sep 06 14:27:12 haonan-XPS-15-9560 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 06 14:27:12 haonan-XPS-15-9560 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 06 14:27:12 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:12.511862669-07:00" level=error msg="PullImage "clara/argo-controller:0.2.0" failed" error="failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:12 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:12.512342    4070 remote_image.go:113] PullImage "clara/argo-controller:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:12 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:12.512465    4070 kuberuntime_image.go:51] Pull image "clara/argo-controller:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:12 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:12.512630    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/argo-controller:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:12 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:12.512742    4070 pod_workers.go:190] Error syncing pod e2bc13cd-3f2e-477d-afed-66617af06f33 ("clara-workflow-controller-796b77b8d-bhfxz_default(e2bc13cd-3f2e-477d-afed-66617af06f33)"), skipping: failed to "StartContainer" for "controller" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/argo-controller:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:15 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:15.605731    4070 pod_workers.go:190] Error syncing pod 060bcc1a-d357-44bf-ab83-5fddb9a8c80a ("clara-results-service-58d7cdc45-mkc2j_default(060bcc1a-d357-44bf-ab83-5fddb9a8c80a)"), skipping: failed to "StartContainer" for "results-service" with ImagePullBackOff: "Back-off pulling image \"clara/resultsservice:0.2.0\""
Sep 06 14:27:22 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
--
Sep 06 14:27:23 haonan-XPS-15-9560 kernel: pcieport 0000:00:1d.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Transmitter ID)
Sep 06 14:27:23 haonan-XPS-15-9560 kernel: pcieport 0000:00:1d.0:   device [8086:a118] error status/mask=00001000/00002000
Sep 06 14:27:23 haonan-XPS-15-9560 kernel: pcieport 0000:00:1d.0:    [12] Timeout               
Sep 06 14:27:23 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:23.606374    4070 pod_workers.go:190] Error syncing pod 20eebe37-f431-4f68-80e4-ba33c3cf6aae ("clara-ui-58b97bc76b-pglz2_default(20eebe37-f431-4f68-80e4-ba33c3cf6aae)"), skipping: failed to "StartContainer" for "ui" with ImagePullBackOff: "Back-off pulling image \"clara/argoui:0.2.0\""
Sep 06 14:27:23 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:23.606621    4070 pod_workers.go:190] Error syncing pod e2bc13cd-3f2e-477d-afed-66617af06f33 ("clara-workflow-controller-796b77b8d-bhfxz_default(e2bc13cd-3f2e-477d-afed-66617af06f33)"), skipping: failed to "StartContainer" for "controller" with ImagePullBackOff: "Back-off pulling image \"clara/argo-controller:0.2.0\""
Sep 06 14:27:26 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:26.606054888-07:00" level=info msg="PullImage "clara/platformapiserver:0.2.0""
Sep 06 14:27:27 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:27.172501382-07:00" level=error msg="PullImage "clara/platformapiserver:0.2.0" failed" error="failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:27 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:27.172871    4070 remote_image.go:113] PullImage "clara/platformapiserver:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:27 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:27.172986    4070 kuberuntime_image.go:51] Pull image "clara/platformapiserver:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:27 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:27.173129    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/platformapiserver:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:27 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:27.173191    4070 pod_workers.go:190] Error syncing pod 7fffe677-d713-4b94-a059-88309e26a417 ("clara-clara-platformapiserver-59dfbcd6b6-rcvtk_default(7fffe677-d713-4b94-a059-88309e26a417)"), skipping: failed to "StartContainer" for "platformapiserver" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/platformapiserver:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:27 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:27.604750520-07:00" level=info msg="PullImage "clara/resultsservice:0.2.0""
Sep 06 14:27:28 haonan-XPS-15-9560 microk8s.daemon-containerd[3993]: time="2019-09-06T14:27:28.141118946-07:00" level=error msg="PullImage "clara/resultsservice:0.2.0" failed" error="failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:28 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:28.141498    4070 remote_image.go:113] PullImage "clara/resultsservice:0.2.0" from image service failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:28 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:28.141596    4070 kuberuntime_image.go:51] Pull image "clara/resultsservice:0.2.0" failed: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:28 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:28.141755    4070 kuberuntime_manager.go:779] container start failed: ErrImagePull: rpc error: code = Unknown desc = failed to resolve image "docker.io/clara/resultsservice:0.2.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Sep 06 14:27:28 haonan-XPS-15-9560 microk8s.daemon-kubelet[4070]: E0906 14:27:28.141861    4070 pod_workers.go:190] Error syncing pod 060bcc1a-d357-44bf-ab83-5fddb9a8c80a ("clara-results-service-58d7cdc45-mkc2j_default(060bcc1a-d357-44bf-ab83-5fddb9a8c80a)"), skipping: failed to "StartContainer" for "results-service" with ErrImagePull: "rpc error: code = Unknown desc = failed to resolve image \"docker.io/clara/resultsservice:0.2.0\": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"
Sep 06 14:27:32 haonan-XPS-15-9560 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Sep 06 14:27:32 haonan-XPS-15-9560 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 128.
#20

The problem is due to a microk8s installation on your system. If you’re ok with uninstalling microk8s, I can guide you through the steps to redeploy clara.

# to remove microk8s
microk8s.reset
sudo snap remove microk8s

# uninstall pre-requisites
sudo ./uninstall-prereqs.sh -s

# reinstall pre-requisites
sudo ./install-prereqs.sh

# redeploy clara
sudo ./deploy.sh