Jarvis Quick Start: Installation fails

petasisg · June 2, 2021, 5:15pm

Hi all,

I am trying to install Jarvis, but I am getting the following error:

Error occurred downloading 'nvidia/jarvis/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming:1.2.0-beta'. Exiting.

Is there a solution?

Regards,
George

SunilJB · June 3, 2021, 9:06am

Could you please share the complete error log/command output so we can help better? Also please share docker logs jarvis-speech output as well.

Thanks

petasisg · June 3, 2021, 12:23pm

The logs are:

bash jarvis_init.sh 
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Docker version insufficient. Please use Docker 19.03 or later
Logging into NGC docker registry if necessary...
Pulling required docker images if necessary...
Note: This may take some time, depending on the speed of your Internet connection.
> Pulling Jarvis Speech Server images.
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  > Image nvcr.io/nvidia/jarvis/jarvis-speech:1.2.0-beta-server exists. Skipping.
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  > Pulling nvcr.io/nvidia/jarvis/jarvis-speech-client:1.2.0-beta. This may take some time...
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
  > Image nvcr.io/nvidia/jarvis/jarvis-speech:1.2.0-beta-servicemaker exists. Skipping.

Downloading models (JMIRs) from NGC...
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing JMIRs set the location and corresponding flag in config.sh.
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.

==========================
== Jarvis Speech Skills ==
==========================

NVIDIA Release devel (build 22382700)

Copyright (c) 2018-2021, NVIDIA CORPORATION.  All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

/data/artifacts /opt/jarvis
Directory jmir_punctuation_v1.2.0-beta already exists, skipping. Use '--force' option to override.
  > Downloading nvidia/jarvis/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming:1.2.0-beta...
Client Error: 403 Response:  - Request Id:  Url: https://api.ngc.nvidia.com/v2/org/nvidia/team/jarvis/models/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming/versions/1.2.0-beta
  > Attempt 1 out of 3 failed
  > Trying again...
Client Error: 403 Response:  - Request Id:  Url: https://api.ngc.nvidia.com/v2/org/nvidia/team/jarvis/models/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming/versions/1.2.0-beta
  > Attempt 2 out of 3 failed
  > Trying again...
Client Error: 403 Response:  - Request Id:  Url: https://api.ngc.nvidia.com/v2/org/nvidia/team/jarvis/models/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming/versions/1.2.0-beta
  > Attempt 3 out of 3 failed
Error occurred downloading 'nvidia/jarvis/jmir_jarvis_asr_citrinet_1024_asrset1p7_streaming:1.2.0-beta'. Exiting.
Error in downloading models.

There is no container jarvis-speech yet, the procedure fails in the init script.

SunilJB · June 4, 2021, 8:55am

Hi @petasisg ,

It seems to access issue, could you please cross check if NGC key is set correctly.

Will request you upgrade the docker to supported version.
Also, please run jarvis_clean.sh before retrying the deployment. Do let me know if you still face while setting up the service?

Thanks

petasisg · June 4, 2021, 9:14am

Dear @SunilJB,

Yes, ngc key is correct, and if I modify config.sh to not include these citrinet packages, the rest download ok.

The docker warnings are irrelevant, I am using podman in fedora :-).

shashikant.ghangare · June 4, 2021, 10:47am

Facing the same issue. The models are not downloading.
`bash jarvis_init.sh
Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.

Pulling Jarvis Speech Server images.
Pulling nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-server. This may take some time…
Pulling nvcr.io/nvidia/jarvis/jarvis-speech-client:1.1.0-beta. This may take some time…
Pulling nvcr.io/nvidia/jarvis/jarvis-speech:1.1.0-beta-servicemaker. This may take some time…

Downloading models (JMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing JMIRs set the location and corresponding flag in config.sh.

==========================
== Jarvis Speech Skills ==

NVIDIA Release (build 21060478)

Various files include modifications (c) NVIDIA CORPORATION. All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.

NOTE: The SHMEM allocation limit is set to the default of 64MB. This may be
insufficient for the inference server. NVIDIA recommends the use of the following flags:
nvidia-docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 …

/data/artifacts /opt/jarvis

Downloading nvidia/jarvis/jmir_punctuation:1.0.0-b.1…
Downloaded 0 B in 5s, Download speed: 0 B/s
Client Error: 403 Response: - Request Id: Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/models/jmir_punctuation/versions/1.0.0-b.1/files.zip?ak-token=exp=1622813401~acl=%2Fmodels%2Forg%2Fnvidia%2Fteam%2Fjarvis%2Fmodels%2Fjmir_punctuation%2Fversions%2F1.0.0-b.1%2Ffiles.zip*~hmac=1751bcc2767a59c4416de7f32925d65209633e5b3c23cb30a755626c998cdff1
Downloaded 0 B in 6s, Download speed: 0 B/s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-04 12:29:58.020858
Completed at: 2021-06-04 12:30:04.033553
Duration taken: 6s

Attempt 1 out of 3 failed
Trying again…
Downloaded 0 B in 6s, Download speed: 0 B/s
Client Error: 403 Response: - Request Id: Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/models/jmir_punctuation/versions/1.0.0-b.1/files.zip?ak-token=exp=1622813418~acl=%2Fmodels%2Forg%2Fnvidia%2Fteam%2Fjarvis%2Fmodels%2Fjmir_punctuation%2Fversions%2F1.0.0-b.1%2Ffiles.zip*~hmac=094a1e25d63e86f6e3023bc10bcd6a80eeeed3fd48a67203b1fb0685096b0681
Downloaded 0 B in 7s, Download speed: 0 B/s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-04 12:30:15.056714
Completed at: 2021-06-04 12:30:22.073949
Duration taken: 7s

Attempt 2 out of 3 failed
Trying again…
Downloaded 0 B in 6s, Download speed: 0 B/s
Client Error: 403 Response: - Request Id: Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/models/jmir_punctuation/versions/1.0.0-b.1/files.zip?ak-token=exp=1622813437~acl=%2Fmodels%2Forg%2Fnvidia%2Fteam%2Fjarvis%2Fmodels%2Fjmir_punctuation%2Fversions%2F1.0.0-b.1%2Ffiles.zip*~hmac=51d78a87421d44dc584b75178dd891a8fcbb4e4578e3fc2a236ae49aaed999d7
Downloaded 0 B in 7s, Download speed: 0 B/s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-04 12:30:34.085043
Completed at: 2021-06-04 12:30:41.097749
Duration taken: 7s

Attempt 3 out of 3 failed
Error occurred downloading ‘nvidia/jarvis/jmir_punctuation:1.0.0-b.1’. Exiting.
Error in downloading models.
`

shashikant.ghangare · June 4, 2021, 12:32pm

faced the same issue on 1.2.0-beta

HansieB · June 4, 2021, 2:52pm

@SunilJB Hi, any estimate as to when this will be fixed, ngc giving problems downloading the models 403 errors when trying to download the punctuation model in beta 1.1 and 1.2. Thanks in advance

rleary · June 4, 2021, 5:59pm

Fixed now. Sorry about that!

ellengsu · June 4, 2021, 6:55pm

thanks, but its not working for me still. What can i do to fix this? Please help. Thanks

rleary · June 4, 2021, 7:12pm

@ellengsu I recommend first pulling the latest quickstart (1.2.0-beta). Check that you have either $NGC_API_KEY variable set or the ngc utility installed and logged in (you would have a file $HOME/.ngc/config). Keep in mind that if you go to the NGC website to retrieve your API key, it will invalidate any previously used keys and you will need to update $NGC_API_KEY or login with the ngc command again. The 403 Response you are seeing is NGC reporting that you are not authorized to download the model. These models are available for any registered user, so it should work for you once the proper API key is used.

The problem earlier in the thread was a permission problem on NVIDIA’s side which prevented download of the new CitriNet model.

HansieB · June 4, 2021, 9:37pm

@rleary Hi, unfortunately I can’t even pull the quickstart anymore, using the same command I’ve been using for a month, even made a new API key and config set it. ngc registry resource download-version nvidia/jarvis/jarvis_quickstart:1.2.0-beta just returning failed.

kyletyler17 · June 4, 2021, 10:54pm

@rleary I’m in the same boat as HansieB, I was able to pull the quickstart yesterday, and attempting it today it fails. I also tried a new API key and config set it.

ellengsu · June 5, 2021, 4:31am

Thanks rleary,
I did have the ngc api key in place and ngc utility installed and logged in. And to be safe, i followed the instruction on this page to ensure the ngc configured properly.
https://ngc.nvidia.com/setup/installers/cli

But when running jarvis_init.sh, it’s the same 403 error like above.
Same error when trying to get the latest quick start guide. ngc registry resource download-version nvidia/jarvis/jarvis_quickstart:1.2.0-beta

john.patton · June 5, 2021, 7:04pm

I was able to pull the quickstart 1.2 a day or two ago but it has been failing to download models with the same 403 error. I also just want back to pull the quickstart again on a cloud gpu instead of local machine and I can’t even download that anymore.

$ docker login nvcr.io
Authenticating with existing credentials…
WARNING! Your password will be stored unencrypted in /.docker/config.json.
Configure a credential helper to remove this warning. See
docker login | Docker Documentation

Login Succeeded
$
$ ngc registry resource download-version “nvidia/jarvis/jarvis_quickstart:1.2.0-beta”

Downloaded 0 B in 1s, Download speed: 0 B/s
Client Error: 403 Response: - Request Id: Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/recipes/jarvis_quickstart/versions/1.2.0-beta/files.zip?ak-token=exp=1622923343~acl=%2Fmodels%2Forg%2Fnvidia%2Fteam%2Fjarvis%2Frecipes%2Fjarvis_quickstart%2Fversions%2F1.2.0-beta%2Ffiles.zip*~hmac=ae9dedeefd83262786fd08fdb7ae2f92f52da079af396c52ef966445ae7de164
Downloaded 0 B in 2s, Download speed: 0 B/s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed.
Downloaded local path: /jarvis2/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-05 15:02:21.752089
Completed at: 2021-06-05 15:02:23.756653
Duration taken: 2s

shashikant.ghangare · June 6, 2021, 1:22pm

I was also able to pull the quickstart:1.2.0-bata day before yesterday . but the 403 response now.
`
$ ngc registry resource download-version nvidia/jarvis/jarvis_quickstart:1.2.0-beta
Downloaded 0 B in 2s, Download speed: 0 B/s
Client Error: 403 Response: - Request Id: Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/recipes/jarvis_quickstart/versions/1.2.0-beta/files.zip?ak-token=exp=1622989180~acl=%2Fmodels%2Forg%2Fnvidia%2Fteam%2Fjarvis%2Frecipes%2Fjarvis_quickstart%2Fversions%2F1.2.0-beta%2Ffiles.zip*~hmac=73cc6e293323b481d978ac6cadec1afc1f21904160e9aece9b88e24a6bd12835
Downloaded 0 B in 3s, Download speed: 0 B/s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed.
Downloaded local path: /home/ubuntu/cv/jarvis2/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-06 09:19:38.262730
Completed at: 2021-06-06 09:19:41.268084
Duration taken: 3s

`

petasisg · June 7, 2021, 7:06am

Nothing works. Not even this:

ngc registry resource download-version nvidia/jarvis/jarvis_quickstart:1.2.0-beta
Downloaded 0 B in 3s, Download speed: 0 B/s               
Client Error: 403 Response:  - Request Id:  Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/recipes/jarvis_quickstart/versions/1.2.0-beta/files.zip?ak-token=exp=1623053126~acl=%2fmodels%2forg%2fnvidia%2fteam%2fjarvis%2frecipes%2fjarvis_quickstart%2fversions%2f1.2.0-beta%2ffiles.zip*~hmac=521faf7d4b148b048a791670c5a0d9be765ab1ac84062533e1dba56a23e3a5c7
Downloaded 0 B in 4s, Download speed: 0 B/s               
----------------------------------------------------
Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed.
Downloaded local path: /home/pepper/NVIDIA/jarvis/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 0 
Total downloaded size: 0 B
Started at: 2021-06-07 10:05:24.595887
Completed at: 2021-06-07 10:05:28.601328
Duration taken: 4s
----------------------------------------------------

jpcordovae · June 7, 2021, 1:16pm

Hi, same problem here, looks like nvidia don’t want jarvis to conquer the world…

jpcordovae@MelKor:~/Projects/NVidia-Jarvis$ ./ngc registry resource download-version nvidia/jarvis/jarvis_quickstart
No version specified, downloading latest version: ‘1.2.0-beta’.
Downloaded 0 B in 7s, Download speed: 0 B/s
Client Error: 403 Response: - Request Id: Url: https://ngc.download.nvidia.com/models/org/nvidia/team/jarvis/recipes/jarvis_quickstart/versions/1.2.0-beta/files.zip?ak- <long line here, I delete it>
Downloaded 0 B in 8s, Download speed: 0 B/s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed.
Downloaded local path: /home/jpcordovae/Projects/NVidia-Jarvis/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-07 09:10:13.909968
Completed at: 2021-06-07 09:10:21.925565
Duration taken: 8s

ruze55 · June 7, 2021, 8:53pm

I’m also not able to download jarvis_quickstart with NGC. I downloaded all the files manually from the web site, then I cannot pull the following docker image during jarvis_init.sh:

docker pull nvidia/jarvis/jmir_punctuation:1.2.0-beta
Error response from daemon: pull access denied for nvidia/jarvis/jmir_punctuation, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied

Here’s the full output since I know you will ask:

Logging into NGC docker registry if necessary…
Pulling required docker images if necessary…
Note: This may take some time, depending on the speed of your Internet connection.

Pulling Jarvis Speech Server images.
Image nvcr.io/nvidia/jarvis/jarvis-speech:1.2.0-beta-server exists. Skipping.
Image nvcr.io/nvidia/jarvis/jarvis-speech-client:1.2.0-beta exists. Skipping.
Image nvcr.io/nvidia/jarvis/jarvis-speech:1.2.0-beta-servicemaker exists. Skipping.

Downloading models (JMIRs) from NGC…
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing JMIRs set the location and corresponding flag in config.sh.

==========================
== Jarvis Speech Skills ==

NVIDIA Release devel (build 22382700)

… I had to remove this stuff because the forum complains that I am a new user and posting too many links …

Attempt 1 out of 3 failed
Trying again…
…etc… 2 more times
Attempt 3 out of 3 failed
Error occurred downloading ‘nvidia/jarvis/jmir_punctuation:1.2.0-beta’. Exiting.
Error in downloading models.

It seems clearly someone in the server development team has broken it for now. I changed https:// to h***s:/ / because I’m a new user and not allowed to post so many links.

ee_nvidiangc · June 9, 2021, 2:30pm

It appears to work again:

$ ngc registry resource download-version "nvidia/jarvis/jarvis_quickstart:1.2.0-beta"
Downloaded 261.8 KB in 6s, Download speed: 43.56 KB/s
----------------------------------------------------
Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Completed.
Downloaded local path: /path/to/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 28
Total downloaded size: 261.8 KB
Started at: 2021-06-09 16:17:09.206733
Completed at: 2021-06-09 16:17:15.216292
Duration taken: 6s
----------------------------------------------------

Topic		Replies	Views
Jarvis Support for GPU RTX 2060 Riva riva	4	930	July 19, 2021
Jarvis Health Ready check failed - Need Help DirectX, DXR, DirectCompute	0	617	July 1, 2021
Jarvis: Triton server timed-out before ready state Riva riva	2	1896	July 5, 2021
Build_waveglow failed to generate waveglow.eng. (in jarvis_init.sh) Riva riva	6	1038	September 22, 2021
Rub jarvis_start.sh: Got Health ready check failed DirectX, DXR, DirectCompute	0	566	June 26, 2021
Health ready check failed error whe running bash jarvis_start.sh Riva riva	4	1658	May 28, 2021
Quick start jarvis_init cuda out of memory Riva riva	3	1630	March 31, 2021
Waiting for Jarvis server to load all models...retrying in 10 seconds Riva riva	7	2631	April 30, 2021
Error during jarvis_init.sh for jarvis 1.2.1 beta Riva riva	7	1683	December 22, 2021
Init. Jarvis with german model Riva riva	9	1467	November 4, 2021

Jarvis Quick Start: Installation fails

========================== == Jarvis Speech Skills ==

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed. Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1 Total files downloaded: 0 Total downloaded size: 0 B Started at: 2021-06-04 12:29:58.020858 Completed at: 2021-06-04 12:30:04.033553 Duration taken: 6s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed. Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1 Total files downloaded: 0 Total downloaded size: 0 B Started at: 2021-06-04 12:30:15.056714 Completed at: 2021-06-04 12:30:22.073949 Duration taken: 7s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed. Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1 Total files downloaded: 0 Total downloaded size: 0 B Started at: 2021-06-04 12:30:34.085043 Completed at: 2021-06-04 12:30:41.097749 Duration taken: 7s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed. Downloaded local path: /jarvis2/jarvis_quickstart_v1.2.0-beta Total files downloaded: 0 Total downloaded size: 0 B Started at: 2021-06-05 15:02:21.752089 Completed at: 2021-06-05 15:02:23.756653 Duration taken: 2s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed. Downloaded local path: /home/ubuntu/cv/jarvis2/jarvis_quickstart_v1.2.0-beta Total files downloaded: 0 Total downloaded size: 0 B Started at: 2021-06-06 09:19:38.262730 Completed at: 2021-06-06 09:19:41.268084 Duration taken: 3s

========================== == Jarvis Speech Skills ==

Related topics

==========================
== Jarvis Speech Skills ==

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-04 12:29:58.020858
Completed at: 2021-06-04 12:30:04.033553
Duration taken: 6s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-04 12:30:15.056714
Completed at: 2021-06-04 12:30:22.073949
Duration taken: 7s

Transfer id: jmir_punctuation_v1.0.0-b.1 Download status: Failed.
Downloaded local path: /data/artifacts/jmir_punctuation_v1.0.0-b.1
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-04 12:30:34.085043
Completed at: 2021-06-04 12:30:41.097749
Duration taken: 7s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed.
Downloaded local path: /jarvis2/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-05 15:02:21.752089
Completed at: 2021-06-05 15:02:23.756653
Duration taken: 2s

Transfer id: jarvis_quickstart_v1.2.0-beta Download status: Failed.
Downloaded local path: /home/ubuntu/cv/jarvis2/jarvis_quickstart_v1.2.0-beta
Total files downloaded: 0
Total downloaded size: 0 B
Started at: 2021-06-06 09:19:38.262730
Completed at: 2021-06-06 09:19:41.268084
Duration taken: 3s

==========================
== Jarvis Speech Skills ==