NGC-Public Repo Sync Fails : Connection Timed out

• Hardware Platform (Jetson / GPU) Tesla P4
• DeepStream Version 6.1.1
• JetPack Version (valid for Jetson only)
• TensorRT Version 8.4.1
• NVIDIA GPU Driver Version (valid for GPU only) 515.65.01
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ?
By syncing the Graph Composer’s registry to ngc-public repo.

Terminal output:

2022-09-22 14:34:39,043 - Registry - INFO - Syncing repo ngc-public ...
2022-09-22 14:34:42,027 - Registry - INFO - 38 extensions found
2022-09-22 14:34:43,811 - Registry - INFO - Syncing extension NvDsBaseExt version 1.0.0
2022-09-22 14:35:09,270 - Registry - INFO - Syncing extension NvDsAnalyticsExt version 1.0.0
HTTPSConnectionPool(host='prod-model-registry-ngc-bucket.s3.us-west-2.amazonaws.com', port=443): Max retries exceeded with url: /org/nvidia/team/graph-composer/recipes/nvdsanalyticsext/versions/1.0.0/files/extension.yaml?response-content-disposition=attachment%3B%20filename%3D%22extension.yaml%22&response-content-type=application%2Foctet-stream&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEN7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJIMEYCIQCeALx2WxRBvTWFTiLn63necXkfsbifaaMbqFrNq%2BB3SgIhAMtn4NF0%2BjED19AGZThRPOvJizyfHbK1DiNEJ%2FcLmFdjKtUECIf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQBBoMNzg5MzYzMTM1MDI3IgxTiz8NiCt4oJxHn%2BwqqQQBjMsn%2FjIGKx6fHr0TamW%2F5M0dxWrSN8Mk4sLMi7MHjBHcOSDVVhaut9Yh28EcyVh5g59dJGC8YVZtd2r97uZ7jcVThi500BGlQE%2FwdBecZdfRAnxlD0dpUXQeUgnGZNaL5vjFtwgnChO4F2Yzm4XxWYYA6T2Ef41uNWQ%2BFtpYOjAOwdRwqGIByu%2FKrVPQ5RAzvIQLi2BREpxzDlpv%2BjHa7ylkx%2FEqJGeSmqgJylAqINUQBQoZtnu8gmKEQRn8AkA5Cqt%2FZ%2BGpder%2B5RtZGLXsncuLVgdUX8kxkGWhznYkrY%2Fbe4cap5UY5aNzbQZOF0DGYfVn%2FyaDTDmVDoywgOu4mGKYhBP1MQzyRYoHNQDLR7iJBxiynb2hkz21es8Bdcfd%2FKztIMhxuDFYtwUjCdsqYQkRHjkF2uwREQsxwusbKnTlIoZ8psh%2B10kexqdFUMSEhW%2FC5KmcwNDZR8e0lhXP9IRXizZtpvhYOPd%2BjynciJc9GEoQaL5Z1OURfsddrbeTwGt6rStXPfEAYnoueAwJGZI%2FgzVjcH6PqTU%2BTXh9JK8%2Bz7aIi1LK3a1UJmwN2Myy2XwfKQIxP2kiNckmpilS%2Bx%2Bq2vhqkKRy3LMo9CvtUKgzRKXm4GLziVicBgHYByq%2FbGtI4ql4TE1zZVdmpIYrpt8mbethQsHNqMAOQ71PvPkyuqKS3ZNNVXkv2CrkkBqFsP%2F5LfORiCzRKmqCiSdPmyp4cQE7%2BS4EMP%2Fwr5kGOqgBZYoTIujVnv3SObEmexigRwnTEr89o2ywp2SgdcwFdU5NkThl%2BL4LroUsL5IxdrQOZ7r2KYdjSlZDBCbkuIjlMjSQCEt9EAXPVbHRiBfwI1bphZ6zQH7NA0thZL5WvzDKoRJ9PN2S%2FRqLFI5W2mJORngdV7sVA29yfwWoxyUkp%2B%2BqEaMDrLCsc7ESQNv6rYyUsp%2F3UtRVgj1hD%2BaHPL3xA8GnHOfO%2B6J%2B&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20220922T063512Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3599&X-Amz-Credential=ASIA3PSNVSIZ4B54GEMC%2F20220922%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=e4c405282c049e8ea80627681d324dd5e81e78b2d92e893d1541a71446c4babd (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fca345793a0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Internet connection is fine and I’m already logged into NGC container registry.

We will check this issue soon.

Have you used Graph Composer before?

Thank you.

Yes, and I had no issues syncing the ngc-public repo during my previous installation of Graph Composer. This is a fresh installation on a different machine.

Can you try more times? Sometimes the http service is not stable.

I’ve tried for more than 20 times already including today. Well, I’ll try a few more times then.

It’s still the same @Fiona.Chen.

It may take some time for the team to investigate the issue.
Can you reinstall graph composer, reboot the device and try registry sync after reinstallation?

I don’t think rebooting the device or reinstallation would resolve this issue although, I’ve already tried them. However, I managed to find a work-around by using a different network connection (personal mobile hotspot). Hence, the issue seems to be with my organization’s Internet (or IP). I don’t think it’s due to being blacklisted by or blacklisting the AWS host because connections could be established; in fact, a number of extensions (inconstant) get synced successfully all the time before the connection times out. By the way, I didn’t have this issue till last week. I’ve successfully synced the repository several times on several machines before.

I’d also like to suggest the sync process to be tweaked slightly if that’s possible. It’d be better to check for existing extensions, then syncing the remaining ones instead of syncing all the extensions by overwriting, every time the sync process is run. This way one can successfully sync all the extensions over a few tries, in case of connection timeouts.

@Fiona.Chen Any updates on this? It also happened to me too. Failed to sync every time

Try using a different network connection. I used my personal mobile internet (hotspot) and got them synced successfully.

Still no luck. I’ve tried with different providers/isps

Quick update, I managed to sync with the repo using VPN. Is it possible store the registry in other region as an alternative? We’re in Asia Pacific and the closest is Singapore (ap-southeast-1)

@nisyhaal It is hard for the team to reproduce the timeout. Is there any special settings?

@Fiona.Chen No special settings were made