Graph Composer Build and Run Stopped working

I have build and run scripts for a series of graphs as test fixtures on a x86 Ubuntu 20.04 Graph Composer Server. This morning while trying to get a container running on a Jetson Jetpack 5.0.2 (another ticket) my Graphs stopped running or building… I have a terminal log showing successful builds then doing a registry repo sync -n ngc-public (with errors which is new) and then the next time I tried to build I got errors… I haven’t changed the build scripts, the run scripts, or the yaml for my test harnesses…

Any Ideas where I can look for logs? or other information? I can send more information if there’s something useful.

Here’s key points out of my terminal history:

Build ran fine

2022-08-28 11:51:38,370 - Registry - INFO - Installing graph …
2022-08-28 11:51:38,394 - Registry - INFO -
Requested target configuration:
arch: aarch64 os: linux distribution: ubuntu_20.04 cuda: 11.4 cudnn: 8.3.1.22 tensorrt: 8.4.0.2 deepstream: 6.1 vpi: 2.0.7

Selected extension variants:
StandardExtension
Version: 2.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 cuda: 11.4
MultimediaExtension
Version: 2.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04
NvDsInterfaceExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsBaseExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsMuxDemuxExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsTrackerExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsOutputSinkExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsVisualizationExt
Version: 1.0.1 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsInferenceExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsInferenceUtilsExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1

Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04
NvDsSourceExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1

Was afk then decided to do a registry update to see if it would help the other issues.

2022-08-28 13:21:37,161 - Registry - INFO - Syncing repo ngc-public …
2022-08-28 13:21:38,980 - Registry - INFO - 38 extensions found
2022-08-28 13:21:39,395 - Registry - INFO - Syncing extension NvDsBaseExt version 1.0.0
2022-08-28 13:21:52,285 - Registry - INFO - Syncing extension NvDsAnalyticsExt version 1.0.0
2022-08-28 13:22:04,577 - Registry - INFO - Syncing extension NvDsCloudMsgExt version 1.0.0
2022-08-28 13:22:17,065 - Registry - INFO - Syncing extension NvDsConverterExt version 1.0.0
2022-08-28 13:22:29,704 - Registry - INFO - Syncing extension NvDsDewarperExt version 1.0.0
2022-08-28 13:22:42,081 - Registry - INFO - Syncing extension NvDsInferenceExt version 1.0.0
2022-08-28 13:22:54,491 - Registry - INFO - Syncing extension NvDsInferenceUtilsExt version 1.0.0
2022-08-28 13:23:06,902 - Registry - INFO - Syncing extension NvDsInterfaceExt version 1.0.0
2022-08-28 13:23:19,146 - Registry - INFO - Syncing extension NvDsMuxDemuxExt version 1.0.0
2022-08-28 13:23:31,557 - Registry - INFO - Syncing extension NvDsOpticalFlowExt version 1.0.0
2022-08-28 13:23:44,116 - Registry - INFO - Syncing extension NvDsOutputSinkExt version 1.0.0
2022-08-28 13:23:56,579 - Registry - INFO - Syncing extension NvDsSampleExt version 1.0.0
2022-08-28 13:24:08,903 - Registry - INFO - Syncing extension NvDsSampleModelsExt version 1.0.0
2022-08-28 13:24:21,606 - Registry - INFO - Syncing extension NvDsSourceExt version 1.0.0
2022-08-28 13:24:34,516 - Registry - INFO - Syncing extension NvDsTemplateExt version 1.0.0
2022-08-28 13:24:47,165 - Registry - INFO - Syncing extension NvDsTrackerExt version 1.0.0
2022-08-28 13:24:59,404 - Registry - INFO - Syncing extension NvDsTranscodeExt version 1.0.0
2022-08-28 13:25:11,850 - Registry - INFO - Syncing extension NvDsTritonExt version 1.0.0
2022-08-28 13:25:24,297 - Registry - INFO - Syncing extension NvDsVisualizationExt version 1.0.2
2022-08-28 13:25:26,442 - Registry - WARNING - Skipping extension NvDsVisualizationExt version 1.0.2
2022-08-28 13:25:26,442 - Registry - WARNING - Failed to sync extension NvDsVisualizationExt version 1.0.2
2022-08-28 13:25:26,442 - Registry - INFO - Syncing extension NvDsVisualizationExt version 1.0.1
2022-08-28 13:25:34,744 - Registry - INFO - Syncing extension NvDsVisualizationExt version 1.0.0
2022-08-28 13:25:43,172 - Registry - INFO - Syncing extension StandardExtension version 2.0.0
2022-08-28 13:25:53,750 - Registry - INFO - Syncing extension NvDsBodyPose2D version 1.0.0
2022-08-28 13:26:05,931 - Registry - INFO - Syncing extension NvDsFacialLandmarks version 1.0.0
2022-08-28 13:26:18,246 - Registry - INFO - Syncing extension NvDsEmotionExt version 1.0.0
2022-08-28 13:26:30,594 - Registry - INFO - Syncing extension NvDsGazeExt version 1.0.0
2022-08-28 13:26:43,013 - Registry - INFO - Syncing extension NvDsHeartRateExt version 1.0.0
2022-08-28 13:26:55,642 - Registry - INFO - Syncing extension MultimediaExtension version 2.0.0
2022-08-28 13:27:03,873 - Registry - INFO - Syncing extension NvDsGesture version 1.0.0
2022-08-28 13:27:16,004 - Registry - INFO - Syncing extension NvDsActionRecognitionExt version 1.0.0
2022-08-28 13:27:28,216 - Registry - INFO - Syncing extension CudaExtension version 2.0.0
2022-08-28 13:27:38,556 - Registry - INFO - Syncing extension DeepStreamExtensionTest version 1.0.0
2022-08-28 13:27:51,059 - Registry - INFO - Syncing extension NvDs3dProcessingExt version 0.0.3
2022-08-28 13:28:02,970 - Registry - INFO - Syncing extension NvDs3dProcessingExt version 0.0.2
2022-08-28 13:28:11,148 - Registry - INFO - Syncing extension NvDs3dProcessingExt version 0.0.1
2022-08-28 13:28:19,305 - Registry - INFO - Syncing extension NvDsUcxExt version 0.0.1
2022-08-28 13:28:31,636 - Registry - INFO - Syncing extension NvTritonExt version 0.0.8
2022-08-28 13:28:43,951 - Registry - INFO - Syncing extension PythonCodeletExtension version 0.0.1
2022-08-28 13:28:54,294 - Registry - INFO - Syncing extension SampleExtension version 1.0.0
2022-08-28 13:29:02,406 - Registry - INFO - Syncing extension SerializationExtension version 2.0.0
2022-08-28 13:29:10,551 - Registry - INFO - Syncing extension TensorRTExtension version 2.0.0
2022-08-28 13:29:22,783 - Registry - INFO - Syncing extension NvDsUdpExt version 1.0.0
Repository synced

Now when I try to run a graph:
2022-08-28 14:09:37,425 - Registry - INFO - Installing graph …
Traceback (most recent call last):
File “registry/run.py”, line 10, in
File “registry/run.py”, line 7, in main
File “”, line 982, in main
File “”, line 969, in read_entry_and_execute
File “”, line 949, in execute
File “”, line 388, in execute_on
File “”, line 160, in call_subparsers
File “”, line 377, in install
File “”, line 305, in install_graph_with_dir
File “”, line 979, in install_graph_with_dir
File “”, line 907, in install_graph
File “”, line 233, in get_best_variants
File “”, line 272, in _compute_dependency_match
File “”, line 318, in _version_compare
TypeError: ‘module’ object is not callable
[397086] Failed to execute script ‘run’ due to unhandled exception!


When I try to build a container:
2022-08-28 14:30:42,930 - Registry - INFO - Installing graph …
Traceback (most recent call last):
File “container_builder/cli/cli.py”, line 59, in
File “container_builder/cli/cli.py”, line 53, in main
File “container_builder/cli/container_builder_runtime.py”, line 44, in build
File “container_builder/docker_manager/docker_manager.py”, line 457, in build_main_yaml
File “container_builder/docker_manager/docker_manager.py”, line 514, in build_stage_image
File “container_builder/docker_manager/docker_workfolder.py”, line 106, in prepare_all
File “container_builder/docker_manager/graph_handler.py”, line 226, in install_graph
File “container_builder/docker_manager/graph_handler.py”, line 168, in _download_unpack_extensions
File “registry/core/core_interface.py”, line 284, in install_graph_with_archive
File “registry/core/core.py”, line 935, in install_graph_with_archive
File “registry/core/core.py”, line 979, in install_graph_with_dir
File “registry/core/core.py”, line 907, in install_graph
File “registry/core/dependency_governer.py”, line 233, in get_best_variants
File “registry/core/dependency_governer.py”, line 269, in _compute_dependency_match
File “registry/core/dependency_governer.py”, line 318, in _version_compare
TypeError: ‘module’ object is not callable
[3792] Failed to execute script ‘cli’ due to unhandled exception!
X86 container build unsuccessful

Which Jetson device are you working on?

@Fiona.Chen
Jetson Xavier NX 16GB
(3668 - 0003)

Which graph are you working on?

@Fiona.Chen
Several of my own including ones with output to the screen, to a file, to an rtsp and I also tried the samples (I just retested deepstream-test1 with the same results).

They worked up until yesterday afternoon and all stopped working at the same time… The only thing that I knowingly did between was a registry sync as noted…

This seems like it may be related to Deepstream 6.1 reference_graphs fail with TypeError: 'module' object is not callable

So the graphs worked before you sync the extensions for the secondary time?

Yes, Exactly…

I don’t know that it’s caused by the registry update, but it is the only change that I did between the time that it worked and the time that it failed.

It is worth noting that the sync has an error if you look at the logs that I attached. The other thread reporting this error says that he also gets that error on sync.

We have published DeepStream 6.1.1 now. Please upgrade to DeepStream 6.1.1 and try again. DeepStream 6.1.1 is now available with support for JetPack 5.0.2 and an updated compute stack - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

Thanks for the information… I have upgraded to the new version and the error went away…

It appears that the ngc-public ended up with 6.1.1 dependent modules before 6.1.1 was released?

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

“ngc-public” is updated the same day with DeepStream 6.1.1.