I have build and run scripts for a series of graphs as test fixtures on a x86 Ubuntu 20.04 Graph Composer Server. This morning while trying to get a container running on a Jetson Jetpack 5.0.2 (another ticket) my Graphs stopped running or building… I have a terminal log showing successful builds then doing a registry repo sync -n ngc-public (with errors which is new) and then the next time I tried to build I got errors… I haven’t changed the build scripts, the run scripts, or the yaml for my test harnesses…
Any Ideas where I can look for logs? or other information? I can send more information if there’s something useful.
Here’s key points out of my terminal history:
Build ran fine
2022-08-28 11:51:38,370 - Registry - INFO - Installing graph …
2022-08-28 11:51:38,394 - Registry - INFO -
Requested target configuration:
arch: aarch64 os: linux distribution: ubuntu_20.04 cuda: 11.4 cudnn: 8.3.1.22 tensorrt: 8.4.0.2 deepstream: 6.1 vpi: 2.0.7
Selected extension variants:
StandardExtension
Version: 2.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 cuda: 11.4
MultimediaExtension
Version: 2.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04
NvDsInterfaceExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsBaseExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsMuxDemuxExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsTrackerExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsOutputSinkExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsVisualizationExt
Version: 1.0.1 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsInferenceExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
NvDsInferenceUtilsExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04
NvDsSourceExt
Version: 1.0.0 Variant: arch: aarch64 os: linux distribution: ubuntu_20.04 deepstream: 6.1
Was afk then decided to do a registry update to see if it would help the other issues.
2022-08-28 13:21:37,161 - Registry - INFO - Syncing repo ngc-public …
2022-08-28 13:21:38,980 - Registry - INFO - 38 extensions found
2022-08-28 13:21:39,395 - Registry - INFO - Syncing extension NvDsBaseExt version 1.0.0
2022-08-28 13:21:52,285 - Registry - INFO - Syncing extension NvDsAnalyticsExt version 1.0.0
2022-08-28 13:22:04,577 - Registry - INFO - Syncing extension NvDsCloudMsgExt version 1.0.0
2022-08-28 13:22:17,065 - Registry - INFO - Syncing extension NvDsConverterExt version 1.0.0
2022-08-28 13:22:29,704 - Registry - INFO - Syncing extension NvDsDewarperExt version 1.0.0
2022-08-28 13:22:42,081 - Registry - INFO - Syncing extension NvDsInferenceExt version 1.0.0
2022-08-28 13:22:54,491 - Registry - INFO - Syncing extension NvDsInferenceUtilsExt version 1.0.0
2022-08-28 13:23:06,902 - Registry - INFO - Syncing extension NvDsInterfaceExt version 1.0.0
2022-08-28 13:23:19,146 - Registry - INFO - Syncing extension NvDsMuxDemuxExt version 1.0.0
2022-08-28 13:23:31,557 - Registry - INFO - Syncing extension NvDsOpticalFlowExt version 1.0.0
2022-08-28 13:23:44,116 - Registry - INFO - Syncing extension NvDsOutputSinkExt version 1.0.0
2022-08-28 13:23:56,579 - Registry - INFO - Syncing extension NvDsSampleExt version 1.0.0
2022-08-28 13:24:08,903 - Registry - INFO - Syncing extension NvDsSampleModelsExt version 1.0.0
2022-08-28 13:24:21,606 - Registry - INFO - Syncing extension NvDsSourceExt version 1.0.0
2022-08-28 13:24:34,516 - Registry - INFO - Syncing extension NvDsTemplateExt version 1.0.0
2022-08-28 13:24:47,165 - Registry - INFO - Syncing extension NvDsTrackerExt version 1.0.0
2022-08-28 13:24:59,404 - Registry - INFO - Syncing extension NvDsTranscodeExt version 1.0.0
2022-08-28 13:25:11,850 - Registry - INFO - Syncing extension NvDsTritonExt version 1.0.0
2022-08-28 13:25:24,297 - Registry - INFO - Syncing extension NvDsVisualizationExt version 1.0.2
2022-08-28 13:25:26,442 - Registry - WARNING - Skipping extension NvDsVisualizationExt version 1.0.2
2022-08-28 13:25:26,442 - Registry - WARNING - Failed to sync extension NvDsVisualizationExt version 1.0.2
2022-08-28 13:25:26,442 - Registry - INFO - Syncing extension NvDsVisualizationExt version 1.0.1
2022-08-28 13:25:34,744 - Registry - INFO - Syncing extension NvDsVisualizationExt version 1.0.0
2022-08-28 13:25:43,172 - Registry - INFO - Syncing extension StandardExtension version 2.0.0
2022-08-28 13:25:53,750 - Registry - INFO - Syncing extension NvDsBodyPose2D version 1.0.0
2022-08-28 13:26:05,931 - Registry - INFO - Syncing extension NvDsFacialLandmarks version 1.0.0
2022-08-28 13:26:18,246 - Registry - INFO - Syncing extension NvDsEmotionExt version 1.0.0
2022-08-28 13:26:30,594 - Registry - INFO - Syncing extension NvDsGazeExt version 1.0.0
2022-08-28 13:26:43,013 - Registry - INFO - Syncing extension NvDsHeartRateExt version 1.0.0
2022-08-28 13:26:55,642 - Registry - INFO - Syncing extension MultimediaExtension version 2.0.0
2022-08-28 13:27:03,873 - Registry - INFO - Syncing extension NvDsGesture version 1.0.0
2022-08-28 13:27:16,004 - Registry - INFO - Syncing extension NvDsActionRecognitionExt version 1.0.0
2022-08-28 13:27:28,216 - Registry - INFO - Syncing extension CudaExtension version 2.0.0
2022-08-28 13:27:38,556 - Registry - INFO - Syncing extension DeepStreamExtensionTest version 1.0.0
2022-08-28 13:27:51,059 - Registry - INFO - Syncing extension NvDs3dProcessingExt version 0.0.3
2022-08-28 13:28:02,970 - Registry - INFO - Syncing extension NvDs3dProcessingExt version 0.0.2
2022-08-28 13:28:11,148 - Registry - INFO - Syncing extension NvDs3dProcessingExt version 0.0.1
2022-08-28 13:28:19,305 - Registry - INFO - Syncing extension NvDsUcxExt version 0.0.1
2022-08-28 13:28:31,636 - Registry - INFO - Syncing extension NvTritonExt version 0.0.8
2022-08-28 13:28:43,951 - Registry - INFO - Syncing extension PythonCodeletExtension version 0.0.1
2022-08-28 13:28:54,294 - Registry - INFO - Syncing extension SampleExtension version 1.0.0
2022-08-28 13:29:02,406 - Registry - INFO - Syncing extension SerializationExtension version 2.0.0
2022-08-28 13:29:10,551 - Registry - INFO - Syncing extension TensorRTExtension version 2.0.0
2022-08-28 13:29:22,783 - Registry - INFO - Syncing extension NvDsUdpExt version 1.0.0
Repository synced
Now when I try to run a graph:
2022-08-28 14:09:37,425 - Registry - INFO - Installing graph …
Traceback (most recent call last):
File “registry/run.py”, line 10, in
File “registry/run.py”, line 7, in main
File “”, line 982, in main
File “”, line 969, in read_entry_and_execute
File “”, line 949, in execute
File “”, line 388, in execute_on
File “”, line 160, in call_subparsers
File “”, line 377, in install
File “”, line 305, in install_graph_with_dir
File “”, line 979, in install_graph_with_dir
File “”, line 907, in install_graph
File “”, line 233, in get_best_variants
File “”, line 272, in _compute_dependency_match
File “”, line 318, in _version_compare
TypeError: ‘module’ object is not callable
[397086] Failed to execute script ‘run’ due to unhandled exception!
When I try to build a container:
2022-08-28 14:30:42,930 - Registry - INFO - Installing graph …
Traceback (most recent call last):
File “container_builder/cli/cli.py”, line 59, in
File “container_builder/cli/cli.py”, line 53, in main
File “container_builder/cli/container_builder_runtime.py”, line 44, in build
File “container_builder/docker_manager/docker_manager.py”, line 457, in build_main_yaml
File “container_builder/docker_manager/docker_manager.py”, line 514, in build_stage_image
File “container_builder/docker_manager/docker_workfolder.py”, line 106, in prepare_all
File “container_builder/docker_manager/graph_handler.py”, line 226, in install_graph
File “container_builder/docker_manager/graph_handler.py”, line 168, in _download_unpack_extensions
File “registry/core/core_interface.py”, line 284, in install_graph_with_archive
File “registry/core/core.py”, line 935, in install_graph_with_archive
File “registry/core/core.py”, line 979, in install_graph_with_dir
File “registry/core/core.py”, line 907, in install_graph
File “registry/core/dependency_governer.py”, line 233, in get_best_variants
File “registry/core/dependency_governer.py”, line 269, in _compute_dependency_match
File “registry/core/dependency_governer.py”, line 318, in _version_compare
TypeError: ‘module’ object is not callable
[3792] Failed to execute script ‘cli’ due to unhandled exception!
X86 container build unsuccessful