Tegra System Profiler keep saying "internal error"

Good morning everyone,
I’m trying to profile a simple application that is using VisionWorks on my TX1 using the Tegra System Profiler.

However, despite a “Stop profiling success” message that appear on console in the board, the profiler says an “Internal error” and shows me only the CPU Timeline view but not the GPU one.

There are a lot of warnings, but almost all of that says:
“Events fetch failed: Source id=
Type=ErrorInformation (18)
Error information:
ProcessEventsError (4005)”

and the exact point in (I presume) the source code of the profiler that throws exceptions (InvalidArgumentException, LogicException).

How can I solve this problem? I need the informations extracted from the profiler. Thank you all in advance for support

Stefano

Hi,

Tegra system profiler work correctly on our side.
We can profile the nvx_demo_feature_tracker sample without error.

Could you share more information about your environment?
What Jetpack/VisionWorks/Host version do you use?

We run Tegra System Profiler contained in Jetpack3.1 on Ubuntu14.04.
Thanks.

Thank you for the input, that demo works flawless in my configuration too.

I have Ubuntu 14.04 as NVIDIA guidelines, I am using JetPack 3.1, release 28.1 of Linux, and VisionWorks 1.6. I have the Jetson TX1 development kit, and I am using Tegra System Profiler 3.8. All was flashed with standard configuration, no kernel modifications were made.

Guided by this tip, I reduced the code to the minimal possible configuration to find the issue. It’s not so long, I’ll post it here with the compiler line to compile it.

If you run this in the profiler, most of the time (even if not always) it will produce an error, in my case the error is the one that I’ll post here.

Basically, it just create a pyramid with 2 levels, and it will run a FAST detector on the 2nd level. The problem seems to be linked with image size (in this case, 1226x370). With a more common size (640x480, for example) the problems is reduced, albeit sometimes the profiler crash even with this common size.

So, can you try with some run of the profiler if I am not the only one with this problem? If I missed some important information let me know.
Thank you in advance

Error of the profiler

Events fetch failed: Source id=
Type=ErrorInformation (18)
 Error information:
 ProcessEventsError (4005)
  Properties:
  ErrorText (100)=/home/devtools/teamCityBuildAgent/work/20a3cfcd1c25021d/QuadD/Host/Analysis/Modules/EventCollection.cpp(1021): Throw in function void QuadDAnalysis::EventCollection::CheckOrder(QuadDAnalysis::EventCollectionHelper::EventContainer&, const QuadDAnalysis::ConstEvent&) const
Dynamic exception type: N5boost16exception_detail10clone_implIN11QuadDCommon24InvalidArgumentExceptionEEE
std::exception::what: InvalidArgumentException
[PN11QuadDCommon14tag_error_textE] = Wrong event order has been detected when adding events to the collection:
new event ={ StartNs=1528182448 StopNs=1528183750 GlobalId=281624193269760 Event={ CudaEvent=[{ CorrelationId=236 DeviceId=0 ContextId=1 StreamId=15 EventClass=2 EventCategory=0 Event={ Memset={ SizeBytes=4 Value=0 } } },] } Type=80 }
last event ={ StartNs=1528280520 StopNs=1528340416 GlobalId=281624193269760 Event={ CudaEvent=[{ CorrelationId=224 DeviceId=0 ContextId=1 StreamId=15 EventClass=2 EventCategory=0 Event={ Memset={ SizeBytes=1297448 Value=0 } } },] } Type=80 }

To compile the following file, use this line (in a standard Jetson TX1 configuration)

g++ -std=c++11 main.cpp -I /usr/share/visionworks/sources/nvxio/include/ -lvisionworks `pkg-config --cflags --libs opencv` -o main.x

File main.cpp

/*
 * Minimal example of VisionWorks problem with Tegra System Profiler.
 */

#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <exception>

#include <NVX/nvx.h>
#include <NVX/Utility.hpp>
#include <OVX/UtilityOVX.hpp>

using namespace std;

int main(int argc, char **argv)
{
	int width; int height;
	if(argc < 3)
	{
		width  = 1226;
		height = 370;
	}
	else
	{
		try
		{
			width  = std::stoi(argv[1]);
			height = std::stoi(argv[2]);
		}
		catch(std::exception& e)
		{
			std::cout << "Launch: " << argv[0] << " width height" << std::endl;
			exit(1);
		}
	}
	// context
	vx_context context;
	// graph
	vx_graph global_graph;
	// images
	vx_image image_start;
	vx_image image_level_one;
	// scalars
	vx_scalar s_strength_threshold;
	vx_scalar corners;
	// array
	vx_array keypoints;
	// pyramid
	vx_pyramid gaussian_pyramid;
	
	context = vxCreateContext();
	NVXIO_CHECK_REFERENCE( context );
	vxDirective((vx_reference)context, NVX_DIRECTIVE_ENABLE_PERFORMANCE);

	vx_pixel_value_t p;
	p.U8 = 0;
	image_start      = vxCreateUniformImage(context, width, height, VX_DF_IMAGE_U8, &p);
	NVXIO_CHECK_REFERENCE(image_start);
	
	global_graph = vxCreateGraph(context);
	NVXIO_CHECK_REFERENCE( global_graph );

	float fast_t = 10;
	s_strength_threshold = vxCreateScalar(context, VX_TYPE_FLOAT32, &fast_t);
	NVXIO_CHECK_REFERENCE( s_strength_threshold );
	// allocate some space to store the keypoints
	keypoints = vxCreateVirtualArray(global_graph, VX_TYPE_KEYPOINT, 400);
	NVXIO_CHECK_REFERENCE( keypoints );
	vx_size sz_corners = 0;
	corners = vxCreateScalar(context, VX_TYPE_SIZE, &sz_corners);
	NVXIO_CHECK_REFERENCE( corners );

	// build the pyramid
	gaussian_pyramid = vxCreateVirtualPyramid(global_graph, 2, VX_SCALE_PYRAMID_ORB, width, height, VX_DF_IMAGE_U8);
	NVXIO_CHECK_REFERENCE( gaussian_pyramid );
	vxGaussianPyramidNode(global_graph, image_start, gaussian_pyramid);
	// get reference to the second level
	image_level_one = vxGetPyramidLevel(gaussian_pyramid, 1);
	NVXIO_CHECK_REFERENCE( image_level_one );
	// do the FAST corners detection
	vxFastCornersNode(global_graph, image_level_one, s_strength_threshold, vx_true_e, keypoints, corners);

	NVXIO_SAFE_CALL( vxVerifyGraph(global_graph) );

	// process 2 times the graph to arise the issue!
	for(int i = 0; i < 2; i++)
	{
		vxProcessGraph(global_graph);
	}

	vxReleaseImage(&image_level_one);
	vxReleaseImage(&image_start);
	
	vxReleaseScalar(&corners);
	vxReleaseScalar(&s_strength_threshold);
	
	vxReleasePyramid(&gaussian_pyramid);
	
	vxReleaseArray(&keypoints);

	vxReleaseGraph(&global_graph);
	vxReleaseContext(&context);
	
	return 0;
}

Hi,

Thanks for your sharing.

We have tried your source ten times, but all test work correctly.
(Launch with ‘Attach or launch’ mode)

Could you share more information to help us reproduce this issue?
Thanks.

I wrote all the steps I did to my Jetson to reproduce the issue.
I attach the link to the qdrep file, maybe you will find some interesting information that I am not able to see:

https://www.dropbox.com/s/lpqfmmfplse4khq/test_report_fail.qdrep?dl=0

I noticed another strange thing: sometimes, Tegra System Profiler disconnect from the Jetson TX1 while it is being profiled. The selection menu went from “nvidia@xxx.xxx.xxx.xxx” to blank. Jetson is wired while host is on Wi-Fi.

To launch the tests, I used the default selections with “Attach or launch”, and I tick the “Collect NVTX trace” and “Collect CUDA trace” flags.

Is there a way to use the profiler only on the device side, or a way to provide you a log file of all the operations that are performing on Jetson and host?
Some ideas to help to identify if the problem is in the source code, compilation flags, the device configuration or the host configuration?

Thank you for the support

Hi,

Thanks for your sharing.
We are discussing the possible cause of this issue internally. Will update information with you later.

Thanks and sorry for any inconvenience.

Hello, do you have any information about this issue?
It’s blocking to my analysis. How can I do to fix these errors?

Thank you in advance

Hi,

Sorry for the inconvenience.
We are still working on this issue. Will update information with you later.

Thanks.

Hi,

Sorry for the late reply. Here is a WAR on this issue.

1. Download Tegra System Profiler 3.9 from JetPack3.2:
https://developer.nvidia.com/embedded/jetpack-notes

2. Set a configuration option

echo "SkipInvalidCudaEvents = true" > config.ini
./SystemProfiler

Thanks.