caffe make runtest failed

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX1
1

ubuntu@tegra-ubuntu:~/caffe$
sudo make -j4 runtest
.build_release/tools/caffe
caffe: command line brew
usage: caffe

commands:
train train or finetune a model
test score a model
device_query show GPU diagnostic information
time benchmark model execution time

Flags from tools/caffe.cpp:
-gpu (Optional; run in GPU mode on given device IDs separated by ‘,’.Use
‘-gpu all’ to run on all available GPUs. The effective training batch
size is multiplied by the number of devices.) type: string default: “”
-iterations (The number of iterations to run.) type: int32 default: 50
-level (Optional; network level.) type: int32 default: 0
-model (The model definition protocol buffer text file.) type: string
default: “”
-phase (Optional; network phase (TRAIN or TEST). Only used for ‘time’.)
type: string default: “”
-sighup_effect (Optional; action to take when a SIGHUP signal is received:
snapshot, stop or none.) type: string default: “snapshot”
-sigint_effect (Optional; action to take when a SIGINT signal is received:
snapshot, stop or none.) type: string default: “stop”
-snapshot (Optional; the snapshot solver state to resume training.)
type: string default: “”
-solver (The solver definition protocol buffer text file.) type: string
default: “”
-stage (Optional; network stages (not to be confused with phase), separated
by ‘,’.) type: string default: “”
-weights (Optional; the pretrained weights to initialize finetuning,
separated by ‘,’. Cannot be set simultaneously with snapshot.)
type: string default: “”
.build_release/test/test_all.testbin 0 --gtest_shuffle --gtest_filter=“-GPU”
Cuda number of devices: 1
Setting to use device 0
Current device id: 0
Current device name: NVIDIA Tegra X1
Note: Google Test filter = -GPU
Note: Randomizing tests’ orders with a seed of 79088 .
[==========] Running 2057 tests from 271 test cases.
[----------] Global test environment set-up.
[----------] 26 tests from IOTest
[ RUN ] IOTest.TestReadImageToDatumReference
[ OK ] IOTest.TestReadImageToDatumReference (21 ms)
[ RUN ] IOTest.TestReadImageToCVMat
[ OK ] IOTest.TestReadImageToCVMat (6 ms)
[ RUN ] IOTest.TestReadImageToCVMatResized
[ OK ] IOTest.TestReadImageToCVMatResized (11 ms)
[ RUN ] IOTest.TestReadImageToDatumGray
[ OK ] IOTest.TestReadImageToDatumGray (4 ms)
[ RUN ] IOTest.TestReadImageToDatumReferenceResized
[ OK ] IOTest.TestReadImageToDatumReferenceResized (13 ms)
[ RUN ] IOTest.TestReadImageToCVMatResizedSquare
[ OK ] IOTest.TestReadImageToCVMatResizedSquare (8 ms)
[ RUN ] IOTest.TestReadImageToDatumContent
[ OK ] IOTest.TestReadImageToDatumContent (16 ms)
[ RUN ] IOTest.TestReadImageToDatum
[ OK ] IOTest.TestReadImageToDatum (9 ms)
[ RUN ] IOTest.TestReadImageToDatumResized
[ OK ] IOTest.TestReadImageToDatumResized (6 ms)
[ RUN ] IOTest.TestReadImageToCVMatGray
[ OK ] IOTest.TestReadImageToCVMatGray (3 ms)
[ RUN ] IOTest.TestCVMatToDatum
[ OK ] IOTest.TestCVMatToDatum (7 ms)
[ RUN ] IOTest.TestReadImageToCVMatResizedGray
[ OK ] IOTest.TestReadImageToCVMatResizedGray (4 ms)
[ RUN ] IOTest.TestDecodeDatum
[ OK ] IOTest.TestDecodeDatum (25 ms)
[ RUN ] IOTest.TestDecodeDatumToCVMatNativeGray
[ OK ] IOTest.TestDecodeDatumToCVMatNativeGray (3 ms)
[ RUN ] IOTest.TestReadImageToDatumResizedSquare
[ OK ] IOTest.TestReadImageToDatumResizedSquare (9 ms)
[ RUN ] IOTest.TestDecodeDatumNative
[ OK ] IOTest.TestDecodeDatumNative (21 ms)
[ RUN ] IOTest.TestReadImageToDatumResizedGray
[ OK ] IOTest.TestReadImageToDatumResizedGray (5 ms)
[ RUN ] IOTest.TestDecodeDatumToCVMatContentNative
[ OK ] IOTest.TestDecodeDatumToCVMatContentNative (24 ms)
[ RUN ] IOTest.TestCVMatToDatumContent
[ OK ] IOTest.TestCVMatToDatumContent (18 ms)
[ RUN ] IOTest.TestDecodeDatumNativeGray
[ OK ] IOTest.TestDecodeDatumNativeGray (8 ms)
[ RUN ] IOTest.TestDecodeDatumToCVMat
[ OK ] IOTest.TestDecodeDatumToCVMat (10 ms)
[ RUN ] IOTest.TestDecodeDatumToCVMatNative
[ OK ] IOTest.TestDecodeDatumToCVMatNative (5 ms)
[ RUN ] IOTest.TestDecodeDatumToCVMatContent
[ OK ] IOTest.TestDecodeDatumToCVMatContent (23 ms)
[ RUN ] IOTest.TestCVMatToDatumReference
[ OK ] IOTest.TestCVMatToDatumReference (20 ms)
[ RUN ] IOTest.TestReadFileToDatum
[ OK ] IOTest.TestReadFileToDatum (0 ms)
[ RUN ] IOTest.TestReadImageToDatumContentGray
[ OK ] IOTest.TestReadImageToDatumContentGray (10 ms)
[----------] 26 tests from IOTest (293 ms total)

[----------] 3 tests from FilterLayerTest/2, where TypeParam = caffe::GPUDevice
[ RUN ] FilterLayerTest/2.TestReshape
[ OK ] FilterLayerTest/2.TestReshape (621 ms)
[ RUN ] FilterLayerTest/2.TestGradient
[ OK ] FilterLayerTest/2.TestGradient (35558 ms)
[ RUN ] FilterLayerTest/2.TestForward
[ OK ] FilterLayerTest/2.TestForward (4 ms)
[----------] 3 tests from FilterLayerTest/2 (36184 ms total)

[----------] 5 tests from MemoryDataLayerTest/2, where TypeParam = caffe::GPUDevice
[ RUN ] MemoryDataLayerTest/2.AddDatumVectorDefaultTransform
[ OK ] MemoryDataLayerTest/2.AddDatumVectorDefaultTransform (5 ms)
[ RUN ] MemoryDataLayerTest/2.TestSetBatchSize
[ OK ] MemoryDataLayerTest/2.TestSetBatchSize (12 ms)
[ RUN ] MemoryDataLayerTest/2.TestSetup
[ OK ] MemoryDataLayerTest/2.TestSetup (2 ms)
[ RUN ] MemoryDataLayerTest/2.AddMatVectorDefaultTransform
[ OK ] MemoryDataLayerTest/2.AddMatVectorDefaultTransform (5 ms)
[ RUN ] MemoryDataLayerTest/2.TestForward
[ OK ] MemoryDataLayerTest/2.TestForward (61 ms)
[----------] 5 tests from MemoryDataLayerTest/2 (86 ms total)

[----------] 4 tests from ContrastiveLossLayerTest/3, where TypeParam = caffe::GPUDevice
[ RUN ] ContrastiveLossLayerTest/3.TestGradient
F0528 15:20:21.323849 1124 contrastive_loss_layer.cu:102] Check failed: error == cudaSuccess (8 vs. 0) invalid device function
*** Check failure stack trace: ***
@ 0x7f7a510718 google::LogMessage::Fail()
@ 0x7f7a512614 google::LogMessage::SendToLog()
@ 0x7f7a510290 google::LogMessage::Flush()
@ 0x7f7a512eb4 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f78fd3d70 caffe::ContrastiveLossLayer<>::Backward_gpu()
@ 0x5259d8 caffe::Layer<>::Backward()
@ 0x526ff4 caffe::GradientChecker<>::CheckGradientSingle()
@ 0x52b0b4 caffe::GradientChecker<>::CheckGradientExhaustive()
@ 0x7d194c caffe::ContrastiveLossLayerTest_TestGradient_Test<>::TestBody()
@ 0x959684 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x95292c testing::Test::Run()
@ 0x952a68 testing::TestInfo::Run()
@ 0x952b28 testing::TestCase::Run()
@ 0x953c88 testing::internal::UnitTestImpl::RunAllTests()
@ 0x953f9c testing::UnitTest::Run()
@ 0x5195e8 main
@ 0x7f78a688a0 __libc_start_main
Aborted
Makefile:532: recipe for target ‘runtest’ failed
make: *** [runtest] Error 134
ubuntu@tegra-ubuntu:~/caffe$

i have add the 53 in Makefile.config,but it still failed.can you give me a fever

2

Hi,

Thanks for your question.

We also found this issue before.
In our experiment, caffe can run properly although it hit some error in unit-test.

It’s hard for us to make sure all the three-party library run successfully in our environment.
It’s recommended to use tensorRT which always be verified with our latest OS system and can be installed via JetPack directly.

TensorRT can import caffe model and achieve better performance.
Thanks.

3

Thanks for your help.now, i wan to
./cascadeclassifier --cascade ~/opencv-3.1.0/data/haarcascades/haarcascade_frontalface_alt.xml --camera 1
but

ubuntu@tegra-ubuntu:~/opencv-3.1.0/samples/gpu$ ./cascadeclassifier --cascade ~/opencv-3.1.0/data/haarcascades/haarcascade_frontalface_alt.xml --camera 0
Device 0: “NVIDIA Tegra X1” 3995Mb, sm_53, Driver/Runtime ver.8.0/8.0
OpenCV Error: Unspecified error (The node does not represent a user object (unknown type?)) in cvRead, file /home/ubuntu/opencv-3.1.0/modules/core/src/persistence.cpp, line 4976
terminate called after throwing an instance of ‘cv::Exception’
what(): /home/ubuntu/opencv-3.1.0/modules/core/src/persistence.cpp:4976: error: (-2) The node does not represent a user object (unknown type?) in function cvRead

Aborted

i’m very appreciate for your help.

4

Hi,

Thanks for your feedback.
We will check #3 and update information to topic 1011118:
[url]https://devtalk.nvidia.com/default/topic/1011118/jetson-tx1/error-a-case-of-face-recognition-program-to-test-the-camera/post/5158934/#5158934[/url]

5

I am trying to install caffe from newly released Jetpack 3.2. when I run runtest I get

3 tests, listed below:
[ FAILED ] DetectNetTransformationLayerTest/1.TestAllAugmentation, where TypeParam = caffe::CPUDevice
[ FAILED ] DetectNetTransformationLayerTest/1.TestHueRotation, where TypeParam = caffe::CPUDevice
[ FAILED ] DetectNetTransformationLayerTest/1.TestDesaturation, where TypeParam = caffe::CPUDevice

my echo $LD_LIBRARY_PATH
/usr/local/cuda-9.0/lib64:

Any ideas? I am using Ubuntu 16.04

6

Hi,

Although there are some failed tests, we didn’t find any issue when inferencing with Caffe frameworks on Jetson.

From your log, this issue is related to OpenMP. Maybe you can get more information from Caffe developer.
https://github.com/BVLC/caffe/issues/4083

Thanks.