op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceSelect::Flagged to copy indices out, status: too many resources requested for launch

Autonomous Machines Jetson & Embedded Systems Jetson TX2
#1

getting following error: on tx2. if it can be solved by inc the RAM how can I do it ?

2018-03-28 14:13:54.116758: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceSelect::Flagged to copy indices out, status: too many resources requested for launch
2018-03-28 14:13:54.117907: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceSelect::Flagged to copy indices out, status: too many resources requested for launch
[[Node: SecondStagePostprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/FilterGreaterThan/Where = Where_device="/job:localhost/replica:0/task:0/device:GPU:0"]]
2018-03-28 14:13:54.118051: W tensorflow/core/framework/op_kernel.cc:1192] Internal: WhereOp: Could not launch cub::DeviceSelect::Flagged to copy indices out, status: too many resources requested for launch
terminate called after throwing an instance of ‘std::out_of_range’
what(): vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

sudo ~/tegrastats output

RAM 5549/7851MB (lfb 212x4MB) SWAP 8/16384MB (cached 0MB) cpu
RAM 6498/7851MB (lfb 3x4MB) SWAP 29/16384MB (cached 1MB) cpu [54%@2032,76%@2034,95%@2034,49%@2030,61%@2029,60%@2031] EMC 26%@1866 APE 150 GR3D 7%@1300
RAM 6582/7851MB (lfb 2x4MB) SWAP 36/16384MB (cached 0MB) cpu [62%@2031,68%@2034,98%@2035,62%@2032,57%@2031,60%@2031] EMC 39%@1866 APE 150 GR3D 99%@1300
RAM 6581/7851MB (lfb 2x4MB) SWAP 36/16384MB (cached 0MB) cpu [7%@2033,100%@2035,98%@2035,11%@2032,10%@2031,7%@2031] EMC 43%@1866 APE 150 GR3D 99%@1300
RAM 5414/7851MB (lfb 192x4MB) SWAP 37/16384MB (cached 0MB) cpu [12%@2033,66%@2036,57%@2035,13%@2033,23%@2035,26%@2034] EMC 28%@1866 APE 150 GR3D 8%@1300
RAM 3575/7851MB (lfb 486x4MB) SWAP 37/16384MB (cached 0MB) cpu [5%@2022,1%@2035,9%@2036,3%@2034,6%@2034,15%@2033] EMC 16%@1866 APE 150 GR3D 0%@1300
RAM 3575/7851MB (lfb 486x4MB) SWAP 37/16384MB (cached 0MB) cpu [1%@2020,1%@2035,0%@2035,1%@2033,0%@2033,2%@2036] EMC 9%@1866 APE 150 GR3D 0%@1300

#2

Are you running in debug mode (–device-debug) ? If yes, you may try to turn it off.

#3

Thanks :) I did that change
But now I get this error

2018-03-28 15:03:49.500617: E tensorflow/stream_executor/cuda/cuda_driver.cc:1080] failed to synchronize the stop event: CUDA_ERROR_LAUNCH_FAILED
2018-03-28 15:03:49.500873: E tensorflow/stream_executor/cuda/cuda_timer.cc:54] Internal: error destroying CUDA event in context 0x1cc27a0: CUDA_ERROR_LAUNCH_FAILED
2018-03-28 15:03:49.500917: E tensorflow/stream_executor/cuda/cuda_timer.cc:59] Internal: error destroying CUDA event in context 0x1cc27a0: CUDA_ERROR_LAUNCH_FAILED
2018-03-28 15:03:49.501184: E tensorflow/stream_executor/cuda/cuda_dnn.cc:2279] failed to enqueue convolution on stream: CUDNN_STATUS_EXECUTION_FAILED
terminate called after throwing an instance of ‘std::out_of_range’
what(): vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

RAM 3190/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [45%@2026,92%@2035,80%@2034,42%@2031,43%@2024,54%@2033] EMC 27%@1866 APE 150 GR3D 10%@1300
RAM 3139/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [41%@2027,93%@2034,88%@2035,46%@2025,47%@2031,45%@2028] EMC 28%@1866 APE 150 GR3D 30%@1300
RAM 3289/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [40%@2033,89%@2035,69%@2034,46%@2030,52%@2026,60%@2023] EMC 28%@1866 APE 150 GR3D 22%@1300
RAM 3556/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [43%@2034,96%@2035,77%@2035,49%@2032,49%@2035,43%@2034] EMC 28%@1866 APE 150 GR3D 27%@1300
RAM 3739/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [57%@2026,81%@2035,69%@2034,49%@2029,49%@2030,50%@2027] EMC 28%@1866 APE 150 GR3D 30%@1300
RAM 3739/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [21%@2032,100%@2034,100%@2034,94%@2030,18%@2033,21%@2029] EMC 16%@1866 APE 150 GR3D 99%@1300
RAM 3740/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [4%@2030,100%@2036,100%@2034,100%@2031,7%@2031,2%@2028] EMC 9%@1866 APE 150 GR3D 99%@1300
RAM 2526/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [32%@2031,59%@2035,54%@2035,14%@2034,13%@2035,13%@2034] EMC 10%@1866 APE 150 GR3D 0%@1300
RAM 2516/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [2%@2017,0%@2034,0%@2035,2%@2035,1%@2034,1%@2034] EMC 6%@1866 APE 150 GR3D 0%@1300
RAM 2516/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [3%@2018,1%@2035,0%@2035,4%@2031,6%@2034,2%@2034] EMC 4%@1866 APE 150 GR3D 0%@1300
RAM 2516/7851MB (lfb 5x2MB) SWAP 708/16384MB (cached 10MB) cpu [2%@2033,0%@2035,0%@2034,1%@2030,3%@2033,2%@2033] EMC 3%@1866 APE 150 GR3D 0%@1300
RAM 2516/7851MB (lfb 5x2MB) SWAP 707/16384MB (cached 10MB) cpu [5%@2017,2%@2036,0%@2035,3%@2031,9%@2034,3%@2034] EMC 3%@1866 APE 150 GR3D 0%@1300

I am working on finding the cause and solution.

#4

Seems debug mode just gives different (earlier?) symptoms, but root cause is probably elsewhere.
Sorry I cannot say more. Someone else may help further.