GPU does not work when running SSD

Hi,

Could you try the memory check sample below:
https://devtalk.nvidia.com/default/topic/1013464/jetson-tx2/gpu-out-of-memory-when-the-total-ram-usage-is-2-8g/post/5170376/#5170376
In case some memory issue cause GPU unlimited waiting.

By the way, we want to confirm; you use the official version TX2, not a custom one, right?
Thanks.

Hi,

the sample I tried

#include <stdio.h>
#include "cuda.h"
#define ONE_MBYTE (1024*1024)

void printMemInfo()
{
    size_t free_byte ;
    size_t total_byte ;
    cudaError_t cuda_status = cudaMemGetInfo( &free_byte, &total_byte ) ;

    if ( cudaSuccess != cuda_status ){
        printf("Error: cudaMemGetInfo fails, %s\n", cudaGetErrorString(cuda_status));
        exit(1);
    }

    double free_db = (double)free_byte ;
    double total_db = (double)total_byte ;
    double used_db = total_db - free_db ;

    printf("GPU memory usage: used = %.2f MB, free = %.2f MB, total = %.2f MB\n", used_db/ONE_MBYTE, free_db/ONE_MBYTE, total_db/ONE_MBYTE);
}

int main(){
    void *p;

    int amount = 0;
    while(true){
        cudaError_t rval = cudaMalloc( &p, long(amount)*ONE_MBYTE);
        printf( "cudaAlloc( ..., %dMByte, ... ) returns %d\n", amount, rval );
        printMemInfo();

        if( rval != cudaSuccess ) break;
        amount += 100;
        cudaFree(p);
    }
    return 0;
}

And the result of the sample

nvidia@tegra-ubuntu:~/tom$ nvcc test3.cu -o test3
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
nvidia@tegra-ubuntu:~/tom$ ./test3
cudaAlloc( ..., 0MByte, ... ) returns 0
GPU memory usage: used = 1774.04 MB, free = 6080.02 MB, total = 7854.06 MB
cudaAlloc( ..., 100MByte, ... ) returns 0
GPU memory usage: used = 1864.39 MB, free = 5989.66 MB, total = 7854.06 MB
cudaAlloc( ..., 200MByte, ... ) returns 0
GPU memory usage: used = 2075.83 MB, free = 5778.23 MB, total = 7854.06 MB
cudaAlloc( ..., 300MByte, ... ) returns 0
GPU memory usage: used = 2275.57 MB, free = 5578.48 MB, total = 7854.06 MB
cudaAlloc( ..., 400MByte, ... ) returns 0
GPU memory usage: used = 2475.07 MB, free = 5378.99 MB, total = 7854.06 MB
cudaAlloc( ..., 500MByte, ... ) returns 0
GPU memory usage: used = 2762.29 MB, free = 5091.77 MB, total = 7854.06 MB
cudaAlloc( ..., 600MByte, ... ) returns 0
GPU memory usage: used = 3032.36 MB, free = 4821.70 MB, total = 7854.06 MB
cudaAlloc( ..., 700MByte, ... ) returns 0
GPU memory usage: used = 3072.31 MB, free = 4781.75 MB, total = 7854.06 MB
cudaAlloc( ..., 800MByte, ... ) returns 0
GPU memory usage: used = 3160.82 MB, free = 4693.24 MB, total = 7854.06 MB
cudaAlloc( ..., 900MByte, ... ) returns 0
GPU memory usage: used = 2939.66 MB, free = 4914.39 MB, total = 7854.06 MB
cudaAlloc( ..., 1000MByte, ... ) returns 0
GPU memory usage: used = 3411.85 MB, free = 4442.21 MB, total = 7854.06 MB
cudaAlloc( ..., 1100MByte, ... ) returns 0
GPU memory usage: used = 3027.45 MB, free = 4826.61 MB, total = 7854.06 MB
cudaAlloc( ..., 1200MByte, ... ) returns 0
GPU memory usage: used = 3719.27 MB, free = 4134.79 MB, total = 7854.06 MB
cudaAlloc( ..., 1300MByte, ... ) returns 0
GPU memory usage: used = 3067.76 MB, free = 4786.30 MB, total = 7854.06 MB
cudaAlloc( ..., 1400MByte, ... ) returns 0
GPU memory usage: used = 4040.91 MB, free = 3813.15 MB, total = 7854.06 MB
cudaAlloc( ..., 1500MByte, ... ) returns 0
GPU memory usage: used = 3268.38 MB, free = 4585.68 MB, total = 7854.06 MB
cudaAlloc( ..., 1600MByte, ... ) returns 0
GPU memory usage: used = 4288.99 MB, free = 3565.07 MB, total = 7854.06 MB
cudaAlloc( ..., 1700MByte, ... ) returns 0
GPU memory usage: used = 3469.96 MB, free = 4384.09 MB, total = 7854.06 MB
cudaAlloc( ..., 1800MByte, ... ) returns 0
GPU memory usage: used = 4487.06 MB, free = 3367.00 MB, total = 7854.06 MB
cudaAlloc( ..., 1900MByte, ... ) returns 0
GPU memory usage: used = 3670.36 MB, free = 4183.70 MB, total = 7854.06 MB
cudaAlloc( ..., 2000MByte, ... ) returns 0
GPU memory usage: used = 4624.62 MB, free = 3229.44 MB, total = 7854.06 MB
cudaAlloc( ..., 2100MByte, ... ) returns 0
GPU memory usage: used = 3868.75 MB, free = 3985.31 MB, total = 7854.06 MB
cudaAlloc( ..., 2200MByte, ... ) returns 0
GPU memory usage: used = 4865.00 MB, free = 2989.06 MB, total = 7854.06 MB
cudaAlloc( ..., 2300MByte, ... ) returns 0
GPU memory usage: used = 4068.54 MB, free = 3785.52 MB, total = 7854.06 MB
cudaAlloc( ..., 2400MByte, ... ) returns 0
GPU memory usage: used = 5047.34 MB, free = 2806.72 MB, total = 7854.06 MB
cudaAlloc( ..., 2500MByte, ... ) returns 0
GPU memory usage: used = 4279.55 MB, free = 3574.51 MB, total = 7854.06 MB
cudaAlloc( ..., 2600MByte, ... ) returns 0
GPU memory usage: used = 5098.90 MB, free = 2755.16 MB, total = 7854.06 MB
cudaAlloc( ..., 2700MByte, ... ) returns 0
GPU memory usage: used = 4471.36 MB, free = 3382.70 MB, total = 7854.06 MB
cudaAlloc( ..., 2800MByte, ... ) returns 0
GPU memory usage: used = 5419.17 MB, free = 2434.89 MB, total = 7854.06 MB
cudaAlloc( ..., 2900MByte, ... ) returns 0
GPU memory usage: used = 4671.50 MB, free = 3182.56 MB, total = 7854.06 MB
cudaAlloc( ..., 3000MByte, ... ) returns 0
GPU memory usage: used = 5577.15 MB, free = 2276.91 MB, total = 7854.06 MB
cudaAlloc( ..., 3100MByte, ... ) returns 0
GPU memory usage: used = 4873.66 MB, free = 2980.39 MB, total = 7854.06 MB
cudaAlloc( ..., 3200MByte, ... ) returns 0
GPU memory usage: used = 5759.48 MB, free = 2094.58 MB, total = 7854.06 MB
cudaAlloc( ..., 3300MByte, ... ) returns 0
GPU memory usage: used = 5080.70 MB, free = 2773.36 MB, total = 7854.06 MB
cudaAlloc( ..., 3400MByte, ... ) returns 0
GPU memory usage: used = 5842.10 MB, free = 2011.96 MB, total = 7854.06 MB
cudaAlloc( ..., 3500MByte, ... ) returns 0
GPU memory usage: used = 5271.21 MB, free = 2582.84 MB, total = 7854.06 MB
cudaAlloc( ..., 3600MByte, ... ) returns 0
GPU memory usage: used = 5716.90 MB, free = 2137.16 MB, total = 7854.06 MB
cudaAlloc( ..., 3700MByte, ... ) returns 0
GPU memory usage: used = 5874.41 MB, free = 1979.64 MB, total = 7854.06 MB
cudaAlloc( ..., 3800MByte, ... ) returns 0
GPU memory usage: used = 5832.64 MB, free = 2021.42 MB, total = 7854.06 MB
cudaAlloc( ..., 3900MByte, ... ) returns 2
GPU memory usage: used = 2758.22 MB, free = 5095.84 MB, total = 7854.06 MB
nvidia@tegra-ubuntu:~/tom$

I use the official version TX2, and SN: 945-82771-0000-000 U.

Hi,

We are discussing this issue internally. Will update information to you later.

Hi,

Thanks for your patience.
After discussing, we still suspect software more since hardware behaviour is under expectation.

We tried SSD and found sometime the GPU utilization also drop to 0%.
Guess this is related to rendering and display latency.

Could we know which sample you try to run, and the display device of your environment?
For example, a HDMI display directly connected to TX2 or display results via SSH?

Thanks.

Hi,

The sample I used has been provied.(https://devtalk.nvidia.com/default/topic/1019020/jetson-tx2/gpu-does-not-work-when-running-ssd/1 at the #11)

And I use a HDMI display directly connected to TX2.

So, I don’t think the problem has anything to do with the display latency.

Hi,

Sorry, we are still no idea about this.
But we keep co-working with the different team to narrow down the issue. Will update information to you later.

We also want to know which sample you want to run, for example, ssd_pascal_video.py. This will require video encoder.

Thanks.

Hi,

Please go through RMA process, see link: https://developer.nvidia.com/embedded/support

Here’s the process to get an RMA via NVIDIA Support:

  1. Go to NVIDIA Support
  2. Select “Live Chat” or “Ask a Question” from the options near the top of the page.
  3. Enter your personal information.
  4. Select the appropriate product from the drop-down list (“Tegra” in this case).
  5. Submit the request.
    If the Service agent is unable to resolve the technical issue, an RMA (Return Material Authorization) will be initiated.
    =================================================

Thanks.