I downloaded TensorFlow code that works fine in 2D, and then extended it to 3D. However, I randomly get segmentation faults that seem to point to libcuda. Sometimes the training runs for 5-6 hours and then ends with a segmentation fault, sometimes the segmentation fault comes after 2 minutes. Any ideas what the problem can be?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Segmentation fault when running TF detection tutorial | 2 | 2000 | December 16, 2019 | |
| Segmentation fault at training network | 6 | 2749 | September 5, 2021 | |
| Segmentation faults and illegal memory address accesses when running Tensorflow code | 5 | 1750 | February 11, 2021 | |
| New Build, Crashes Repeatedly on even light CUDA Use (TensorFlow) | 0 | 440 | May 3, 2017 | |
| Segmentation fault on the simplest example | 0 | 816 | October 25, 2020 | |
| Random freezes and CUDA errors | 1 | 1112 | August 27, 2020 | |
| Segmentation fault | 0 | 973 | July 20, 2022 | |
| Recursively training a network crashes with "Segmentation Fault (Core Dumped)" | 7 | 2081 | October 18, 2021 | |
| Intermittent CUDA_ERROR_ILLEGAL_ADDRESS error on Ubuntu 18.04 with TensorFlow 2.2.0 | 3 | 8071 | January 5, 2023 | |
| run tensorflow 1.3 on tx2 stuck | 20 | 5857 | October 18, 2021 |