Aneurysm do not work

Physics Simulation PhysicsNeMo-ML Model Framework Technical Support (PhysicsNeMo Only)
1

Hi!

With a version update, the aneurysm module stopped working.
problem.txt (14.7 KB)

In the old version everything was fine, but to guarantee I will load the module with the installation that I use in google colab

!pip3 install matplotlib transforms3d future typing numpy quadpy\
 numpy-stl==2.11.2 h5py sympy==1.5.1 termcolor\
 psutil symengine==0.6.1 numba Cython chaospy
!pip3 install -U https://github.com/paulo-herrera/PyEVTK/archive/v1.1.2.tar.gz
!pip install tensorflow-gpu==1.15
!python -m pip install horovod==0.21.0

fold_path = 'gdrive/MyDrive/simnet'
%cd {fold_path}

!tar -xvzf ./SimNet_source_v21.06.tar.gz
%cd ./SimNet/
!python setup.py install
%cd ..

%env LD_LIBRARY_PATH=/content/gdrive/My Drive/simnet/SimNet/
external/pysdf/build/:/usr/lib64-nvidia

%cd ./SimNet/external/pysdf/
!python setup.py install
%cd ../../..

!tar -xvzf ./SimNet_examples_v21.06.tar.gz
2

Hello, can you send your driver version? In this version we had to upgrade the optix version and because of this it needs a driver update to 465. That might be your issue.

3

Hey,

I’m encountering a similar issue with optix.

I’ve managed to get SimNet working on my local machine with a BareMetal installation through anaconda. I’m running driver 460.56, with this setup the optix and the aneurysm case work just fine.

bash-4.2$ nvidia-smi
Thu Oct  7 10:39:54 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.56       Driver Version: 460.56       CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro M4000        Off  | 00000000:03:00.0  On |                  N/A |
| 46%   39C    P8    12W / 120W |    130MiB /  8126MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

Repeating the exact same installation steps on on a hpc cluster then running $python aneurysm.py I get a similar error to @bim159.

[OptiX][ 4][       KNOBS]: All knobs on default.

[OptiX][ 4][  DISK CACHE]: Opened database: "/var/tmp/OptixCache_myUsrNm/cache7.db"
[OptiX][ 4][  DISK CACHE]:     Cache data size: "0 Bytes"
terminate called after throwing an instance of 'std::runtime_error'
  what():  the provided PTX was compiled with an unsupported toolchain.
Aborted

Checking the driver on the HPC:

(SimNet_BareMetal) [myUsrNm@hpc002 aneurysm]$ nvidia-smi
Thu Oct  7 02:37:51 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:15:00.0 Off |                    0 |
| N/A   33C    P0    63W / 300W |   9752MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:16:00.0 Off |                    0 |
| N/A   39C    P0    67W / 300W |   8524MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:3A:00.0 Off |                    0 |
| N/A   33C    P0    65W / 300W |   9598MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:3B:00.0 Off |                    0 |
| N/A   37C    P0    63W / 300W |   8282MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  Off  | 00000000:89:00.0 Off |                    0 |
| N/A   33C    P0    52W / 300W |   9392MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  Off  | 00000000:8A:00.0 Off |                    0 |
| N/A   39C    P0    52W / 300W |   9002MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  Off  | 00000000:B2:00.0 Off |                    0 |
| N/A   36C    P0    65W / 300W |   8396MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  Off  | 00000000:B3:00.0 Off |                    0 |
| N/A   39C    P0    65W / 300W |   7986MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

The HPC I’m using is a shared resource so it will be difficult to change the driver, do you have any other suggestions I could ask the cluster manager to implement?