Nvc/nvc++ hang compiling C code, nvcc/gcc & others succeed

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

We have extracted some C code that seems to be hanging nvc & nvc++. Interestingly, nvcc succeeds, as do gcc, Intel icx, and Cray cc.

To replicate:

git clone https://github.com/benkirk/bugreports.git
cd bugreports/RC-15991
./run.sh
# or
nvc -v -I. -c -g ./odf_init.c

What’s very interesting is that nvcc works, while the other nvhpc components do not. I’ve tested this with versions through 22.9.

Any help or advice is appreciated!!

-Ben

2

Thanks Ben!

I was able to reproduce the issue here and filed a problem report, TPR #32742.

The problem is this line:

 /* new structure, 22/03/2020 */
 const ODF_LUT odf_lut[] ={
#include "odf_lut.h"
};

“odf_lut.h” contains over 11,000 elements which seems to cause problems. I reduced this down to 500 elements, and then it only took a few seconds. Best guess is the compiler isn’t actually hung, just taking a realllllly long time to process. Hopefully engineering can find a way to speed this up.

What’s very interesting is that nvcc works, while the other nvhpc components do not.

By default, nvcc uses g++ as the host compiler so will match the behavior to gcc.

-Mat

3

Thanks for the quick reply!

Sorry for the noob question, but is TPR #32742 something I can see & monitor externally?

-Ben

4

“TPR” stands for Technical Problem Report. It’s the bug tracking system that we’ve been using for about 30 years, since our PGI days. When NVIDIA bought us, our team kept it given the history and ease of use. The one drawback is that it’s not externally visible.

NVIDIA does have NVBUG which you can use and that does have a way to monitor your bugs. Though you’d you need to submit it via your NVIDIA Developer account page (i.e. I can’t submit the bug for you). We get these as well so which ever works best for you is fine.

Though I’m happy to look up a status of a TPR, just send me a direct message or respond to this post. Also, I do post a notification once a TPR has been fixed in a release.

5

Regarding your “taking a really long time” suggestion:

Yes, the compiler eventually completes in 18 minutes on my machine, using default or -O0 optimization levels.

With -O3, its still running after 12+ hours.

Which begs a question, what’s the proper #pragma for nvc to disable optimization in a code block, should:

#pragma opt=0

 const ODF_LUT odf_lut[] ={
#include "odf_lut.h"
};

do the right thing, or something else??

(I have tried the above, and it doesn’t return in the 18 minutes that the -O0 version does, so I don’t think it is exactly what I am looking for.)

Regards,

-Ben

6

Checking back on this issue, do you have any guidance on #pragmas that can be used in a translation unit to force -O0?

I’d like to extract the troubling array initialization into its own file and force it to compile with -O0, if possible.

-Ben

7

Hi Ben,

It would be “#pragma opt 0”, i.e. no “=”. Placement is also key in that you want to put the pragma inside the function so it has “routine” scope so the lower opt level only applies to this one routine and not the whole file.

However, I’m not sure it’s working in this case, but that may only because I wasn’t being patient enough.

If you move this to its own file, then no need to use the pragma, just set “-O0”.

-Mat

8

Thanks a lot Mat!

As a temporary workaround , this seems to mostly do the trick:

...

const LINE_ODF_prelim *get_odf_src()
{
#ifdef __NVCOMPILER
#pragma opt 0
#endif

  static const LINE_ODF_prelim odf_src[]={
#include "odf_hitran.h"
  };

  return odf_src;
}


const ODF_LUT *get_odf_lut()
{
#ifdef __NVCOMPILER
#pragma opt 0
#endif

  static  const ODF_LUT odf_lut[] ={
#include "odf_lut.h"
  };

  return odf_lut;
}


int init_odf (double            **Qrot,
              ATMOSPHERE        *atmos,
              VIBLEVEL          *v_level,
              MOLECULE          *mol,
              PARAMETERINFO     *pars,
              BAND_ODF          *band_odf)

{

 const LINE_ODF_prelim *odf_src = get_odf_src();
 const ODF_LUT *odf_lut = get_odf_lut();

 LINE_ODF       *line_odf;
 ...

-Ben

9

Hi, I was curious if there has been any progress on this issue in TPR #32742, and perhaps if there is a target resolution date?

23.1 is the latest iterate I have tested, and confirmed the issue is still present.

Thanks!

10

Hi Ben,

Looks like this was given a low priority and not yet assigned to an engineer. Likely since it’s a narrow use case and that you have a work around.

Though given it now been 6 months since you reported it, I’ll ping the compiler engineering manager and see if we can bump the priority.

-Mat

11

Thanks Mat, I appreciate the update & effort.

While I do have a technical workaround, not all our users can readily adopt it in their workflows, so this is prohibiting our uptake of NVHPC for some codes.

-Ben