Nvc++ and stdexec leading to a segfault in my environment but can't pinpoint why

danialjavady96 · August 9, 2023, 3:51pm

Using WSL2, Ubuntu 22.04, nvc++ 23.1 and/or 23.3 with a 3080, I am running into a segfault that doesn’t show up on the pipeline.

github.com

NVIDIA/stdexec/blob/main/examples/nvexec/reduce.cpp

/*
 * Copyright (c) 2022 NVIDIA Corporation
 *
 * Licensed under the Apache License Version 2.0 with LLVM Exceptions
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 *   https://llvm.org/LICENSE.txt
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

#include <nvexec/stream_context.cuh>
#include <stdexec/execution.hpp>

#include <thrust/device_vector.h>

This file has been truncated. show original

This code, runs just fine on the pipelines V100, but does not on my 3080 with WSL. It seems, that it doesn’t even hit the GPU before it segfaults and I’m lost as to why.

github.com

NVIDIA/libcudacxx/blob/a57dbed580e49b14ac1ad7f98496176407208aa4/include/cuda/std/detail/libcxx/include/__functional/invoke.h#L405


      
              { return          (*static_cast<_A0&&>(__a0)).*__f; }
          
          // bullet 7
          
          _LIBCUDACXX_DISABLE_EXEC_CHECK
          template <class _Fp, class ..._Args>
          inline _LIBCUDACXX_INLINE_VISIBILITY
          _LIBCUDACXX_CONSTEXPR decltype(_CUDA_VSTD::declval<_Fp>()(_CUDA_VSTD::declval<_Args>()...))
          __invoke(_Fp&& __f, _Args&& ...__args)
              _NOEXCEPT_(noexcept(static_cast<_Fp&&>(__f)(static_cast<_Args&&>(__args)...)))
              { return          static_cast<_Fp&&>(__f)(static_cast<_Args&&>(__args)...); }
          
          // __invokable
          template <class _Ret, class _Fp, class ..._Args>
          struct __invokable_r
          {
            template <class _XFp, class ..._XArgs>
            _LIBCUDACXX_INLINE_VISIBILITY
            static decltype(_CUDA_VSTD::__invoke(_CUDA_VSTD::declval<_XFp>(), _CUDA_VSTD::declval<_XArgs>()...)) __try_call(int);
          
            template <class _XFp, class ..._XArgs>

Segfault occurs here, and the problem would seem to be that the code is trying to read memory from a address that doesn’t have read permissions enabled.

I don’t have enough experience with nvc++ to navigate what’s going on here… I feel like WSL might be a problem here but I don’t know. Any ideas? I’m using the exact same Cuda toolkit(12.0), exact same nvc++ version (23.1 and 23.3 both produce this problem).

danialjavady96 · August 9, 2023, 6:09pm

Turns out unified memory isn’t supported on WSL which is causing this issue. Setting

the env var export CUDA_LAUNCH_BLOCKING=1 has no effect as well, unfortunately. I could try using VMware with a ubuntu OS over WSL I suppose.

edit:

Turns out I needed to just do CUDA_LAUNCH_BLOCKING ./executable and it works.

MatColgrove · August 9, 2023, 11:57pm

Excellent, I’m glad you were able to determine the issue.

-Mat

ericniebler · August 10, 2023, 6:00pm

For others reading this, the correct incantation is:

CUDA_LAUNCH_BLOCKING=1 ./executable