Possible NVCC Warp-Aggregated Atomic Issue

[zilkhanir1]

Possible NVCC Warp-Aggregated Atomic Issue on Ampere (CUDA 11.4) — atomicAdd(+2) Produces Unexpected Results

Environment

• OS: Windows
• CUDA Toolkit: 11.4
• GPU architecture: sm_86 (Ampere)
• Build system: CMake / Visual Studio
• Kernel launch: 691,200 blocks × 64 threads (1D grid)
• Kernel contains substantial branching, nested loops, and divergent control flow
• Counter type: uint32_t

Problem description

I am investigating unexpected results from atomicAdd() in a large CUDA kernel.

The kernel processes 20 subframes, each having its own global-memory counter.

The counter is initialized to zero before launching the kernel, with explicit synchronization before and after execution.

The relevant code is:

uint32_t idx = atomicAdd(

&NumOfDetSpots\[sfInx\],

2u

);

For my test data, only one thread per subframe is expected to execute this operation.

Therefore, the expected result for each subframe is:

• idx = 0
• NumOfDetSpots[sfInx] = 2

However, with the optimized kernel, I consistently observe counters equal to 4, and in earlier tests sometimes 6. The returned idx can also be unexpectedly nonzero.

The behavior is highly sensitive to seemingly unrelated source-code modifications, but is generally reproducible for a given compiled binary.

Debugging performed

1. Initialization and synchronization

I explicitly reset the counters using cudaMemsetAsync() and synchronize before launching the kernel.

After the kernel, I synchronize and copy the counters to host memory.

I also tested directly copying the counter values rather than copying the entire containing structure. The unexpected results persisted.

2. Separate debug counter

I added an additional atomic counter immediately before the original atomic:

atomicAdd(&debugCount[sfInx], 1u);

idx = atomicAdd(&NumOfDetSpots[sfInx], 2u);

Both arrays are initialized to zero.

Observed results for all 20 subframes:

debugCount[sfInx] = 1

NumOfDetSpots[sfInx] = 4

This suggests that only one execution of the debug atomic is being counted, while the second atomic produces an unexpected total.

3. Thread identity and warp participation

I printed the subframe index, global thread index, lane number, and active mask at the atomic location.

The output contains 20 lines, one per subframe.

Example:

sf=0 tid=9599 lane=31 mask=80000000

sf=1 tid=42367 lane=31 mask=80000000

sf=2 tid=75135 lane=31 mask=80000000

sf=3 tid=107903 lane=31 mask=80000000

The global thread indices differ by 32,768 between consecutive subframes.

All observed threads are in separate warps and report only lane 31 active at the diagnostic point.

I understand that __activemask() does not necessarily represent the exact participation mask used internally by compiler-generated warp aggregation.

4. Changing the atomic increment

I found that:
Implementation
Result
Inline atomicAdd(…, 1)
Correct
Inline atomicAdd(…, 2)
Incorrect
Inline atomicAdd(…, increment) with runtime parameter equal to 2
Correct in tested builds
Non-inlined device wrapper with increment 2
Correct
Inline atomic with CUDA device debugging (-G)
Correct

5. Non-inlined wrapper fixes the issue

I implemented:

_device_ _noinline_

uint32_t MyAtomicAdd(uint32_t* ptr, uint32_t value)

{

return atomicAdd(ptr, value);

}

And changed the original call to:

idx = MyAtomicAdd(&NumOfDetSpots[sfInx], 2u);

With this version, all 20 counters consistently produce the expected values:

NumOfDetSpots[sfInx] = 2

idx = 0

6. SASS comparison

I compared the generated SASS for the working and failing versions using cuobjdump.

The optimized inline atomic uses instructions consistent with compiler-generated warp aggregation:

VOTEU.ALL UR4, UPT, PT;

FLO.U32 R4, UR4;

POPC R0, UR4;

ISETP.EQ.U32.AND P0, PT, R4, R3, PT;

IMAD.SHL.U32 R3, R0, 0x2, RZ;

@P0 ATOMG.E.ADD.STRONG.GPU PT, R3, [R62.64], R3;

The population count is scaled by 2 before the atomic.

The working _noinline_ implementation instead generates a direct atomic instruction without the surrounding warp-aggregation sequence:

ATOMG.E.ADD.STRONG.GPU PT, R4, [R4.64], R6;

The larger kernel also contains instructions such as SHFL.IDX and POPC following the aggregated atomic, suggesting reconstruction of per-thread values.

The compiler generates substantially different SASS when changing the increment from 1 to 2.!!!

7. Sensitivity to control flow

The original kernel contains nested loops and divergent branching.

I found a particular loop where changing:

if (condition)

{

RiseLine = i;

break;

}

affected the final atomic result.

Replacing break with i = -1 still reproduced the failure, whereas removing early termination made the result correct in the tested build.

Other small changes, such as replacing:

BBInfo[sfInx][idx].SpotID = 1;

with:

BBInfo[sfInx][0].SpotID = 1;

also caused the failure to disappear in certain builds.

I recognize that these changes alter both program behavior and compiler code generation, so they are not by themselves proof of a compiler issue.

8. Compute Sanitizer

I ran Compute Sanitizer Memcheck, which reported zero errors in the tested configuration.

I understand that this does not exclude data races, uninitialized local variables, or other forms of undefined behavior.

9. Kernel reduction

I removed most of the original kernel logic while preserving the atomic operation.

The simplified kernel worked correctly.

When I progressively restored the original logic, the failure reappeared. The result was particularly sensitive to loop termination and branches.

Current hypothesis

The evidence suggests that the issue is associated with optimization of atomic operations in CUDA 11.4, potentially involving warp aggregation and divergent control flow.

The compiler appears to aggregate atomic additions using VOTEU.ALL, POPC, and related instructions.

However, I cannot rule out undefined behavior elsewhere in the large kernel or a race that becomes visible only under certain optimization conditions.

The fact that _noinline_ and -G both eliminate the failure makes the code-generation differences particularly interesting.

Questions

1. Is this pattern of warp-aggregated atomic generation expected in CUDA 11.4 for atomicAdd(…, 2) when the returned old value is used?
2. Are there any known CUDA 11.4 / sm_86 issues involving warp aggregation, divergent loops, or incorrect reconstruction of atomic return values?
3. Is there a compiler option or supported technique to disable automatic warp aggregation for one atomic operation without disabling optimization for the entire kernel?
4. Is using a _noinline_ device wrapper a reasonable temporary workaround?
5. What additional diagnostics would help distinguish an NVCC optimization issue from undefined behavior elsewhere in the kernel?

Next planned experiment

I plan to test the same code on an NVIDIA L40S GPU running Linux with CUDA 12.8, using equivalent inputs and optimization settings.

I can provide SASS comparisons, compiler output, and additional diagnostics if useful.

Any guidance from NVIDIA engineers would be appreciated.

[Nanodeoclus]

How do you select the thread that is supposed to perform the atomic add? If the compiler implements warp aggregation, that would indicate it not being able to deduce that a single lane is meant perform the operation.
Have you tried using PTX %laneid and/or elect.sync to select one lane?

Given the warp aggregation SASS, it seems the most likely cause is that, for some reason, two warp lanes are actually active at that point. How does the SASS look with the debugCount atomic add in place?

How do you use the idx value afterwards, is it broadcast to other lanes?

[zilkhanir1]

Thank you for the suggestions.

The thread reaching the atomic is selected through the algorithm’s conditions, rather than an explicit lane-ID selection.

i have a boolean vector. Valid elements enter the logic others just skip and return

So currently only 20 elements are valid (in fixed but arbitrary positions)

Each element uses atomic add on a different address

I instrumented the kernel at the atomic location and observed one thread per subframe (20 subframes total). The global thread indices are separated by 32768, so the observed threads belong to different warps.

All 20 reported lane 31 with an __activemask() value of 0x80000000.

I also added a separate debug atomic immediately before the problematic one:

atomicAdd(&debugCount[sfInx], 1u);

idx = atomicAdd(&NumOfDetSpots[sfInx], 2u);

With both arrays initialized to zero, I consistently observe:

debugCount[sfInx] = 1

NumOfDetSpots[sfInx] = 4

I understand that the active mask recorded at the source location may not be identical to the participation mask in the compiler-generated warp-aggregation sequence.

I have not yet inspected the SASS with both debug atomics present. I will do that next and compare their predicates, participating-lane counts, and destination-address calculations.

Regarding idx, I use the returned old value in the same thread, including for bounds checks and indexing:

if (idx < MaxNumOfTFClusters)

{

BBInfo\[sfInx\]\[idx\].SpotID = 1;

}

There is no explicit source-level warp shuffle or broadcast of idx.

I also confirmed that the original optimized kernel works correctly on CUDA 11.8/Linux/A30 and CUDA 12.8/Linux/L40S, while CUDA 11.4/Windows/sm_86 fails. In CUDA 11.4, both -G and a _noinline_ device wrapper around atomicAdd eliminate the failure.

I will also test explicit lane selection as suggested.

[njuffa]

Proper markup for all code should be used when posting here, otherwise it is much harder to follow a lengthy post. Surround individual items with backticks: `this is a code snippet` renders as this is a code snippet . Surround code blocks with a sequence of three back ticks:
```
code block line 1
code block line 2
```
renders as:

code block line 1
code block line 2

One can also use the </> button at the top of the input mask to generate the markup, by highlighting the relevant text and clicking this button.

Code generation issues are possible but rare at the current maturity level of the CUDA compiler. The best way to assess this possibility is to study the smallest possible example code that still demonstrates the issue and that others can compile and run for themselves. This is often called a “minimal reproducer”. I understand that it can take time to cut down real-life code to a minimal example; in my personal experience I have spent up to an hour doing that.

Generally speaking, when filing compiler bug reports with NVIDIA, they are likely to ask for a minimal reproducer, because a bug report can only proceed to the analysis stage after in-house reproduction has been achieved. The next thing they would likely ask is whether the issue reproduces with the latest shipping toolchain, which I think is version 13.3 or 13.4 at this time. For obvious reasons they are reluctant to spend time diagnosing issues in older versions a couple of generations back. CUDA 11.4 was released five years ago.

Is there a strong reason for not switching to the latest toolchain? I understand that such reasons may exist due to corporate or regulatory restrictions on changing toolchain versions, or for the simple reason that the hardware one uses is no longer supported by the newer toolchain. If there is no strong reason, I would suggest upgrading to the latest version, or at least the newest version possible.

The description of a marked change to the generated SASS (machine code) based on whether the increment of the atomicAdd is 1 or 2 is certainly curious but not necessarily indicative of a bug.

The post states that compute-sanitizer was used to check the code and it came up clean. compute-sanitizer is a collection of multiple tools, a correctness checking suite. The description of the observations suggests the possible presence of a race condition in the code, so I wonder whether computer-sanitizer --tool racecheck was run. Note that there are also various options to control the race check tool.

The nature of race conditions is that they can remain hidden or become observable due to change in the executed machine code, so the possibility of different observations with different compiler versions is certainly there. In other words, a change in toolchain may expose a latent bug in the code.

[zilkhanir1]

Thank you for the detailed response and suggestions.

I am planning to switch to CUDA 12.8, since the original kernel works correctly with that version.

However, I fully agree that the fact that the issue does not reproduce with CUDA 12.8 does not necessarily mean the underlying problem is gone. There could still be a latent race condition or undefined behavior that is simply not exposed by the newer compiler’s code generation.

What I still find particularly puzzling is that the issue was initially triggered by changing only the hardcoded increment in atomicAdd from 1 to 2. This single change resulted in substantially different generated SASS and unexpected behavior, while using a _noinline_ wrapper or disabling device optimization eliminated the problem.

I understand that this does not necessarily indicate a compiler bug, but I believe it is an interesting observation worth investigating.

In any case, I will continue working on a minimal reproducible example to help determine whether the issue originates from the compiler or from something in my code.

Thank you again for your help and insights.

Note: This response was drafted with the assistance of ChatGPT, based on my own debugging experiments, observations, and test results.

[Curefab]

For the debug output: Instead of the activemask function, you can use __ballot_sync outside of the conditional (if) block with the same condition as used for the if.
As it has a _sync suffix the results are more reliable. (compared to activemask, which possibly shows less lanes than are actually active)