Confusion about implicit inter-stream synchronization brought by cudaMemsetAsync

[Robert_Crovella]

@C.Yang

The implicit synchronization section is only about synchronous APIs, in other words, those who don’t follow stream semantic. cudaMemset, cudaMemcpy, cudaMalloc, etc.

I would certainly expect that any CUDA API call that has the potential to take a stream argument should be able to follow stream semantics, which does not necessarily involve synchronization or blocking of any type.

@C.Yang

Whether an op can start is subject to:

1. The op order maintained within a stream.
2. The implicit and explicit synchronization across streams.

If you are a beginner, my suggestion is that you don’t make use of the null stream, or any synchronizing API such as cudaMalloc, during any performance critical work-issuance area.

If we focus on stream-able operations, then whether an op can start is entirely determined by 2 rules of stream semantics:

1. Operations issued into the same stream will execute in issue order. Operation B, issued after operation A, will not begin until A has completed.

2. Operations issued into different streams have no defined order prescribed by CUDA. Without other information, it is possible that operation B, issued into stream X, may execute before, during, or after operation A, issued into stream Y.

If we layer on null-stream behavior/interaction, then the additional rule is as follows:

An operation issued into a default (ie. non-modified) null stream will not begin until all previous work issued to that device is complete. Furthermore, any operation issued after an operation issued into the null stream cannot begin until the null stream operation is complete. For a modified null stream, this additional rule does not apply. The modified null stream behaves like user-created streams.

All of the above statements have a single device in view. When multiple devices are involved, the work execution is independent, between devices. Any stream of device 0 does not in any way impact any stream on device 1. Even operations issued into the null streams of separate devices can overlap with each other.

The usage of multiple CPU threads, or not, has no bearing on the above statements. The above statements are true whether you issue work from multiple CPU threads, or not.

For additional treatment, I recommend unit 7 of this online training series.

@C.Yang

For this statement Two commands from different streams cannot run concurrently if any one of the following operations is issued in-between them by the host thread, does it mean this kind of operation can only start after all previously submitted ops ended, and all ops submitted after this kind of op can only start after this op ends?

It means that the operation issued in between, in the delineated list, is issued as if it were a default null stream activity, following the description given above. An ordinary cudaMalloc call falls into this category, for example.

@C.Yang

If so, this means other than per stream op order, GPU will also maintain global op submission order internally?

I don’t believe it is necessary to answer that question, and I don’t know what the GPU maintains internally. The rules of stream behavior are as I described, and AFAIK they are sufficient to sort out and predict behavior of the GPU, from the programmer’s perspective.