Consistent ILLEGAL_OPCODE exception during Raytracing

In my raytracer module (initially created from microsoft DXR sample) during rendering still frames I noticed that some resolutions combined with some sample-per-pixel values would cause exception in nvidia driver and, in the end, ‘device removed’ event.
In short, I generate N*N rays in ‘raygen’ shader, where N is a number passed in through constant buffer.
All processing is handled by single ‘closesthit’ shader, which implements something like ‘material resolution’ and would result in recursive calls (currently limited at 8) tracing reflections, refractions or collecting AO.

There seems to be some kind of ‘boundary’, which depends on Number of samples-per-pixel and frame size, which is quite predictable and well reproducible.
Going over this ‘boundary’ would result in series of exceptions (in system Even log these are logged in pairs, Graphics Exception: ESR 0x404490=0x80000004 and Graphics Exception: ILLEGAL_OPCODE).

Could anybody point me in the right direction?