The latest version of the Pytorch NGC image contains an older version of the flash-attention library which does not have a a critical out-of-bounds memory fix to fash attention 2. Over long contexts on the order of 240k tokens, the installed version of flash attention 2 overflows and computes attention incorrectly. This bug is fixed in the 2.8.3.post1 release of flast attention 2.
Given that nemotron ultra officially has support for 1 million context length, this means that Nvidia’s official Pytorch images, or any images derived from it which do not manually upgrade this dependency will not have correct long context support for Nemotron Ultra.