PxParticleBase::createParticles failing only on GPU Mode (PhysX 3.4, SPH Fluids)

I have a scene that adds a small number of GPU accelerated fluid particles to the PhysX scene every tick. (16 or so per tick, increasing over time). The logic used is a modified version of the Samples project that came with the PhysX source files.

100% of the time, at the same tick, (96 active particles, and attempting to create 32 more) the call into PxParticleBase::createParticles returns false and the GPU acceleration cannot be be used until the application is restarted. Particles can still be created if we turn off the eGPU flag (all during runtime).

What could be causing an issue like this?

Edit: It should be noted we’ve created dozens of other scenes of various scales and complexity, and we’ve only just now encountered this issue. It appears to be exceedingly rare.

System Information
NVIDIA System Information report created on: 05/18/2017 15:26:10

[Display]
Operating System: Windows 8.1, 64-bit
DirectX version: 11.0
GPU processor: GeForce GTX 670MX
Driver version: 382.05
Direct3D API version: 11.2
Direct3D feature level: 11_0
CUDA Cores: 960
Core clock: 601 MHz
Memory data rate: 2800 MHz
Memory interface: 192-bit
Memory bandwidth: 67.20 GB/s
Total available graphics memory: 9191 MB
Dedicated video memory: 3072 MB GDDR5
System video memory: 0 MB
Shared system memory: 6119 MB
Video BIOS version: 80.04.97.00.14
IRQ: Not used
Bus: PCI Express x16 Gen2
Device Id: 10DE 11A1 10AD1043
Part Number: 2051 0003

[Components]

NvGFTrayPluginr.dll 3.6.0.74 NVIDIA GeForce Experience
NvGFTrayPlugin.dll 3.6.0.74 NVIDIA GeForce Experience
nvui.dll 8.17.13.8205 NVIDIA User Experience Driver Component
nvxdplcy.dll 8.17.13.8205 NVIDIA User Experience Driver Component
nvxdbat.dll 8.17.13.8205 NVIDIA User Experience Driver Component
nvxdapix.dll 8.17.13.8205 NVIDIA User Experience Driver Component
NVCPL.DLL 8.17.13.8205 NVIDIA User Experience Driver Component
nvCplUIR.dll 8.1.950.0 NVIDIA Control Panel
nvCplUI.exe 8.1.950.0 NVIDIA Control Panel
nvWSSR.dll 6.14.13.8205 NVIDIA Workstation Server
nvWSS.dll 6.14.13.8205 NVIDIA Workstation Server
nvViTvSR.dll 6.14.13.8205 NVIDIA Video Server
nvViTvS.dll 6.14.13.8205 NVIDIA Video Server
NVSTVIEW.EXE 7.17.13.8205 NVIDIA 3D Vision Photo Viewer
NVSTTEST.EXE 7.17.13.8205 NVIDIA 3D Vision Test Application
NVSTRES.DLL 7.17.13.8205 NVIDIA 3D Vision Module
nvDispSR.dll 6.14.13.8205 NVIDIA Display Server
NVMCTRAY.DLL 8.17.13.8205 NVIDIA Media Center Library
nvDispS.dll 6.14.13.8205 NVIDIA Display Server
PhysX 09.17.0329 NVIDIA PhysX
NVCUDA.DLL 6.14.13.8205 NVIDIA CUDA 8.0.0 driver
nvGameSR.dll 6.14.13.8205 NVIDIA 3D Settings Server
nvGameS.dll 6.14.13.8205 NVIDIA 3D Settings Server

[Licenses]

3DTV Play

Digging further down, this is as far as I can get in the stack:

Sample Project

//Problem occurs when this returns false:
mParticleSystem->createParticles(particleCreationData);

calls:

Into the PhysX libraries
ScbParticleSystem.cpp

bool Scb::ParticleSystem::createParticles(const PxParticleCreationData& creationData)
{
...
bool ret = mParticleSystem.createParticles(creationData);
...
}

to:
ScParticleSystemCore.cpp

bool Sc::ParticleSystemCore::createParticles(const PxParticleCreationData& creationData)
{
...
return getParticleState().addParticlesV(creationData);
...
}

However, I can’t step through to the “addParticlesV” logic because the line apparently requires the symbols file for PhysX3GPU_64.dll to resolve getParticleState().

Best I can figure, it goes to this method:

ScParticleSystemSim.cpp

bool ParticleData::addParticlesV(const PxParticleCreationData& creationData)
{
	PX_ASSERT(creationData.numParticles <= mMaxParticles);
	PX_ASSERT(creationData.indexBuffer.ptr() && creationData.positionBuffer.ptr());
	PX_ASSERT((mRestOffsetBuffer != NULL) == (creationData.restOffsetBuffer.ptr() != NULL));

	const PxVec3 zeroVector(0.0f);

	PxStrideIterator<const PxU32> indexIt = creationData.indexBuffer;
	PxStrideIterator<const PxVec3> positionIt = creationData.positionBuffer;
	PxStrideIterator<const PxVec3> velocityIt =
	    creationData.velocityBuffer.ptr() ? creationData.velocityBuffer : PxStrideIterator<const PxVec3>(&zeroVector, 0);

	for(PxU32 i = 0; i < creationData.numParticles; i++)
	{
		const PxU32 particleIndex = *indexIt;
		PX_ASSERT(particleIndex <= mMaxParticles);

		Particle& particle = mParticleBuffer[particleIndex];
		PX_ASSERT(!mParticleMap.test(particleIndex));
		mParticleMap.set(particleIndex);

		if(particleIndex + 1 > mValidParticleRange)
		{
			mValidParticleRange = particleIndex + 1;
		}
		else
		{
			PX_ASSERT(!(particle.flags.api & PxParticleFlag::eVALID));
		}

		particle.position = *positionIt;
		particle.velocity = *velocityIt;
		particle.flags.low = 0;
		particle.flags.api = PxParticleFlag::eVALID;
		particle.density = 0.0f;

		mWorldBounds.include(particle.position);

		positionIt++;
		velocityIt++;
		indexIt++;
	}

	if(mRestOffsetBuffer)
	{
		PxStrideIterator<const PxF32> restOffsetIt = creationData.restOffsetBuffer;
		indexIt = creationData.indexBuffer;

		for(PxU32 i = 0; i < creationData.numParticles; i++)
		{
			const PxU32 particleIndex = *indexIt;
			mRestOffsetBuffer[particleIndex] = *restOffsetIt;
			restOffsetIt++;
			indexIt++;
		}
	}

	mValidParticleCount += creationData.numParticles;
	return true;
}

but even if I set breakpoints in it, it never gets there, even on ticks where it successfully adds particles.