I figured out the issue. The dataset was being read in as NCHW instead of NHWC because the GPU on the AWS instance I was using was CUDA COMPUTE < 5.2. This lead the training script to ignore the GPU and try to use the CPU instead. Shifting to a beefier GPU and creating the results directory new everytime solved the issue.