ASUS Ascent GX10: changing file checksums; cached reads differ from O_DIRECT

,

Hi all,

I have two ASUS Ascent GX10 systems connected over ConnectX-7. One node is returning inconsistent data when reading unchanged model files. I’m trying to determine whether this is a known kernel/driver issue or something I should take to ASUS for hardware diagnosis.

This started while trying NVIDIA’s official two-node GLM-5.3-Flash NIM deployment. The issue was present before updating and still reproduces after updating and rebooting both machines.

Environment

Both nodes now report:

  • ASUS Ascent GX10, NVIDIA GB10, 128 GB unified memory
  • DGX OTA version: 7.5.0
  • Kernel: 6.17.0-1032-nvidia
  • NVIDIA driver: 580.173.02
  • NIM image: nvcr.io/nim/zai-org/glm-5.3-flash:2.1.2-variant

The checks below were run with both inference containers stopped. The diagnostic processes used temporary CPU-only containers, no GPU access, and a read-only mount of the model cache.

Initial symptoms

Docker pulls failed on the worker with:

failed to register layer: flate: corrupt input before offset 3829160527
failed to register layer: archive/tar: invalid tar header

Loading the head’s image over SSH succeeded. NIM then failed model-file checksum verification on the worker. The head’s complete set of 120 weight shards passed verification against the image’s model manifest.

Before reboot, worker shards 42, 76 and 119 failed. Repeated reads of shard 76 produced different hashes. After reboot, shard 76 passed three reads without replacing it, but a complete audit instead found a mismatch in shard 70.

Byte-level comparison

For model-00070-of-00120.safetensors:

  1. The head returned the expected BLAKE3 and SHA-256 hashes on three complete reads.
  2. The worker returned different BLAKE3 and SHA-256 hashes on each complete read.
  3. Comparing SHA-256 hashes of 1 MiB blocks against the head isolated the difference to block 1398 (zero-based). Immediately hashing the same in-memory buffer twice gave consistent results.
  4. Comparing the actual bytes against that block from the head: five buffered reads each contained 125–128 incorrect bytes, with changing values.
  5. Five O_DIRECT reads of the same region matched the head byte-for-byte. These used an aligned anonymous mmap buffer and Python os.readv.

The block read starts at 1398 * 1048576 bytes. The first differing offsets within it included 860188, 860190, 860220 and 860222.

Evicting only that region with:

os.posix_fadvise(fd, 1398 * 1048576, 1048576, os.POSIX_FADV_DONTNEED)

made the next three buffered reads correct. Three complete reads of shard 70 then matched both expected hashes, without rewriting or downloading the file.

However, another complete 120-shard audit then found mismatches in different shards: 1 and 78. Shard 70 passed. So cache eviction was only a temporary workaround.

Other context

  • This worker previously became unresponsive and needed a manual power cycle. I don’t know whether that event is related.
  • My previous EXL3 inference setup could run, but I have not established equivalent file-integrity checks for it.
  • No failed systemd services or unfinished package configuration were reported after the update.
  • A scoped current-boot kernel-log check showed no explicit NVMe I/O errors, uncorrectable-memory reports or NVIDIA Xid errors.
  • I have not run Field Diagnostics yet.

The direct reads suggest the tested region is intact on disk, but I don’t want to assume this proves the SSD is healthy or identifies a particular faulty component.

I found this similar report of Docker digest failures and changing checksums, which ended in an RMA recommendation:

Has anyone seen this specific buffered-read versus O_DIRECT behavior on GB10/GX10? Is there a known software issue or a recommended diagnostic sequence for the ASUS model before pursuing hardware service? I can provide the checksum and byte-comparison scripts if useful.

Thanks!