We’re investigating the possiblity to use a Disk to GPU to RAM pipeline to read, decompress, process and ship data. The targeted data would be known beforehand, which means if there was a way to predetermine which compressed chunks or partitions of the whole dataset to load (“random access”) we could significantly speed up the process because we wouldn’t have to load the whole dataset to VRAM. Is this process feasible with nvCOMP using zstd? Or what would be the idiomatic approach to the same end?
FYI, I moved this post from the NVHPC compilers forum to the NV Libraries forum snice it’s a question about NVComp library, not about the compilers.