Num shard limit on SSD

Please provide the following information when requesting support.

• TLT Version tao-toolkit:5.0.0-tf1.15.5)

in the ssd docs

it says num_shards is 1-20

does anything bad or unproductive beyond that?
what is the max data size we can have (if that is maxing out at 20?)

for example approx n 1.5MB images?


There is not limitation for num_shards. The description here is not correct. But if user has few images(for example, only 10 images) and set num_shards: 20, when run dataset_convert, the 10 images can not be split into each shard. So, it may bring unexpected issue since some tfrecords file will have 0 byte size.

1 Like

Thanks I thought so and I have been using much higher numbers! but it is always good to make sure as no there is no doubts!

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.