Hello，I’m using the replicator to generation dataset，but with the frame increases，the generate speed become slower and slower，as the picture follow，
At the beginning of the program run，there are 8 image can be generate in one minutes，however，with the program going run，there is only 2 image can be generate in one minuter，and the speed is slower and slower. how to solve this problem? Is there some configuration that i forget to set in my code?
Hi there,
if you get a warning message with Throttling generation due to I/O bottleneck. it is because the writing data to your hardrive is slower that generating the data, so the framerate will be forced to drop. From the images you shared this does not seem to be the case. Can you share more information about your scenario/hardware? Best would be a script we can replicate. Try following your PC and GPU memory.
Best,
Andrei
Hi，my hardware configuration are：
---------------------------------------------------------------------------------------------|
| Driver Version: 516.94 | Graphics API: D3D12
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 3070 | Yes: 0 | | 8043 MB | 10de | c9240000.. |
| | | | | | 2488 | 0 |
|=============================================================================================|
| OS: Windows, Version: 10.0, Build: 19044
| Processor: 12th Gen Intel(R) Core(TM) i7-12700F | Cores: 12 | Logical: 20
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 65293 | Free Memory: 58424
| Total Page/Swap (MB): 75021 | Free Page/Swap: 65315
|---------------------------------------------------------------------------------------------|
and my code of the replicator part is as follow：
def start(name):
#region read config
Material = []
Demos=[]
Material=Readfile(MaterialPath,ratio=6) #read the material path and save as a list
Demos=Readfile(DomePath)
print(len(Material))#about 150
print(len(Demos))#about 25
DemosIndex=np.random.randint(0, len(Demos)-1, size=1)[0]
#endregion
carb.settings.get_settings().set("/omni/replicator/RTSubframes", 5)
#rep.settings.set_render_pathtraced()
with rep.new_layer():
path=CarPath
portprims=["/Replicator/Ref_Xform/Ref/compound_23/port"]
mats = rep.create.material_omnipbr(diffuse=rep.distribution.uniform((0, 0, 0), (1, 1, 1)), count=100)
car = rep.create.from_usd(path)
camera=rep.create.camera(
position=cameraInitpos,
rotation=cameraInitrotation,
focus_distance=rep.distribution.normal(400.0, 100),
f_stop=0,
focal_length=8
)
rep.modify.semantics([("class", "port")], input_prims=portprims)
# region plane
def get_planeDR():
with car:
rep.randomizer.materials(materials=Material,input_prims =["/Replicator/Ref_Xform/Ref/compound_23/compound_23/Material_00"])
rep.randomizer.materials(materials=Material,input_prims=["/Replicator/Ref_Xform/Ref/compound_23/compound_23/Material_01"])
rep.randomizer.materials(materials=Material,input_prims=["/Replicator/Ref_Xform/Ref/compound_23/compound_23/Material_03"])
return car.node
#endregion
def get_PortDR():
spheres = rep.get.prims(semantics=[('class', "port")])
with spheres:
rep.randomizer.materials(materials=Material)
return spheres.node
def get_CameraDR():
with camera:
rep.modify.pose(
position=rep.distribution.uniform((cameraInitpos[0]-15, cameraInitpos[1]-20, cameraInitpos[2]-20),
(cameraInitpos[0]+15, cameraInitpos[1]+20, cameraInitpos[2]+20),
seed=np.random.randint(0, 1000, size=1)),
rotation=rep.distribution.uniform((cameraInitrotation[0]-5, cameraInitrotation[1]-5, cameraInitrotation[2]-30),(cameraInitrotation[0]+5,cameraInitrotation[1]+ 5,cameraInitrotation[2]+ 30)),
input_prims=["/Replicator/Camera_Xform/Camera"]
)
return camera.node
def get_Light1DR():
domelight = rep.create.light(
light_type="Dome",
rotation=rep.distribution.uniform((0, 0, 0), (0, 0, 360)),
texture=rep.distribution.choice(Demos)
)
return domelight.node
def get_GoblaLightDR():
goballight = rep.create.light(light_type="Sphere",
temperature=rep.distribution.normal(6500, 500),
intensity=rep.distribution.normal(3500000, 5000),
position=rep.distribution.uniform((-300, -300, 200), (300, 300, 300)),
rotation=rep.distribution.uniform((0, 0, 0), (360, 360, 360)),
scale=rep.distribution.uniform(50, 100),
count=1)
return goballight.node
rep.randomizer.register(get_planeDR)
rep.randomizer.register(get_PortDR)
rep.randomizer.register(get_CameraDR)
rep.randomizer.register(get_Light1DR)
rep.randomizer.register(get_GoblaLightDR)
numf=0
with rep.trigger.on_frame(num_frames=num_frames):
numf=numf+1
print("now frames is=",numf)
rep.randomizer.get_planeDR()
rep.randomizer.get_PortDR()
rep.randomizer.get_CameraDR()
rep.randomizer.get_Light1DR()
rep.randomizer.get_GoblaLightDR()
# Initialize and attach writer
writer = rep.WriterRegistry.get("BasicWriter")
output_directory = "C:/Users/robot/Desktop/writer/"
now = datetime.now()
timestr = now.strftime("%Y%m%d%H%M%S")
output_directory = output_directory + timestr
if os.path.exists(output_directory):
print("exist")
else:
os.makedirs(output_directory)
print("###################################Outputting data to ", output_directory)
writer.initialize(
output_dir=output_directory,
rgb=True,
bounding_box_2d_tight=True
)
RESOLUTION = (1024, 1024)
driver_rp = rep.create.render_product(camera, RESOLUTION)
writer.attach([driver_rp])
run_orchestrator()
the warning message when running are，It seems that there has no message about the IO bottleneck
[47.373s] app ready
[50.471s] Simulation App Startup Complete
120
27
2023-01-14 05:34:08 [50,963ms] [Warning] [carb.flatcache.plugin] UsdRelationship /Replicator/SDGPipeline/OgnGroup.inputs:prims has multiple targets, which is not supported
now frames is= 1
###################################Outputting data to C:/Users/robotkk/Desktop/writer/20230114133408
2023-01-14 05:34:08 [51,264ms] [Warning] [carb.flatcache.plugin] Type tag does not have a corresponding USD type
2023-01-14 05:34:10 [53,065ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4EDC0 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,065ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4CC00 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,065ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4D6A0 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,065ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC0F080 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,066ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4D420 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,067ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC0F080 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,068ms] [Warning] [omni.syntheticdata.plugin] OgnSdInstanceMapping missing valid input renderVar InstanceSemanticMapSD
2023-01-14 05:34:10 [53,281ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4EDC0 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,282ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4CC00 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,282ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4D6A0 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,283ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC0F080 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:10 [53,283ms] [Warning] [carb.flatcache.plugin] No source has valid data elemSize=8 array=0000025FEEC4D420 usdValid=0 cpuValid=0 gpuValid=0 gpuAllocedWithCuda=0
2023-01-14 05:34:17 [59,949ms] [Warning] [gpu.foundation.plugin] Requesting texture to use texture streaming, but the service isn't available.The texture will be created as a regular resource.
Can you try with 1 Material and Demos instance to see if the issue happens with less examples as well? If the example inputs are public or from the Isaac folder can you share them, otherwise I cannot replicate the issue. Would also make sense to check if any memory gets full with time (GPU/RAM) and that might cause slowing the recording. If you can share a full runnable script I can also try out various configurations to test what the issue might be. Feel free to cut out parts that you think are not needed.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.