Sending render job to farm from Create results in a bunch of warnings and no render

Hello,

I have created some custom code based on the Movie Maker extension to send a render job through to farm for rendering. I’m getting a bunch of warnings in the farm management dashboard that are just repeated infinitely and the render never occurs. I have attached my log

a5f1b9c5-5a00-49a7-aad3-f2bb9ed70879.txt (652.2 KB)

Hello @michaelbaggott! I am connecting you to a developer for more help. We should hear back very soon!

1 Like

Hi Michael,

I had a look at the logs and that is indeed very strange behaviour.
Would you be able to share the job settings itself?

Do you get the same behaviour when you submit to Farm directly from the movie capture extension itself using the submit to Queue option?

Thanks,
Jozef

Hello!

I did a test, and submitted a render from Create to the Farm Queue. It worked and was able to complete the render. So i duplicated the settings in my code from the successful render. The job exists in the agent now with duplicate settings to the working one, yet it won’t render. I will attach the latest log (which is different this time), and paste the extension code so you can see the settings.

import threading

import omni.ext

import omni.ui as ui

import omni.usd

import omni.kit

import asyncio

import json

import sys

import getpass

import carb

import omni.services.client as _services_client

# Any class derived from `omni.ext.IExt` in top level module (defined in `python.modules` of `extension.toml`) will be

# instantiated when extension gets enabled and `on_startup(ext_id)` will be called. Later when extension gets disabled

# on_shutdown() is called.

class MyExtension(omni.ext.IExt):

    # ext_id is current extension id. It can be used with extension manager to query additional information, like where

    # this extension is located on filesystem.

    def on_startup(self, ext_id):

        print("[omni.hello.world] MyExtension startup")

        self._window_title = "ConceptV Render Automation"

        self._menu_path = "Window/ConceptV Render"

        self._window = None

        try:

            self._menu = omni.kit.ui.get_editor_menu().add_item(

                self._menu_path, self.show_window, toggle=True, value=False

            )

        except Exception as e:

            self._menu = None

               

    def on_shutdown(self):

        print("[omni.hello.world] MyExtension shutdown")

        if self._window:

            self._window.set_visibility_changed_fn(None)

        self._window = None

        self._menu = None

    def build_window(self):

        self._window = ui.Window(self._window_title, width=300, height=300, visible=True)

        self._window.set_visibility_changed_fn(self._visibility_changed_fn)

        with self._window.frame:

            with ui.VStack():

                ui.Label("Load layers and begin render")

                def load_layers():

                     if enableExtension == True:

                        async def prepare_stage(masterPath):

                            (result, error) = await omni.usd.get_context().open_stage_async(masterPath)

                            if not result:

                                print(f"Errors: error")

                            stage = omni.usd.get_context().get_stage()

                            root_layer = stage.GetRootLayer()

                            root_layer.subLayerPaths.append(dayPath)

                            root_layer.subLayerPaths.append(nightPath)

                            root_layer.subLayerPaths.append(sdotPath)

                            asyncio.ensure_future(save_stage())

                                                   

                        async def save_stage():

                            (result, error) = await omni.usd.get_context().save_as_stage_async('omniverse://localhost/Projects/Moreton/Moreton_Save.usd')

                            if not result:

                                print(f"Errors: error")

                            print('Stave Saved')

                        asyncio.ensure_future(prepare_stage(masterPath))

                ui.Button("Load Layers", clicked_fn=lambda: load_layers())

                def render():

                    _settings = carb.settings.get_settings()

                    async def submit():

                        settings = {

                                    "camera": "/OmniverseKit_Persp",

                                    "range_type": 0,

                                    "capture_every_nth_frames": 1,

                                    "fps": "24",

                                    "start_frame": 1,

                                    "end_frame": 1,

                                    "start_time": 0,

                                    "end_time": 10,

                                    "res_width": 1920,

                                    "res_height": 1080,

                                    "render_preset": 1,

                                    "debug_material_type": 0,

                                    "spp_per_iteration": 1,

                                    "path_trace_spp": 1,

                                    "ptmb_subframes_per_frame": 1,

                                    "ptmb_fso": 0,

                                    "ptmb_fsc": 1,

                                    "output_folder": "C:/render_output",

                                    "file_name": "Capture",

                                    "file_name_num_pattern": ".####",

                                    "file_type": ".png",

                                    "save_alpha": False,

                                    "hdr_output": False,

                                    "show_pathtracing_single_frame_progress": False,

                                    "preroll_frames": 0,

                                    "overwrite_existing_frames": False,

                                    "movie_type": 0,

                                    "sunstudy_start_time": 0,

                                    "sunstudy_current_time": 0,

                                    "sunstudy_end_time": 0,

                                    "sunstudy_movie_length_in_seconds": 0,

                                    "sunstudy_player": None,

                                    "real_time_settle_latency_frames": 0

                        }

                        try:

                            task_type = "create-render"

                            queue_management_endpoint_prefix = _settings.get_as_string(

                                "exts/omni.kit.window.movie_maker/queue_management_endpoint_prefix"

                            )

                            management_services = _services_client.AsyncClient(uri=f"{'http://localhost:8222'}{queue_management_endpoint_prefix}")

                            await management_services.tasks.submit(

                                user=getpass.getuser(),

                                task_type=task_type,

                                task_args={},

                                task_function="render.run",

                                task_function_args={

                                    "usd_file": "omniverse://localhost/Projects/Moreton/Moreton_Save.usd",

                                    "render_settings": settings,

                                    "render_start_delay": 10,

                                },

                                task_requirements={},

                                task_comment=""

                            )

                        except Exception as exc:

                            print("Task submission failed")

                            print(f"Error submitting render to Queue: {str(exc)}")

                    asyncio.ensure_future(submit())

                ui.Button("Render", clicked_fn=lambda: render())

    def show_window(self, menu, value):

        if not self._window:

            self.build_window()

        elif self._window:

            self._window.visible = value

       

    def _visibility_changed_fn(self, value):

        if self._menu:

            omni.kit.ui.get_editor_menu().set_value(self._menu_path, value)

e9f147a0-9d7e-44ed-94bd-be05c720c935.txt (1.5 MB)

Here i will paste the two settings objects, so you can see they are the same:

The working job:

###### User

michaelbaggott


###### Comment

(No comment available)


###### Definition

{
“task_type”: “create-render”,
“task_args”: {},
“task_function”: “render.run”,
“task_function_args”: {
“usd_file”: “omniverse://localhost/Projects/Moreton/Moreton_Save.usd”,
“render_settings”: {
“camera”: “/OmniverseKit_Persp”,
“range_type”: 0,
“capture_every_nth_frames”: 1,
“fps”: “24”,
“start_frame”: 1,
“end_frame”: 1,
“start_time”: 0,
“end_time”: 10,
“res_width”: 1920,
“res_height”: 1080,
“render_preset”: 1,
“debug_material_type”: 0,
“spp_per_iteration”: 1,
“path_trace_spp”: 1,
“ptmb_subframes_per_frame”: 1,
“ptmb_fso”: 0,
“ptmb_fsc”: 1,
“output_folder”: “C:/render_output”,
“file_name”: “Capture”,
“file_name_num_pattern”: “.####”,
“file_type”: “.png”,
“save_alpha”: false,
“hdr_output”: false,
“show_pathtracing_single_frame_progress”: false,
“preroll_frames”: 0,
“overwrite_existing_frames”: false,
“movie_type”: 0,
“sunstudy_start_time”: 0,
“sunstudy_current_time”: 0,
“sunstudy_end_time”: 0,
“sunstudy_movie_length_in_seconds”: 0,
“sunstudy_player”: null,
“real_time_settle_latency_frames”: 0
},
“render_start_delay”: 10
},
“task_comment”: “”
}

The job that isn’t working:

###### User

michaelbaggott


###### Comment

(No comment available)


###### Definition

{
“task_type”: “create-render”,
“task_args”: {},
“task_function”: “render.run”,
“task_function_args”: {
“usd_file”: “omniverse://localhost/Projects/Moreton/Moreton_Save.usd”,
“render_settings”: {
“camera”: “/OmniverseKit_Persp”,
“range_type”: 0,
“capture_every_nth_frames”: 1,
“fps”: 24,
“start_frame”: 1,
“end_frame”: 1,
“start_time”: 0,
“end_time”: 10,
“res_width”: 1920,
“res_height”: 1080,
“render_preset”: 1,
“debug_material_type”: 0,
“spp_per_iteration”: 1,
“path_trace_spp”: 1,
“ptmb_subframes_per_frame”: 1,
“ptmb_fso”: 0,
“ptmb_fsc”: 1,
“output_folder”: “C:/render_output/”,
“file_name”: “Capture”,
“file_name_num_pattern”: “.####”,
“file_type”: “.png”,
“save_alpha”: false,
“hdr_output”: false,
“show_pathtracing_single_frame_progress”: false,
“preroll_frames”: 0,
“overwrite_existing_frames”: false,
“movie_type”: 0,
“sunstudy_start_time”: 0,
“sunstudy_current_time”: 0,
“sunstudy_end_time”: 0,
“sunstudy_movie_length_in_seconds”: 0,
“sunstudy_player”: null,
“real_time_settle_latency_frames”: 0
},
“render_start_delay”: 10
},
“task_comment”: “”
}

Hi Jozef,

I think i might have found something which could be causing the issue. There is a difference between these jobs, and that is who they are being submitted by. When i submit to queue from Create, it uses my username ‘michaelbaggott’, but when i submit through the Create extension using my own code, it submits with the username of the Windows user. I wonder if this username difference could be causing the render to fail, especially if it is accessing paths on Nucleus?

Hi Michael,

Thanks for the additional information.
The username itself wouldn’t, currently, make a difference as it is mostly used for tracking purposes but the jobs themselves will run under the user that started the agent from the launcher.

If you take the full job submission that works and then go to:
http://localhost:8222/docs, and submit the exact same job from there in the try it out for the submit function does that work ok?

I don’t see anything really standing out in the client code or the job itself that would cause issues.

Thanks,
Jozef

Hi Jozef,

When i do this with the job that worked from Create, i get an error response from the server.

curl -X 'POST' \
  'http://localhost:8222/queue/management/tasks/submit' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "task_type": "create-render",
  "task_args": {},
  "task_function": "render.run",
  "task_function_args": {
    "usd_file": "omniverse://localhost/Projects/Moreton/Moreton_Save.usd",
    "render_settings": {
      "camera": "/OmniverseKit_Persp",
      "range_type": 0,
      "capture_every_nth_frames": 1,
      "fps": "24",
      "start_frame": 1,
      "end_frame": 1,
      "start_time": 0,
      "end_time": 10,
      "res_width": 1920,
      "res_height": 1080,
      "render_preset": 1,
      "debug_material_type": 0,
      "spp_per_iteration": 1,
      "path_trace_spp": 1,
      "ptmb_subframes_per_frame": 1,
      "ptmb_fso": 0,
      "ptmb_fsc": 1,
      "output_folder": "C:/render_output",
      "file_name": "Capture",
      "file_name_num_pattern": ".####",
      "file_type": ".png",
      "save_alpha": false,
      "hdr_output": false,
      "show_pathtracing_single_frame_progress": false,
      "preroll_frames": 0,
      "overwrite_existing_frames": false,
      "movie_type": 0,
      "sunstudy_start_time": 0,
      "sunstudy_current_time": 0,
      "sunstudy_end_time": 0,
      "sunstudy_movie_length_in_seconds": 0,
      "sunstudy_player": null,
      "real_time_settle_latency_frames": 0
    },
    "render_start_delay": 10
  },
  "task_comment": ""
}'
422	
Error: Unprocessable Entity

Response body
Download
{
  "detail": [
    {
      "loc": [
        "body",
        "user"
      ],
      "msg": "field required",
      "type": "value_error.missing"
    }
  ]
}
Response headers
 content-length: 88 
 content-type: application/json 
 date: Thu,10 Mar 2022 19:10:23 GMT 
 server: uvicorn 

Hi again Jozef,

There’s definitely something weird going on with my Farm. I just submitted the job again from my extension, and this time it was successful and rendered ok. So i tried once more, and now it is just sitting on “Submitted” but the task won’t actually start.

Hi Michael,

That is indeed a little strange.
When it sits in this submitted state does the agent have a task processing and is it still connected? The UI seems to indicate that there is an active job but the task ID isn’t in the actual tasks list in the UI.

There could be 3 reasons why the job might not pick up:

  • No agent available (but your UI shows one)

  • A task still active (which is what your screenshot seems to indicate) but I can’t see the e9f job in the actual tasks. (Is there a ghost Create process lingering on the host?)

  • The job definition for the given agent isn’t available. That is unlikely unless you deleted them from the agent.

Hi Jozef,

Yes there is a ghost process lingering, i have noticed this in the past. After a failed render, when i run task manager, the kit process is still open with 7gb GPU RAM still in memory, even though i have cancelled the failed task in the manager, or it has errored. When i have tried to end the task, i get some strange behaviour afterwards, usually i have to restart wiondows, but it prevents me from rerunning failed renders.