Cannot use Farm anymore

Hi Everyone !

I’m a little bit disappointed by the last version of Create (if it’s its fault), because…I just cannot use The Farm Rendering anymore ! All the jobs ends with an “Errored” Status…

I tried several jobs, I completely uninstalled the Omniverse app and launcher, I even used the official Cleaner Tool to make a full clean install…But nothing worked !

You’ll find attached the log of the last job I tried !

Omniverse_Farm_Error_Log.txt (154.3 KB)

I noticed 2 things that might be responsible for…
The first one is :

[Error] [carb] Failed to make directory 'C:/Windows/system32/${job}'! {error = 5}

The second one is :

[Error] [omni.ext.plugin] Failed to resolve extension dependencies. Failure hints:
	[omni.services.farm.agent.runner-0.4.8] dependency: (name: 'omni.services.facilities.log', tag: '', version: '', exact: 0, optional: 0) can't be satisfied. Available versions:
	 (none found)

There are several warnings in the log, but they all seems to concern the dependencies…

Does anyone have the same problem since the last update ?
Does anyone have any idea of a solution ?

Best Regards.

Another information, I’m trying to make this render on the same machine (scene prepared on Create on this machine, sent to the Farm Queue on this machine (localhost), and using the Farm Agent on this machine (localhost)).

The Farm Agent communicates correctly with the Farm Queue, the task is attributed to my Agent and, after some seconds, its status goes to “Errored”

With the previous release of Create (2021.3.8) all the Queue workflow worked with no problems at all. I saw, for a second, the Create loading screen appear, then disappear, and the task was computed in the background, now I don’t see anything, the task is attributed, nothing happens and…“Errored”…

Another log, I retried the same job, by re-submitting it to the Queue :

Omniverse_Farm_Error_Log2.txt (104.6 KB)

The render didn’t work, but surprinsingly, the “Warnings” appearing in the previous try doesn’t appear anymore, but the final “Error” is still the same.

I also tried to render a totally different scene, with only a cube in it, to see if my scene was responsible of the fails…but I had the same result…

Hello @billyjohn1! I’m sorry to hear that you are having trouble with Farm. I asked the development team to take a look at your files. I will post back here as soon as I hear back from them!

Hi @billyjohn1, Sorry for the hassle with this.

We have a newer version of Farm coming but lets see if we can figure out a temporary workaround.
Normally all of the extensions needed to run the various farm jobs come shipped with Farm and should load themselves into the Create runtime.

Would you mind having a looking in the installation folder of the Farm Agent and underneath the jobs directory there should be a create-render job with an exts-* folder in there. Would you mind letting me know whats in that directory?

Thanks,
Jozef

Hi @WendyGram ! Hi @jeenbergen !

Thank you for taking some time to help me solve my problem !

You’ll find attached 2 screenshots of the folders, I made a focus on the “extensions” folders, but if you need more, do not hesitate to ask !

So, @jeenbergen , Is there something missing ?

Hi @billyjohn1 , thanks for that.

I see what the issue is now. With the latest version of Create we have tried to slim down the amount of default extensions that ship with the product as they are generally not used.
It looks like that has caused some of the ones available in the previous version to no longer be there and when we ship the new version of OV Farm we have them bundled up with the render job itself.

I am just checking what is easiest at the moment to share the new render job. OV Farm isn’t far out release wise but wouldn’t want to hold you up.

Thanks,
Jozef

Hi @jeenbergen !

You’re confirming what I was suspecting ^^

Maybe I could just toggle these missing extensions on, in Create, and activate the “Autoload” on them to be sure that they’ll be installed and ready to use by the Farm Agent ?

Which of these extensions are missing ?

Or could you please generate an archive with the folders of these extensions, make it available online and then, I (and the other persons that would have this problem) could download it, and unpack it in the good folder, which should be a good workaround I think…

Best Regards

Hi @billyjohn1

We should have the new release of Farm out on Tuesday at the latest and that should resolve all of the above.
I hope that is ok timeline wise?

Thanks,
Jozef

Hi @jeenbergen !

Sure, I’ll deal with it ! It’l be easier than trying to solve the problem by taking the risk of creating new ones ^^

Thank you for your help !

Hi @jeenbergen !

Sorry Jozef…I downloaded the latest version of Farm Queue and Farm Agent…And I still have the same error…

You’ll find the latest try’s log attached…the error seems to be still the same…

Omniverse_Farm_Error_Log3.txt (152.7 KB)

Best Regards.

Hmmm that is very strange as I ran a similar test earlier to make sure everything was ok.
You are using Create 2022.1.0?

In the Agent UI could you send me what job directory it is using?
Would you also mind double checking the new installation directories of the agent and the jobs directory to check if all of the extensions are there?

Thanks,
Jozef

Hi @jeenbergen

Yes, I’m using Create 2022.1.0

Here are the job directories , one inherited from the previous version, and one created by the new one (I didn’t do anything, the soft created the paths on its own)

And, concerning the extensions folder :

For the Agent folder :

And the Job folder :

image

Thanks @billyjohn1
It is likely down to the two directories having similar jobs and one overwriting the other.
Can you try and remove the farm-agent-102.1.0 from the job directories by clicking the red trash can and then clicking apply settings

The files won’t be removed from disk but it’ll remove the job directory and will make sure the agent only uses the job definitions defined in the farm-agent-103 directory

Hi @jeenbergen

I’ve tried, not better…

I removed the folder from the agent, applied the new settings, and I even re-create the job in the queue…

Here is the log :

Omniverse_Farm_Error_Log4.txt (152.7 KB)

Seems to be still the same…

A problem with dependencies…

Should I try by removing Farm Queue and Farm Agent and by re-installing only the very last version of them ?

Have you noticed that there’s this error at the beginning of the log :

2022-04-04 22:11:20 [34ms] [Error] [carb] Failed to make directory 'C:/Windows/System32/${job}'! {error = 5}

Could it be also responsible ? even if the last error message clearly says :

2022-04-04 22:11:22 [1,645ms] [Error] [omni.ext.plugin] Failed to resolve extension dependencies. Failure hints:
	[omni.services.farm.agent.runner-0.4.8] dependency: (name: 'omni.services.facilities.log', tag: '', version: '', exact: 0, optional: 0) can't be satisfied. Available versions:
	 (none found)

Hi @jeenbergen

Ok, it seems to work now !

I completely uninstalled both Farm Queue and Farm Agent and re-installed them…And now it seems to work !

Stupidly, I launched a quite performance-expensive render, so I’ll really see the progress in a moment…

But it seems to work !

Last thing I haven’t tried yet, a render with several machines…

I take this opportunity to ask you another question ! I have been using Omniverse for a short time, and I haven’t had time to test renderings with several machines yet. Should we create an Omniverse server (with Nucleus) and host the work files on it so that the different machines on the network are able to perform their work and deposit the results in the target folder ? Or is a local folder on a machine sufficient and allows Queue tasks to be performed ?

Best Regards

Hi @billyjohn1

Glad we managed to get it resolved and that things appear to be working!

Nucleus helps for sharing the data but would need a service account configured and you’d need to add the OMNI_USER and OMNI_PASS environment variables with the values for the service accounts username and password to the runtime environment.
It is however also possible to not use nucleus as long as each one of the machines can access the data via a network share or similar.

Hope that helps but let us know if anything else comes up.

Thanks,
Jozef

The Problem is solved, if you are experiencing the same problem, think about updating the Farm Agent and the Farm Queue and even think about uninstalling all the previous versions and only work with the latest ones !

Thank you again @jeenbergen for your (and your team’s) help !

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.