Request for a midi/osc mapper in omniverse

unveil.contact.berlin · May 25, 2022, 6:49am

Hi,

I really would like to have an extension that can take some inputs like osc/midi/tcpIP/udp/ or any other protocol an map those data in to some parameter of the omniverseCreate. I m a realtime guy and I mostly use Touchdesigner for my workflow. After i saw your render quality…. I just WANT IT!!! So can you please make a receiver and mapper for the omniverse? I hope my request is clear… nothing fancy just a mapper like ableton has… but not only for midi, should have at least 1 protocool to get signal from another pc/ or software!! Thanks

erich11 · August 7, 2022, 1:03am

I see this post didn’t generate much interest, but I found it searching for Omniverse + OSC.

I haven’t used TouchDesigner in years, but reviewing the site I noticed a reference there to USD… I wonder if one good solution might be a OV connector for TouchDesigner?

That said, production designers across the spectrum have been looking for a “unified” tool for design and show control for, well, ever. I’m curious if Nvidia imagines Omniverse having “realtime” sync features with external systems?

unveil.contact.berlin · August 8, 2022, 12:57pm

Dear erich11,

I think that is anyway possible to have bilateral communication between TD and omiverse. I think that an extension would have been more appropriate for this since usd works with point and attribute and i just would like to send value and map it to the omiverse. Like the osc receiver in unreal. I was thinking about a node that receive osc (or any other protocoll) that with omnigraf you send the value to what u want. Even a node for parameter in would be fine…. Thank for your interest!!!

erich11 · August 8, 2022, 5:08pm

Hi @unveil.contact.berlin - I probably was over generalizing and assuming that “connectors” extend to live data exchange for i.e. OSC/DMX, etc. I’m sure you’re right that part probably would be an extension.

That said, for some of the things I’m imagining, being able to share set geometry between TD and OV would be important, and it seems like OV (as more CAD connectors come on line) would be a great way to build out complex digital twins of performance spaces that could then be used in TD for a range of things like show control, projection mapping, etc.

jshrake · August 8, 2022, 5:20pm

Hey @unveil.contact.berlin and @erich11! We are actively working on a Kit extension to support OSC in Omniverse. The initial features include:

Sending and receiving OSC messages programmatically from your own Python extension
Receiving OSC messages directly from OmniGraph

You can see an example below that demonstrates a custom action graph handling OSC messages sent from the TouchOSC app running on an iPad

osc_demo

This should be released soon. I’ll ping this thread when it’s available with a more official announcement.

Best,
Justin

erich11 · August 8, 2022, 6:22pm

This is great news, @jshrake! This will be super powerful for so many external tools, and since OmniGraph and TouchDesigner are such similar paradigms, this solution will be great. Will we be able to have an output/sending node in OG as well, or is that easily achieved via a python extension (I’ll RTFM ;-)

re: a TouchDesigner connector for USD, I suppose this should be a separate feature request thread, but I really think OV+TD on the geometry side of things are a great match, and an ideal way to bring OV into the world of live performance design digital twins more broadly.

Thanks again for the great news!

unveil.contact.berlin · August 8, 2022, 9:18pm

Dear Justin,

This is a dream coming true. Please write here as soon this feature will be available, so i ll get notification :-).
I really looking forward to have kinect data from TD moving warp kernel that deform animated parametric usd coming from houdini!!!
Or take joint position from the maxine sdk send those data to TD…
May is time for me to start understand how to use code to create some custom apps…

Thank you again for your amazing work!!!

Regards

daryl.dunlap.ohio · August 15, 2022, 4:11pm

This is awesome!

Can’t wait to try this out.

ruvidan2001 · August 21, 2022, 4:24pm

Please let us know ASAP when OSC comes out. I am also am excitedly waiting this to be a thing. I have a lot of Touchdesigner programs that could sync wonderfully with Omniverse. Plus there are a lot of programs that can benefit from the communication of Omniverse having: OSC and Syphon/Spout

razvanm1 · September 14, 2022, 4:08pm

Just saw that this is available right now via the community extensions tab.
Thank you @jshrake!

unveil.contact.berlin · September 14, 2022, 4:13pm

Yaaaaaahooooooooo !!! Many thanks!!!

jshrake · September 14, 2022, 4:53pm

Hi all! As @razvanm1 mentioned, the OSC extension is now available to install through the community extensions tab. You can find the source and a getting started guide at GitHub - NVIDIA-Omniverse/kit-osc: Omniverse Kit extension for sending and receiving OSC (Open Sound Control) messages. I’ll be working to release some video tutorials in the future.

I’m really excited to see what the community makes with this extension and welcome your feedback.

Best,
Justin

daryl.dunlap.ohio · September 14, 2022, 5:01pm

What control surface software you using in your video? There are so many out there in the iPad app store to choose from…which is a great problem to have, but, just curious…

jshrake · September 14, 2022, 5:05pm

What control surface software you using in your video?

I’m using the TouchOSC app from Hexler TouchOSC | hexler.net but any OSC client capable of sending messages over UDP should work. Another OSC client that I’ve tested with is ‎Syntien on the App Store.

unveil.contact.berlin · September 15, 2022, 7:32am

Hi,

I try the extension yesturday with touchdesigner. Connection is working, but i have problem with the actionGraph: in order to assign the value to a primitive attribute i guess i need a name convention. I have some problem with the attribute extractor: it gives a warning no attribute type … i try with /scale and send 3 float in a … no luck until now…

If i can suggest something… a drag and drop solitution, or some script inside the parameters it self will make all easy. Something like omni.osc.nameOfValue and boom it links automatically.

I have time for a call or to send some example if needed.

Many thanks!!!

razvanm1 · September 16, 2022, 3:05pm

@unveil.contact.berlin try running TouchDesigner in Perfom Mode (F1). I had the same warnings with a Leap Motion but that fixed it.
Looking at the verbose log, it seems that it sometimes sends an array instead of only one value which was triggering those warnings. I guess the interface makes it lag and it helps to not render it.

I’ve also used Notch and my phone to send some sensors and they didn’t have this issue, even when sending from all 3 places at the same time.

jshrake · September 19, 2022, 3:57pm

@unveil.contact.berlin

Take a look at the following example USD file and inspect the action graph https://github.com/NVIDIA-Omniverse/kit-osc/blob/main/exts/omni.osc/data/examples/cube-with-graph.usd.

I have some problem with the attribute extractor: it gives a warning no attribute type … i try with /scale and send 3

The attribute to extract is named arguments. You can use the Bundle Inspector node to see all the atributes named in the bundle.

daryl.dunlap.ohio · December 3, 2022, 1:22am

So, I’m learning Warp right now and having alot of fun, then I thought, hmmm, that OSC extension would be nice to use as well - but, I’m getting errors with Create 2022.3.0

2022-12-03 01:18:26 [Info] [omni.kit.app.plugin] [52.093s] [ext: omni.osc-0.2.0] startup
2022-12-03 01:18:26 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] Add archive dir: ‘d:\apps\omniverse\data\kit\create.next\2022.3\exts\3\omni.osc-0.2.0+cp37\vendor’
2022-12-03 01:18:27 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] Attempting to install ‘python-osc’ from local acrhive: ‘d:\apps\omniverse\data\kit\create.next\2022.3\exts\3\omni.osc-0.2.0+cp37\vendor’
2022-12-03 01:18:27 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] run process: [‘d:\apps\omniverse\pkg\create-2022.3.0\kit\python/python.exe’, ‘-m’, ‘pip’, ‘–isolated’, ‘install’, ‘–target=D:\Apps\Omniverse\data\Kit\Create.Next\2022.3\pip3-envs\default’, ‘python-osc’, ‘–no-index’, ‘–find-links=d:\apps\omniverse\data\kit\create.next\2022.3\exts\3\omni.osc-0.2.0+cp37\vendor’]
2022-12-03 01:18:29 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] Attempting to install ‘python-osc’ from local acrhive: ‘d:\apps\omniverse\pkg\create-2022.3.0\kit\extscore\omni.kit.pip_archive\pip_archive’
2022-12-03 01:18:29 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] run process: [‘d:\apps\omniverse\pkg\create-2022.3.0\kit\python/python.exe’, ‘-m’, ‘pip’, ‘–isolated’, ‘install’, ‘–target=D:\Apps\Omniverse\data\Kit\Create.Next\2022.3\pip3-envs\default’, ‘python-osc’, ‘–no-index’, ‘–find-links=d:\apps\omniverse\pkg\create-2022.3.0\kit\extscore\omni.kit.pip_archive\pip_archive’]
2022-12-03 01:18:30 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] Attempting to install ‘python-osc’ from local acrhive: ‘d:\apps\omniverse\pkg\create-2022.3.0\extscache\omni.services.pip_archive-0.4.1\pip_archive’
2022-12-03 01:18:30 [Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] run process: [‘d:\apps\omniverse\pkg\create-2022.3.0\kit\python/python.exe’, ‘-m’, ‘pip’, ‘–isolated’, ‘install’, ‘–target=D:\Apps\Omniverse\data\Kit\Create.Next\2022.3\pip3-envs\default’, ‘python-osc’, ‘–no-index’, ‘–find-links=d:\apps\omniverse\pkg\create-2022.3.0\extscache\omni.services.pip_archive-0.4.1\pip_archive’]
2022-12-03 01:18:31 [Warning] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] ‘python-osc’ failed to install.
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] Failed to import python module omni.osc. Error: No module named ‘pythonosc’. Traceback:
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] Traceback (most recent call last):
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “d:/apps/omniverse/pkg/create-2022.3.0/kit/kernel/py\omni\ext_impl\custom_importer.py”, line 76, in import_module
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] return importlib.import_module(name)
2022-12-03 01:18:32 [Error] [omni.ext.impl.custom_importer] File "d:\apps\omniverse\pkg\create-2022.3.0\kit\python\lib\importlib_init.py", line 127, in import_module
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] return _bootstrap._gcd_import(name[level:], package, level)
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “”, line 1006, in _gcd_import
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “”, line 983, in _find_and_load
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “”, line 967, in _find_and_load_unlocked
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “”, line 677, in _load_unlocked
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “”, line 728, in exec_module
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] File “”, line 219, in _call_with_frames_removed
2022-12-03 01:18:32 [Error] [omni.ext.impl.custom_importer] File "d:\apps\omniverse\data\kit\create.next\2022.3\exts\3\omni.osc-0.2.0+cp37\omni\osc_init.py", line 19, in
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] from pythonosc import * # noqa: F401
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer] ModuleNotFoundError: No module named ‘pythonosc’
2022-12-03 01:18:32 [Error] [omni.ext._impl.custom_importer]
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] Exception: Extension python module: ‘omni.osc’ in ‘d:\apps\omniverse\data\kit\create.next\2022.3\exts\3\omni.osc-0.2.0+cp37’ failed to load.
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin]
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] At:
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:/apps/omniverse/pkg/create-2022.3.0/kit/kernel/py\omni\ext_impl_internal.py(188): startup
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:/apps/omniverse/pkg/create-2022.3.0/kit/kernel/py\omni\ext_impl_internal.py(280): startup_extension
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] PythonExtension.cpp::startup()(2):
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:\apps\omniverse\pkg\create-2022.3.0\kit\exts\omni.kit.window.extensions\omni\kit\window\extensions\ext_commands.py(29): do
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:\apps\omniverse\pkg\create-2022.3.0\kit\extscore\omni.kit.commands\omni\kit\undo\undo.py(80): execute
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:\apps\omniverse\pkg\create-2022.3.0\kit\extscore\omni.kit.commands\omni\kit\commands\command.py(459): execute
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:\apps\omniverse\pkg\create-2022.3.0\kit\exts\omni.kit.window.extensions\omni\kit\window\extensions\common.py(288): toggle_extension
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin] d:\apps\omniverse\pkg\create-2022.3.0\kit\exts\omni.kit.window.extensions\omni\kit\window\extensions\ext_components.py(89): toggle
2022-12-03 01:18:32 [Error] [carb.scripting-python.plugin]
2022-12-03 01:18:32 [Error] [omni.ext.plugin] [ext: omni.osc-0.2.0] Failed to startup python extension.
2022-12-03 01:18:32 [Info] [omni.ext.plugin] [ext: omni.osc-0.2.0] started, startup time: 5434 (ms)
2022-12-03 01:19:09 [Error] [carb.imaging.plugin] Failed to load image: Image not of any known type, or corrupt
2022-12-03 01:19:09 [Warning] [gpu.foundation.plugin] Couldn’t process the file, it might not have written completely.
2022-12-03 01:19:09 [Error] [gpu.foundation.plugin] Failed to load texture d:/apps/omniverse/data/kit/create.next/2022.3/exts/3/omni.osc-0.2.0+cp37/data/preview.png (reason: 3)

unveil.contact.berlin · December 3, 2022, 10:32am

Hi ,

I did not try the osc extension i 2022.3 but it loads with no problem at the software start up.
I let u know if
I got any issue. I really would like to start writing some warp kernel.

daryl.dunlap.ohio · December 3, 2022, 4:29pm

Yes, that’s why I stated 2022.3, because that’s the version it is not working in - but it was working in previous versions, as you stated.

Hopefully one of the developers can assist here.