Using llama_index Python modules in Omniverse

Hello,

I am trying to integrate some AI scripts into an Omniverse extension, but ran into an issue trying to import the llama_index Python modules. I am able to install the llama_index modules by listing them in the [python.pipapi] as such:

[python.pipapi]
requirements = [
    "llama-index",
    "llama-index-llms-ollama",
    "llama-index-embeddings-ollama",
    "llama-index-vector-stores-chroma",
    "chromadb",
    "sqlalchemy"
]
use_online_index = true

However, when I try to import llama_index in Omniverse (either through an extension or the Script Editor). I get the following error:

ImportError: cannot import name 'async_sessionmaker' from 'sqlalchemy.ext.asyncio' (<redacted>/_build/windows-x86_64/release/extscache/omni.services.pip_archive-0.15.0+107.0.3.wx64.cp311/pip_prebundle/sqlalchemy/ext/asyncio/__init__.py)

Any advice?

Well it seems it has an issue with that specific name ‘async_sessionmaker’ from ‘sqlalchemy.ext.asyncio’, so I would open that up and examine that more. Maybe it has a bad naming convention. I can try to ask the engineers but this is really outside the Omniverse scope.

What are you trying to do with Omniverse kit and AI?

I am trying to use the llama_index Python modules to experiment implementing a custom RAG model. I am no expert in that, so it is a learning experience as well.

I was able to make it work by manually installing SQLAlchemy into the Kit’s Python environment. From what I observed, Omniverse Kit already includes SQLAlchemy, likely for its own internal use, but it might be a different version than what llama_index requires. After installing the latest version with pip, everything started working as expected.

It seems that the manually installed version takes precedence over the bundled one, at least in this context. So far, I haven’t noticed any side effects on other Omniverse features.

You know that we have our very own ChatUSD AI model in Omniverse already? Just install that. If you are on kit 106.5 you can look for ChatUSD. It will be coming for kit 107 soon.

I’ll check it out, thanks!