Hi! When I tried to invoke the NVIDIA NIM API (hosted by NVIDIA, not me) via Langchain (using the meta/llama-3.1-70b-instruct mode), and parsing the output as structured, I always get this error:
Traceback (most recent call last):
File "/localhome/wtest/nv_wso copy.py", line 154, in <module>
agent.invoke(
File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1334, in invoke
for chunk in self.stream(
File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1020, in stream
_panic_or_proceed(all_futures, loop.step)
File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1450, in _panic_or_proceed
raise exc
File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/executor.py", line 60, in done
task.result()
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 26, in run_with_retry
task.proc.invoke(task.input, task.config)
File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2876, in invoke
input = context.run(step.invoke, input, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langgraph/utils.py", line 102, in invoke
ret = context.run(self.func, input, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/localhome/wtest/nv_wso copy.py", line 104, in respond
response = structured_llm.invoke(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2876, in invoke
input = context.run(step.invoke, input, config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 5092, in invoke
return self.bound.invoke(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 277, in invoke
self.generate_prompt(
File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 777, in generate_prompt
return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 634, in generate
raise e
File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 624, in generate
self._generate_with_cache(
File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 846, in _generate_with_cache
result = self._generate(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_nvidia_ai_endpoints/chat_models.py", line 289, in _generate
response = self._client.get_req(payload=payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_nvidia_ai_endpoints/_common.py", line 449, in get_req
response, session = self._post(self.infer_url, payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_nvidia_ai_endpoints/_common.py", line 346, in _post
self._try_raise(response)
File "/usr/local/lib/python3.12/site-packages/langchain_nvidia_ai_endpoints/_common.py", line 439, in _try_raise
raise Exception(f"{header}\n{body}") from None
Exception: [500] Internal Server Error
'bool' object has no attribute 'get'
RequestID: 75efc63a-f9c1-4891-b83a-c8a76987c2c8
Does anyone know how to fix this error? By the way, I also noticed while using NIM on the NVIDIA’s web interface and via API that tool calls takes so much time (28 seconds), and for example, while using Ollama, time is normal?!
Here’s also screenshot of Langsmith’s output for it:
If anyone knows how to solve this problems, please help. Thanks in advance!