'Building RAG Agents for LLMs' Bad Request Error During Assessment

AI & Data Science AI Foundation Models and Endpoints
1

I’ve been working on the assessment for quite a while and I am able to get all routes running in a separate file:

Screenshot 2024-06-11 at 4.06.04 PM
Screenshot 2024-06-11 at 4.06.04 PM1920×1159 315 KB

Additionally, I copied the code from frontend_block.py so that I could test whether rag_chain.invoke & rag_chain.stream were working properly. I was able to confirm that they were:

Screenshot 2024-06-12 at 10.40.48 PM
Screenshot 2024-06-12 at 10.40.48 PM2498×1390 355 KB

Despite this, pressing evaluate presents the below error…

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 282, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 271, in wrap
    await func()
  File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 221, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 568, in receive
    await self.message_event.wait()
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f62051b2790

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
  |     await route.handle(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
  |     await self.app(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
  |     await response(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 268, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 271, in wrap
    |     await func()
    |   File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 251, in stream_response
    |     async for data in self.body_iterator:
    |   File "/usr/local/lib/python3.11/site-packages/langserve/api_handler.py", line 1121, in _stream
    |     async for chunk in self._runnable.astream(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2900, in astream
    |     async for chunk in self.atransform(input_aiter(), config, **kwargs):
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2883, in atransform
    |     async for chunk in self._atransform_stream_with_config(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1980, in _atransform_stream_with_config
    |     chunk: Output = await asyncio.create_task(  # type: ignore[call-arg]
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2853, in _atransform
    |     async for output in final_pipeline:
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/output_parsers/transform.py", line 60, in atransform
    |     async for chunk in self._atransform_stream_with_config(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1944, in _atransform_stream_with_config
    |     final_input: Optional[Input] = await py_anext(input_for_tracing, None)
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 62, in anext_impl
    |     return await __anext__(iterator)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 97, in tee_peer
    |     item = await iterator.__anext__()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4748, in atransform
    |     async for item in self.bound.atransform(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1334, in atransform
    |     async for output in self.astream(final, config, **kwargs):
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 319, in astream
    |     raise e
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 297, in astream
    |     async for chunk in self._astream(
    |   File "/dli/ai-endpoints/langchain_nvidia_ai_endpoints/chat_models.py", line 222, in _astream
    |     async for response in self.get_astream(inputs=inputs, **kwargs):
    |   File "/dli/ai-endpoints/langchain_nvidia_ai_endpoints/_common.py", line 530, in get_req_astream
    |     self._try_raise(response)
    |   File "/dli/ai-endpoints/langchain_nvidia_ai_endpoints/_common.py", line 279, in _try_raise
    |     response.raise_for_status()
    |   File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1070, in raise_for_status
    |     raise ClientResponseError(
    | aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('http://llm_client:9000/v1/chat/completions')
    +------------------------------------
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 282, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 271, in wrap
    await func()
  File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 221, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 568, in receive
    await self.message_event.wait()
  File "/usr/local/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f62044ad050

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
  |     await route.handle(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
  |     await self.app(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
  |     raise exc
  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
  |     await response(scope, receive, send)
  |   File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 268, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 271, in wrap
    |     await func()
    |   File "/usr/local/lib/python3.11/site-packages/sse_starlette/sse.py", line 251, in stream_response
    |     async for data in self.body_iterator:
    |   File "/usr/local/lib/python3.11/site-packages/langserve/api_handler.py", line 1121, in _stream
    |     async for chunk in self._runnable.astream(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2900, in astream
    |     async for chunk in self.atransform(input_aiter(), config, **kwargs):
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2883, in atransform
    |     async for chunk in self._atransform_stream_with_config(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1980, in _atransform_stream_with_config
    |     chunk: Output = await asyncio.create_task(  # type: ignore[call-arg]
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2853, in _atransform
    |     async for output in final_pipeline:
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/output_parsers/transform.py", line 60, in atransform
    |     async for chunk in self._atransform_stream_with_config(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1944, in _atransform_stream_with_config
    |     final_input: Optional[Input] = await py_anext(input_for_tracing, None)
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 62, in anext_impl
    |     return await __anext__(iterator)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 97, in tee_peer
    |     item = await iterator.__anext__()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4748, in atransform
    |     async for item in self.bound.atransform(
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1334, in atransform
    |     async for output in self.astream(final, config, **kwargs):
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 319, in astream
    |     raise e
    |   File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 297, in astream
    |     async for chunk in self._astream(
    |   File "/dli/ai-endpoints/langchain_nvidia_ai_endpoints/chat_models.py", line 222, in _astream
    |     async for response in self.get_astream(inputs=inputs, **kwargs):
    |   File "/dli/ai-endpoints/langchain_nvidia_ai_endpoints/_common.py", line 530, in get_req_astream
    |     self._try_raise(response)
    |   File "/dli/ai-endpoints/langchain_nvidia_ai_endpoints/_common.py", line 279, in _try_raise
    |     response.raise_for_status()
    |   File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1070, in raise_for_status
    |     raise ClientResponseError(
    | aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('http://llm_client:9000/v1/chat/completions')

Could I have some suggestions on how to resolve this?

2

Hey @hoskin41! Nice diligence on checking out the issue! It is weird that your local tests seem to be working actually…

A few debugging steps:

  • Feel free to send me the docs you tried out (if you imported a custom paper).
  • This may be counter-intuitive, but can you make sure that you have the “END” truncation strategy on your retriever system.
  • Which model are you using? It is Mixtral-8x22b?
  • Does the docker-router log of llm_client show anything?
3

Good news! I was able to solve this problem by reading through llm_client/client_server_manual.py.

I was using mistralai/mixtral-8x22b-instruct-v0.1. This model is used throughout the course, such as in Notebook 3, but it is commented out from the dictionary, app_state. You must select a model that is listed within app_state["models"].

1 Like