Bot Functions Infinite Loop

kevinaseegobin · September 24, 2024, 8:11am

I am trying to use bot_functions for nano_llm on jetson orin agx but it does not exit the function neither on a standalone script or agent studio. I am running the latest version of jetpack for the nano_llm container

AastaLLL · September 25, 2024, 6:10am

Hi,

Could you share more details about how you use it?

You can find an example of Bot Functions usage below:

github.com

dusty-nv/NanoLLM/blob/main/nano_llm/chat/main.py#L125


      
          
          if not args.disable_stats:
              print_table(model.stats)
              print('')
          
          # save the output and kv cache
          chat_history.append('bot', reply)
          
          # run tools
          if args.enable_tools:
              tool_response = BotFunctions.run(reply.text, template=chat_history.template)
              if tool_response:
                  chat_history.append('tool_response', tool_response)
                  cprint(tool_response, 'yellow')

Thanks.

kevinaseegobin · September 25, 2024, 8:06pm

Yes i can. In Agent studio, when i connect the clock tool (a bot function), it does not reply with the time when asked for the time.

I have also tried using the sample code given on Chat — NanoLLM 24.7 documentation

Previously, it seemed model dependent where the function would be called but never exited on LLama3 and never called on VILA but now it is not being called by LLama3 either.

I will also try this example

dbvasilache_nvidia · September 30, 2024, 11:00am

I have something similar. I created this post: Function calling on Jetson Orin - Unable to run `bot_functions` in the latest dustynv/nano_llm docker container (dustynv/nano_llm:24.8-r36.3.0).

Will update there if I find something.