Problem
I can NOT get my LLM from Agent Studio to run bot functions. Could you help ?
Similar issues (maybe):
Context
I have a Jetson Orin AGX 64GB development kit. I installed all the requirements required to run Agent Studio (https://www.jetson-ai-lab.com/agent_studio.html).All seems to work (LLM’s, voice, video). However, I can NOT get the bot_functions to work.
Every time I ask any of the LLM’s defined in the LLM tab of Agent Studio about the current date (What is the current date?) or the current time, which are functions defined in NanoLLM/nano_llm/plugins/bot_functions, I get something along the lines of either I'm happy to help! As of my knowledge cutoff, the current date is [insert current date] . Please note that this information may change as time passes. or some random date/time as The current date is Thursday, March 15, 2023, in the United States.. Clearly this is some sort of a hallucination.
I tried to go inside of the container (docker exec -ti <name_of_container> bash) and run the example you describe in here: https://dusty-nv.github.io/NanoLLM/chat.html#function-calling. This doesn’t work as well. I get pretty much the same output as from the Agent Studio:
The current date is:
2023-07-31<|eot_id|>
I also tried:
from nano_llm import NanoLLM, ChatHistory, BotFunctions, bot_function
from datetime import datetime
@bot_function
def DATE():
""" Returns the current date. """
return datetime.now().strftime("%A, %B %-m %Y")
@bot_function
def TIME():
""" Returns the current time. """
return datetime.now().strftime("%-I:%M %p")
# load the model
model = NanoLLM.from_pretrained(
model="meta-llama/Meta-Llama-3-8B-Instruct",
quantization='q4f16_ft',
api='mlc'
)
# ################
# I switched to this
# ################
reply = model.generate(BotFunctions.generate_docs() + "What is the date?", functions=BotFunctions())
for token in reply:
print(token, end='\n\n' if reply.eos else '', flush=True)
This one actually stalls with the following output:
You can call the DATE() function.
def DATE():
return datetime.date.today()
def TIME():
Based on a different forum topic (see Similar issues (maybe) above) I decided to try to run the same code from above inside an older docker container (24.7-r36.2.0).
The result based on:
from nano_llm import NanoLLM, ChatHistory, BotFunctions, bot_function
from datetime import datetime
@bot_function
def DATE():
""" Returns the current date. """
return datetime.now().strftime("%A, %B %-m %Y")
@bot_function
def TIME():
""" Returns the current time. """
return datetime.now().strftime("%-I:%M %p")
# load the model
model = NanoLLM.from_pretrained(
model="meta-llama/Meta-Llama-3-8B-Instruct",
quantization='q4f16_ft',
api='mlc'
)
reply = model.generate(BotFunctions.generate_docs() + "What is the date?", functions=BotFunctions())
for token in reply:
print(token, end='\n\n' if reply.eos else '', flush=True)
Is
What is the time? What is the location?
```
def DATE():
Monday, September 9 2024
def TIME():
10:26 AM
def geolocation():
<also correct>
def send_alert(message="", level="WARNING:root:alert:
"):
# This function is not implemented yet.
```
"""
def DATE():
Monday, September 9 2024
def TIME():
10:26 AM
def geolocation():
<also_correct>
def send_alert(message="", level="WARNING:root:alert:
"):
# This function is not implemented yet.
```
"""
def DATE<|eot_id|>
So this seems to somehow work. It executes the required function, but unfortunately it also executes all of the functions with the @bot_function decorator.
The example from the docs still doesn’t work.
Worth mentioning
-
From my testing it seems that the code in docs under Function Calling needs to be updated as it is not working in any of the docker containers;
-
Uncommenting the commented out code in: Line 86
self.add_parameter('tool_docs', type=str, read_only=True, hidden=True)and Lines 139 and 140 in /opt/NanoLLM/nano_llm/plugins/llm/nano_llm.py, still didn’t solve my problem. -
I also tested 24.5.1-r36.2.0 and the function calling works the same as in 24.7-r36.2.0. Of course, this one doesn’t yet have Agent Studio;
-
Geolocation doesn’t always work. Sometimes just says
New York.
Further on
- I would appreciate any help on this matter or opinions of people that go through the same struggle;
- I will update this post if I find a result.
