Hi, thank you for your reply. I’ve tried the --duration= option and still can’t show the missing fileds.
My python script (the script path in the nsys command) is as follow:
import os
from multiprocessing import Process
def run_stress_test_client(port, script_path):
python_executable = "path/to/my/python"
client_command = [
python_executable, ONLINE_SCRIPT_PATH,
"--backend", "openai-chat",
"--base-url", f"http://localhost:{port}",
"--endpoint", "/v1/chat/completions",
...(other params)
]
os.execv(python_executable, client_command)
if __name__ == "__main__":
processes = []
for port in ["8000", "8001", "8002"...]:
# Each CUDA device corresponds to an online port, and we send requests to these ports at the same time
print(f"\n\t Running process for port: {port}\n")
process = Process(
target=run_stress_test_client,
args=(port, script_path),
)
process.start()
processes.append(process)
# wait until all process finished
for process in processes:
process.join()
print("All processes completed.")
The core content of “ONLINE_SCRIPT_PATH” in the above run_stress_test_client() function is as follow:
import asyncio
async def send_requests(
backend, url, port, model_id, input_requests, max_concurrency...
):
tasks: List[asyncio.Task] = []
async for request in request_list(...):
request_func_input = RequestFuncInput(model,prompt,url,port...)
tasks.append(
asyncio.create_task(_request_func(...))
)
outputs: List[RequestFuncOutput] = await asyncio.gather(*tasks)
# process outputs...
if __name__ == "__main__":
benchmark_result = asyncio.run(
send_requests(...)
)
I’ve found that if I don’t use asyncio, and modify the script called by ‘multiprocessing’ library into a totally offline script that use a local model, the problem will be solved, and all datafields will show fine.
But I want to use nsight on online serving cases. Could you give some suggestions on why this would happen, and how to use nsight on online serving cases correctly?
(I think my previous content may cause some misunderstandings, for I wasn’t much clear about the source of the question when I first commited the post. So I edited the first commited post content. So sorry for troubling you. )
Thank you again for your help.
