TAO 4.0.0
Hi I’m using TAO with a local k8 cluster, for telemetry I use clearml (super easy to setup becase we can include all the data in the values.yaml of the helm chart) like this
# Optional MLOPS setting for ClearML
clearMlWebHost: https://app.clear.ml
clearMlApiHost: https://api.clear.ml
clearMlFilesHost: https://files.clear.ml
clearMlApiAccessKey: <ACCESS-KEY>
clearMlApiSecretKey: <API-KEY>
and by configuring the spec
# get default specification schema for training.
endpoint = f"{base_url}/model/{model_ID}/specs/train/schema"
response = requests.get(endpoint, headers=headers, verify=rootca)
specs = response.json()["default"]
specs["training_config"]["visualizer"]["clearml_config"] = {}
specs["training_config"]["visualizer"]["clearml_config"]["project"] = "my_project"
specs["training_config"]["visualizer"]["clearml_config"]["tags"] = ["training", "tao_toolkit"]
specs["training_config"]["visualizer"]["clearml_config"]["task"] = "training_experiment_1"
This works!
But I’d like to try wandb in a similar manner
I have a wandb server deployed in the local k8 cluster and to sttream telemetry I have to manually stream log entries by repeatedly runing REST api calls to retrive data from TAO and calling
wandb.log({"key1":value1, "key2", value2 ... })
to send data to wandb local server.
Is there a way to run
wandb.login(key="local-API-KEY", host="http://my.local.domin:my_local_domian_port_number", relogin=True)
command within the client that is in the k8 pod? so all I have to do is inlcude
...
specs["training_config"]["num_epochs"] = num_epochs
specs["training_config"]["visualizer"]["enabled"] = True
# add the wandb_config section
specs["training_config"]["visualizer"]["wandb_config"] = {}
specs["training_config"]["visualizer"]["wandb_config"]["project"] = "my_net"
specs["training_config"]["visualizer"]["wandb_config"]["tags"] = ["training", "tao_toolkit"]
specs["training_config"]["visualizer"]["wandb_config"]["notes"] = "training_experiment_1"
Cheers,
Ganindu.