Login issue JWT on TAO-API with Jupyter

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) : rtx3090x2
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) : etc
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) : tao-api 4.0.2
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

jupyter notebook code.
model_name = “mask_rcnn” # FIXME1 (Add the model name from the above mentioned list)
workdir = “workdir_segmentation” # FIXME2
host_url = “http://external-ip-for-ingress-nginx:32080” # FIXME3 example: https://10.137.149.22:32334
ngc_api_key = “my-ngc-key” # FIXME4 example: zZYtczM5amdtdDcwNjk0cnA2bGU2bXQ3bnQ6NmQ4NjNhMDItMTdmZS00Y2QxLWI2ZjktNmE5M2YxZTc0OGyM
dataset_to_be_used = “default” # FIXME5 #default/custom; default for the dataset used in this tutorial notebook; custom for a different dataset

Exchange NGC_API_KEY for JWT

response = requests.get(f"{host_url}/api/v1/login/{ngc_api_key}", verify=False)
print(response.json())

but return response is just {}

check log for tao toolkit api app pod

kubectl logs -f tao-toolkit-api-app-pod-54c9c75fbc-l7wd2
Unauthorized: Credentials error: Invalid audience
192.168.35.143 - - [26/Jun/2023:06:49:50 +0000] “GET /api/v1/login/my-ngc-key HTTP/1.1” 401 83 “-” “python-requests/2.22.0”

So I have check auth_utils/credentials.py on tao-toolkit-api-app-pod-54c9c75fbc-l7wd2

try:
r = requests.get(‘https://authn.nvidia.com/token’, headers={‘Accept’: ‘application/json’, ‘Authorization’: 'ApiKey ’ + key})
if r.status_code != 200:
err = 'Credentials error: Invalid NGC_API_KEY: ’ + key
return creds, err
token = r.json().get(‘token’)
user = None
payload = {}
signing_key = __ngc_jwks_client.get_signing_key_from_jwt(token)
payload = jwt.decode(
token,
signing_key.key,
algorithms = [“RS256”]
)
user = uuid.uuid5(uuid.UUID(int=0), payload.get(‘sub’))
creds = {‘user_id’: str(user), ‘token’: token}
except Exception as e:
err = 'Credentials error: ’ + str(e)

ngc jwks client

__ngc_jwks_client = jwt.PyJWKClient(“https://authn.nvidia.com/pubJWKS”)

Error MSG :

<jwt.jwks_client.PyJWKClient object at 0x7f0c0bb64df0>
<jwt.api_jwk.PyJWK object at 0x7f0c0ac99df0>
Traceback (most recent call last):
File “test2.py”, line 15, in
payload = jwt.decode(
File “/home/surromind/.local/lib/python3.8/site-packages/jwt/api_jwt.py”, line 168, in decode
decoded = self.decode_complete(
File “/home/surromind/.local/lib/python3.8/site-packages/jwt/api_jwt.py”, line 136, in decode_complete
self._validate_claims(
File “/home/surromind/.local/lib/python3.8/site-packages/jwt/api_jwt.py”, line 205, in _validate_claims
self._validate_aud(payload, audience)
File “/home/surromind/.local/lib/python3.8/site-packages/jwt/api_jwt.py”, line 245, in _validate_aud
raise InvalidAudienceError(“Invalid audience”)
jwt.exceptions.InvalidAudienceError: Invalid audience

PyJWKClient has something problem??? or have I something mistake?

Really sorry for the inconvenient. There is change in ngc server. So, “cannot login the TAO API”. It is needed to change accordingly in tao api side. TAO team is working on that with high priority.

1 Like

How long time to wait NGC Server maintenance?

Actually the change is done on ngc server. But the change will cause the issue on TAO API side. So, the TAO API has to modify accordingly based on the change on NGC server. TAO team is still working on that. We will update to you once there is any more info. Thanks.

1 Like

Please use below workaround.

  1. $ kubectl edit services tao-toolkit-api-service
    (1 line added, and 1 line changed):

ports:
- name: api
nodePort: 31951
port: 8000
protocol: TCP
targetPort: 8000

type: NodePort

As seen as below,
image

  1. Then double check with $ kubectl get services

  2. Please note that, login API will still return {} an empty dict. That is expected.
    In the notebook, instead of calling the login API, you can just do :

import uuid
user_id = str(uuid.uuid4())
token = “whatever”

Also change the port of the host_url

Reference for this step.

I tried the method you suggested.

chnage to NodePort and Add 31951 port number

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.109.18.248 10.10.30.24 80:32080/TCP,443:32443/TCP 2d1h
kubernetes ClusterIP 10.96.0.1 443/TCP 2d1h
tao-toolkit-api-service NodePort 10.100.213.164 10.10.30.24 8000:31951/TCP 2d1h

Exchange NGC_API_KEY for JWT

response = requests.get(f"{host_url}/api/v1/login/{ngc_api_key}", verify=False)

print(response)

user_id = str(uuid.uuid4())
print(user_id)
token = “whatever”
print(token)

Set base URL

#base_url = f"{host_url}/api/v1/user/{user_id}"
base_url = f"http://10.10.30.24:31951/api/v1/user/{user_id}"
print(“API Calls will be forwarded to”,base_url)

headers = {“Authorization”: f"Bearer {token}"}
print(f’Headers : {headers}')

Output

<Response [401]>
78941a86-3aee-4ef8-b0b0-e44a2c9ff471
whatever
API Calls will be forwarded to http://10.10.30.24:31951/api/v1/user/78941a86-3aee-4ef8-b0b0-e44a2c9ff471
Headers : {‘Authorization’: ‘Bearer whatever’}

The part of logging in with ngc_api_key seems not to be used, and it seems to use a random user_id created with uuid.
While waiting, I think I figured out how to get the payload with the token I received using ngc_api_key from authn.nvidia.com/token.
However, I don’t know how to apply the code to tao-toolkit-api-service.

Test Code

import requests
import json
import jwt
import base64
import uuid

key = “ZGM3cjhlb3B0amNuZjM2YzBpNzZ0OGJtNTU6NzdjYzJhMGMtYmUwZC00M2Y5LWE2MDMtMWJkYTZlZTcyMDQz” # FIXME: use your latest NGC API KEY
#key = “ZGM3cjhlb3B0amNuZjM2YzBpNzZ0OGJtNTU6MjI0ZmEwZmMtNTQ4MS00NGI5LTkzNzAtNzEyNDRjZGEzZjg6” # FIXME: use your latest NGC API KEY
r = requests.get(‘https://authn.nvidia.com/token’, headers={‘Accept’: ‘application/json’, ‘Authorization’: ‘ApiKey ’ + key})
print(r.status_code)
token = r.json().get(‘token’)
print(f’TOKEN: \n{token}’)
header_base64, payload_base64, signature = token.split(‘.’)
payload = json.loads(base64.urlsafe_b64decode(payload_base64 + ‘=’ * (-len(payload_base64) % 4)))
print(payload)
user_id = uuid.uuid5(uuid.UUID(int=0), payload.get(‘sub’))
print(f’USER ID : {user_id}')

Output

200
TOKEN:
eyJraWQiOiJFUkNPOklCWFY6TjY2SDpOUEgyOjNMRlQ6SENVVToyRkFTOkJJTkw6WkxKRDpNWk9ZOkRVN0o6TVlVWSIsImFsZyI6IlJTMjU2In0.eyJzdWIiOiJkYzdyOGVvcHRqY25mMzZjMGk3NnQ4Ym01NSIsImF1ZCI6Im5nYyIsImFjY2VzcyI6W10sImlzcyI6ImF1dGhuLm52aWRpYS5jb20iLCJvcHRpb25zIjpbXSwiZXhwIjoxNjg3OTM4MTc5LCJpYXQiOjE2ODc5Mzc1NzksImp0aSI6ImU5MjU4ODIwLWFkYjctNDc4ZC1iYTdmLTkyNDZhNDdmMWJkOCJ9.CffY_pk9t88S2m1dHnTa2uFJBE3srzHZyqKEW2gtvfB0UokTzONxoTHvZWnrNnIO_n5CIhTdR1R2-SjmVOPgdyf1Bg0ONkIpBsfUOMbgNAmO_Kk4PGvdaTQot24Lss8-ZMrkXnKsHgXqT8z9An-gyK3ElxZYcWO1CBXTvLWYwxcoVb-PdcJGpeViCjSQsgMhbkY1BiqRcIHYFXYHTP8jCU7P3_cyiijoaaJRyLYSAEs81cKPGVF_0xOzs1GWB8zBwQXtXskX2vMqwHWj726WgAsbj24cwON8QNgY9PpnipqF3g7eyxn9CbGz7EHHr7f7OmJAXZ8DHZfsOIj7Qt7BGVEu4lp9MaGHPC-taxF-3Sd8x-Sq649zbnbPiWn-EIimFknW_jjvKu6CFf0ijppc9XJG2AOyFo5T6FVfAYmLojjRAYCx_TFUfAGfvv2Y5y51OfZZwCx8a7Q02Jlp-OwDSr7WL3ftuECwm0HJRP6lSDWuOiu-_aKDmQZWcMgVH9M1KfxKWLkSOtS-eg6YsGdnQwB4Kxv6QYv5t4yrPYai8ntwIPyHZBuixmx-rsT9IeccXunWf3C4xhGo8bBQVqMJqrA3avtRHD_GL6wwGhqdhPxZc69zn7IMlaMjnh1I7VdLDVzSM22gvsSqA6pVQnVK9cQ2xAYHkvriSLDagWnxEnE
{‘sub’: ‘dc7r8eoptjcnf36c0i76t8bm55’, ‘aud’: ‘ngc’, ‘access’: , ‘iss’: ‘authn.nvidia.com’, ‘options’: , ‘exp’: 1687938179, ‘iat’: 1687937579, ‘jti’: ‘e9258820-adb7-478d-ba7f-9246a47f1bd8’}
USER ID : 53db0338-931e-5ee9-b276-b7c631735389

It seems that the token is not encoded with the signing_key obtained from authn.nvidia.com/pubJWKS, but encoded with base64.

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

That’s expected. In this workaround, ngc_api_key will not be used now.
It is not needed to test the API key.

You can just follow the steps I posted.

I’m still facing this issue. I’m assuming we still have to use this workaround?

Yes, new TAO5.0 can deprecate the workaround.

The 5.0 notebook also gives me the same error. Is it that I have to run $ bash setup.sh install again with a different setup?

Yes, since the tao-api is updated to 5.0 version.

How do I migrate from v4.0.2 to v.5.0 without breaking things?
I tried to run bash setup.sh install from the same conda environment on the same server but from the v5.0 folder, it tries to uninstall the existing setup, reboots the server, and then does nothing after that.

Here are the logs

Could you please try again? If there is still an issue, please open a new forum topic for tracking. Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.