Nvidia_tao_core.microservices.app - ERROR - Exception on /api/v2/tao_api_notebooks.zip [GET]

Accelerated Computing Intelligent Video Analytics TAO Toolkit
,
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) - T4
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) - NA
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) - NA
• Training spec file(If have, please share here) - NA
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.) - <host_url>//api/v2/tao_api_notebooks.zip in browser

2026-01-07 09:08:02,550 - nvidia_tao_core.microservices.app - ERROR - Exception on /api/v2/tao_api_notebooks.zip [GET]
File “/opt/venv/cosmos_rl/lib/python3.12/site-packages/flask/app.py”, line 1511, in wsgi_app
response = self.full_dispatch_request()
File “/opt/venv/cosmos_rl/lib/python3.12/site-packages/flask/app.py”, line 919, in full_dispatch_request
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/venv/cosmos_rl/lib/python3.12/site-packages/flask/app.py”, line 917, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/venv/cosmos_rl/lib/python3.12/site-packages/flask/app.py”, line 902, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/opt/venv/cosmos_rl/lib/python3.12/site-packages/nvidia_tao_core/microservices/decorators.py”, line 40, in decorated_function
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File “/opt/venv/cosmos_rl/lib/python3.12/site-packages/nvidia_tao_core/microservices/blueprints/v2/admin.py”, line 130, in download_folder
shutil.make_archive(“/tmp/tao_api_notebooks”, ‘zip’, “/shared/notebooks/”)
File “/usr/lib/python3.12/shutil.py”, line 1144, in make_archive
stmd = os.stat(root_dir).st_mode
^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ‘/shared/notebooks/’
127.0.0.1 - - [07/Jan/2026:09:08:02 +0000] “GET /api/v2/tao_api_notebooks.zip HTTP/1.1” 500 334 “-” “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0”

Please help resolve this as the notebooks can’t be downloaded at <host_url>/tao_api_notebooks.zip

Thanks & regards,

Projjwal

3

Could you please run from a shell inside the TAO microservice container/pod:
mkdir -p /shared/notebooks
chmod 755 /shared/notebooks

Then retry:
<host_url>/api/v2/tao_api_notebooks.zip
or
<host_url>/tao_api_notebooks.zip

4

I executed the command against tao-api-app-pod-56b454f945-l7r89

I executed the command against tao-api-app-pod-56b454f945-l7r89 pod and it worked fine as I checked the dir and its permission
later I tried both below
<host_url>/api/v2/tao_api_notebooks.zip
it returned
‘404 not found’

<host_url>/tao_api_notebooks.zip
it returned
{
“code”: 404,
“name”: “Not Found”,
“description”: “The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.”
}

attached here’s the log from tao-api-app-pod-56b454f945-l7r89 pod and also below is code from the templates\ingress-notebooks.yaml used for deployment

# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved.

#

# Licensed under the Apache License, Version 2.0 (the “License”);

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

# http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an “AS IS” BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

{{- if .Values.ingressEnabled }}

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

name: {{ .Release.Name }}-ingress-notebooks

namespace: {{ .Release.Namespace }}

annotations:

nginx.ingress.kubernetes.io/proxy-buffer-size: 128k

nginx.ingress.kubernetes.io/proxy-http-version: "1.1"

nginx.ingress.kubernetes.io/connection-proxy-header: ""

{{- if .Values.tlsSecret }}

nginx.ingress.kubernetes.io/ssl-redirect: "true"

nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

nginx.ingress.kubernetes.io/server-snippet: |

  error_page 497 https://$server_name:$server_port$request_uri;

{{- end }}

nginx.ingress.kubernetes.io/use-regex: "true"

nginx.ingress.kubernetes.io/rewrite-target: /$1/tao_api_notebooks.zip

{{- if .Values.corsOrigin }}

nginx.ingress.kubernetes.io/enable-cors: "true"

nginx.ingress.kubernetes.io/cors-allow-origin: {{ .Values.corsOrigin }}

{{- end }}

nginx.ingress.kubernetes.io/configuration-snippet: |

  more_set_headers "X-Frame-Options: SAMEORIGIN";

  more_set_headers "X-XSS-Protection: 1; mode=block";

{{- if .Values.tlsSecret }}

  more_set_headers "Strict-Transport-Security: max-age=31536000; includeSubDomains; preload";

{{- end }}

spec:

ingressClassName: nginx

{{- if .Values.tlsSecret }}

tls:

  • secretName: {{ .Values.tlsSecret }}

{{- if .Values.host }}

hosts:

- {{ .Values.host }}

{{- end }}

{{- end }}

rules:

  • http:

    paths:

    • path: /{{ .Release.Namespace }}/tao_api_notebooks.zip

      pathType: ImplementationSpecific

      backend:

      service:

      name: {{ .Release.Name }}-service

port:

  number: 8000

{{- if eq .Release.Namespace “default” }}

  - path: /tao_api_notebooks.zip

    pathType: ImplementationSpecific

    backend:

      service:

        name: {{ .Release.Name }}-service

        port:

          number: 8000

{{- end }}

{{- if and .Values.host }}

host: {{ .Values.host }}

{{- end }}

{{- end }}

Please suggest further step to help resolve the issue

Thanks & regards,

Projjwal

notebook_zip_error_log.txt (1.7 MB)

5

The notebooks for tao-api are all inside tao_tutorials/notebooks/tao_api_starter_kit at main · NVIDIA/tao_tutorials · GitHub. May I know which .ipynb file you are running? Thanks.

6

Please try below to narrow down.

  1. Run curl inside the Pod. To check if it can return 200.
kubectl exec -it tao-api-app-pod-56b454f945-l7r89 -- bash
curl -v http://127.0.0.1:9000/api/v2/tao_api_notebooks.zip -o /tmp/test_notebooks.zip

The official TAO 5.5 ingress-notebooks.yaml （tao-core/chart/templates/ingress-notebooks.yaml at tao_5.5_release · NVIDIA/tao-core · GitHub ）uses nginx.ingress.kubernetes.io/rewrite-target: /tao_api_notebooks.zip and does not contain $1 in the rewrite-target.
If your current Ingress uses /$1/tao_api_notebooks.zip while the path does not define any capture group (…) , then $1 will always be empty and the rewrite rule will not behave as intended.
To align with the official chart and avoid rewrite issues, please change your Ingress annotation to: nginx.ingress.kubernetes.io/rewrite-target: /tao_api_notebooks.zip and redeploy, then retry <host_url>/tao_api_notebooks.zip.