GitHub - NVIDIA/workbench-llamafactory - can't build container - 403 getting python-cuda122:1.03

AI & Data Science NVIDIA AI Workbench
1

Trying to run this project GitHub - NVIDIA/workbench-llamafactory: This is an NVIDIA AI Workbench example project that demonstrates an end-to-end model development workflow using Llamafactory.

The project fails when trying to build the container with a 403 on an artifact download. Normally there is a secret in the Environment tab where you can put your keys. I don’t see that for this project and it isn’t in .projects/spec.yaml. There is a Huggingface secret to get access to the llama model.

Instructions followed are here RTX-AI-Toolkit/tutorial-llama3-finetune.md at main · NVIDIA/RTX-AI-Toolkit · GitHub

--------------------
   1 | >>> FROM nvcr.io/nvidia/ai-workbench/python-cuda122:1.0.3
   2 |     
   3 |     WORKDIR /opt/project/build/
--------------------
ERROR: failed to solve: nvcr.io/nvidia/ai-workbench/python-cuda122:1.0.3: failed to resolve source metadata for nvcr.io/nvidia/ai-workbench/python-cuda122:1.0.3: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://nvcr.io/proxy_auth?scope=repository%3Anvidia%2Fai-workbench%2Fpython-cuda122%3Apull: 403 Forbidden
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Containerfile
#1 transferring dockerfile: 1.56kB done
#1 DONE 0.0s
#2 [auth] nvidia/ai-workbench/python-cuda122:pull token for nvcr.io
#2 DONE 0.0s
#3 [internal] load metadata for nvcr.io/nvidia/ai-workbench/python-cuda122:1.0.3
#3 ERROR: failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://nvcr.io/proxy_auth?scope=repository%3Anvidia%2Fai-workbench%2Fpython-cuda122%3Apull: 403 Forbidden
------
 > [internal] load metadata for nvcr.io/nvidia/ai-workbench/python-cuda122:1.0.3:

Is there a CLI to create credentials on disk?

Opened No place to provide nvcr.io artifact credentials. No secret for NVIDIA keys. · Issue #2 · NVIDIA/workbench-llamafactory · GitHub

Please tick the appropriate box to help us categorize your post
Bug or Error
Feature Request
Documentation Issue
Other

2

This was on a remote machine so I had to SSH into it to create my oauthoken credentials.

  1. ssh to the remote 1. machine
  2. docker logout nvcr.io
  3. docker login nvcr.io
  4. Username $oauthtoken
  5. Password my nvapi... key

I wish there was some way to run this remote login stuff through the Workbench.

3

Hi Joe

You shouldn’t have to do this.

This is either a bug or some sort of edge case.

The relevant team for the Llamafactory questions will be weighing in soon.

4

After some offline discussion…

It may be that I had previously cached NVidia credentials on that machine where I ran some docker commands while testing NVidia images outside of the Workbench.

This thread can be considered closed.