[SUPPORT] Workbench Example Project: Llama 2 Finetune

AI & Data Science NVIDIA AI Workbench
1

Hi! This is the support thread for the Llama 2 Finetune Example Project on GitHub. Any major updates we push to the project will be announced here. Further, feel free to discuss, raise issues, and ask for assistance in this thread.

Note: You need to signup for the NeMo Frameworks GA to access the container used in the Project.

Please keep discussion in this thread project-related. Any issues with the Workbench application should be raised as a standalone thread. Thanks!

2

Is this project in place to augment the SDXL project, or is it specifically set up as a standalone?

3

Build failed with this error: STEP 1/18: FROM nvcr.io/ea-bignlp/ga-participants/nemofw-training:23.08.03
Trying to pull nvcr.io/ea-bignlp/ga-participants/nemofw-training:23.08.03…
Error: creating build container: initializing source docker://nvcr.io/ea-bignlp/ga-participants/nemofw-training:23.08.03: reading manifest 23.08.03 in nvcr.io/ea-bignlp/ga-participants/nemofw-training: authentication required

4

Hi Brian,

Check the following:

  • Do you have an NGC api key configured in workbench?
  • Did you sign up for the GA program to get access to this container?

Tyler

5

This Project is not related to the SDXL example. It should be “stand alone” in that you can use it without needing anything else.

1 Like
6

Yes and no. I couldn’t find the second item.

7

Found the proper link to be able to log into the NGC.

8

I am trying to to run this projects on remote machine… cloned it successfully, but stuck on how and where to download the model? I have credentials (integrations) created in my local machine…I have access to HF model below

but what command and from what machine I need to type to download it?
the instruction in the ReadMe shows below but it’s not working!!
git lfs clone https://huggingface.co//

which I substituted with
git lfs clone https://huggingface.co/hijer/https://huggingface.co/meta-llama/Llama-2-7b-hf

9

Yes. The way the project is currently written is confusing. Note that your namespace hijer shouldn’t be in the URL.

  1. Then, go to the command line for Workbench, activate the proper location, and open the Project with:
    nvwb activate <location_name>
    nvwb open <project_name>
  2. Then, attach to the Project container on the remote host with
    nvwb attach
  3. Your prompt will change and you are in the container on the remote.
  4. Then you can try the clone command, and you may need to add your HF token when it prompts you.

I will have the person who created the Project respond here with other suggestions.

10

Received the following error message when trying to build this tutorial on the DGX Spark

#7 0.118 exec /bin/bash: exec format error

#0 building with “default” instance using docker driver

#1 [internal] load build definition from Containerfile

#1 transferring dockerfile: 1.55kB done

#1 DONE 0.0s

#2 [internal] load metadata for nvcr.io/nvidia/nemo:24.01.framework

#2 DONE 0.7s

#3 [internal] load .dockerignore

#3 transferring context: 2B done

#3 DONE 0.0s

#4 [ 1/17] FROM nvcr.io/nvidia/nemo:24.01.framework@sha256:e6af316b86c8eb1cfc772aa19ec39bec3a3007e485c0e6cc9e317e031a1d3686

#4 DONE 0.0s

#5 [ 2/17] WORKDIR /opt/project/build/

#5 CACHED

#6 [internal] load build context

#6 transferring context: 137B done

#6 DONE 0.0s

#7 [ 3/17] RUN groupadd -g 1000 workbench || true

#7 0.118 exec /bin/bash: exec format error

#7 ERROR: process “/bin/bash -c groupadd -g 1000 workbench || true” did not complete successfully: exit code: 255

------

> [ 3/17] RUN groupadd -g 1000 workbench || true:

0.118 exec /bin/bash: exec format error

------

1 warning found (use docker --debug to expand):

- InvalidBaseImagePlatform: Base image nvcr.io/nvidia/nemo:24.01.framework was pulled with platform “linux/amd64”, expected “linux/arm64” for current build (line 1)

Containerfile:9

--------------------

7 | USER root

8 |

9 | >>> RUN groupadd -g 1000 workbench || true

10 |

11 | RUN useradd -u 1000 -g 1000 -rm -d /home/workbench -s /bin/bash workbench || usermod -l workbench $(getent passwd 1000 | cut -d: -f1)

--------------------

ERROR: failed to build: failed to solve: process “/bin/bash -c groupadd -g 1000 workbench || true” did not complete successfully: exit code: 255