[SUPPORT] Workbench Example Project: Mistral Finetune

AI & Data Science NVIDIA AI Workbench
1

Hi! This is the support thread for the Mistral Finetune Example Project on GitHub. Any major updates we push to the project will be announced here. Further, feel free to discuss, raise issues, and ask for assistance in this thread.

Please keep discussion in this thread project-related. Any issues with the Workbench application should be raised as a standalone thread. Thanks!

2

I have installed NVidia AI Workbench and am attempting to run the mistral-finetune jupyter notebook.

The notebook is stuck on Kernel Connecting on Step 3: Load In The Base Model.

and then has the error

ReadTimeoutError: HTTPSConnectionPool(host=‘cdn-lfs.huggingface.co’, port=443): Read timed out.

The code is below:

%%capture

model_id = "mistralai/Mistral-7B-v0.1"
bb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bb_config)

I am using Ubuntu 22.04 and NVidia AI Workbench 0.28.29-x86_64

3

Could be a variety of different issues, here is thread with a similar issue. Could be a variety of issues such as a temporary Huggingface Hub outage. Let me know if it persists.

The Mistral project has been refreshed for GTC with updated packages and support for quantization. See if the updated version of the project helps.

Lastly, we also released the GA version of AIWB, should be version 0.44.8. Try updating AIWB to the latest version and see if it helps.

4

We have refreshed the Mistral finetuning project for GTC.

  • Updating packages
  • Adding support for quantization levels
  • Added a mount to save your finetuned model to the underlying host system
  • bug fixes and improvements
5

Hugging Face introduced a breaking change a few days ago to gate the Mistral-7B-v0.1 base model used in this project.

Release Notes (04/24/2024)

  1. Project secret has been added to the project for authentication to the Mistral model.

    • The user will need to supply their Hugging Face Hub token (with the proper permissions) to AI Workbench before starting the project.

  2. Notebook contents have been updated to use the secret for authentication when pulling the model.

  3. README contents updated, including a prerequisite section to create an access token on HF and accept the T&C on the model card.

6

(6/4/2024) Project spec has been updated for AI Workbench version 0.50.17.