How to load custom dataset

AI & Data Science NVIDIA AI Workbench
1

image
image1457×462 25.4 KB

I have my own custom dataset, how to select and load them ? here is a screenshot of my LlamaFactory, i try to find my custom dataset in data dir but didn’t see where I can select the location.

Please describe your issue or request: (tick the boxes after creating this topic):

Please tick the appropriate box to help us categorize your post
Bug or Error
Feature Request
Documentation Issue
Other

2

Hi. Sorry for taking so long to reply. I was waiting on the team that built that project to handle this but…it seems I need to do it!

Remember that Workbench handles git repositories and containerized development environments.

To add data/files you need add them into the project repo. There are many ways to do this.

  • The “easiest” way is the file browser in the relevant project tab in the AIWB UI.
    • You can drag and drop your data into the appropriate folder, I’m assuming that is data. Keep in mind the data folder is versioned by git-lfs, and so if you sync to GitHub you may be charged for lfs storage.
  • If you want to avoid that, then you can still add the data to the repo but you need to make sure it goes into an untracked location in the project repo or the container file system. There are a few ways to do this:
    • Put it in data/scratch/. This folder is not tracked by Git.
    • Make a new folder in the project repo, <your_new_folder>, add the data two it and then go to the changes tab and select git-ignore for that folder.
    • Create a host mount in the container file system but not in the project folder and connect it to the data on disc.

Let us know if this works.

3

for the first method , drag and drop my data file into data folder in AIWB UI, then what do I need to do for my data file to appear in Llamafactory in my browser?

4

Good question. I haven’t used that project yet. Let me ping the team that built it.

5

thanks. please let me know

6

any update?