How to extract textures path list from the nucleus

Hi,
i want to generate arround 5000 randomized materials. For that i want to use all the textures provided in the nucleus under omniverse://localhost/NVIDIA/Materials/2023_3 and its subfolders . I tried with rep.utils.get_files_group, but hadn’t much succses. Does anybody have an a good way to do this? The output should be a list with the paths to all textures (.JPEG, .JPG, .PNG).

Could you provide the code you are trying to use with rep.utils.get_files_group?

Hi @zollfnn

There’s probably a few ways to achieve this. I was working on something a bit like this as an example (though not complete yet). I’ll post the code here, as it might point you in the right direction.

The main issue was that I haven’t been able to randomize each texture, while maintaining the normals, diffuse etc in sync. I’ve put in a feature request to make this much easier.

import glob
import re
import os
from pathlib import Path
from pxr import Usd, UsdGeom, Gf, Sdf, Vt
import omni.usd
import omni.replicator.core as rep

def make_asset_attr_accessible(prim, attribute_name, attr_value):
    if not prim.GetAttribute(attribute_name).IsValid():
        prim.CreateAttribute(attribute_name, Sdf.ValueTypeNames.Asset, custom=True).Set(attr_value)
        print(f'Set prim for {attribute_name}')

def hide_default_light():
    # Hide the default light
    default_light_prim = stage.GetPrimAtPath("/Environment/defaultLight")
    if default_light_prim.IsValid():
        xform = UsdGeom.Xformable(default_light_prim)
        xform.MakeInvisible()

def get_matching_files(directory, regex_pattern, file_extension):
    matching_files = []
    
    # Compile the regex pattern for efficient repeated use
    pattern = re.compile(regex_pattern)
    
    # List all files in the given directory
    for filename in os.listdir(directory):
        # Check if the file has the required extension
        if filename.endswith(file_extension):
            # Check if the file name matches the regex pattern
            if pattern.search(filename):
                # Add the matching file name to the list
                matching_files.append(os.path.join(directory, filename))
                
    return matching_files

stage = omni.usd.get_context().get_stage()

directory = Path("C:/directory_to_texture_files/").as_posix()
file_extension = ".jpg"  # Example file extension
patterns = ['_Color', '_Displacement', '_NormalGL', '_Roughness']

# Make lists of all texture channels
diffuse_files = get_matching_files(directory, patterns[0], file_extension)
normal_files = get_matching_files(directory, patterns[2], file_extension)
roughness_files = get_matching_files(directory, patterns[3], file_extension)

# Scene settings
rep.settings.set_stage_up_axis("Y")
rep.settings.set_stage_meters_per_unit(0.01)

hide_default_light()

# Creating a new layer in the stage ensures all changes done can be isolated, and we do not accidentally change our original scene.
with rep.new_layer():
    
    # Create a material
    pbr_mat = rep.create.material_omnipbr(count=1)
    pbr_primpath = '/Replicator/Looks/OmniPBR/Shader'
    pbr_shader_prim = stage.GetPrimAtPath(pbr_primpath)
    pbr_shader = rep.get.prims(path_pattern=pbr_primpath)
    
    # Create the primvars on the material - this is to work around an optimization where attributes aren't settable via script until they're viewed in GUI. The workaround is to create them on the prim, then they can be set.
    make_asset_attr_accessible(pbr_shader_prim, "inputs:normalmap_texture", "")
    make_asset_attr_accessible(pbr_shader_prim, "inputs:diffuse_texture", "")
    make_asset_attr_accessible(pbr_shader_prim, "inputs:reflectionroughness_texture", "")
    
    # Create basic replicator objects in the stage - These are created under "Replicator/"
    floor = rep.create.cube(position=(0,0,0), scale=(10,0.1,10))
    cone = rep.create.cone(position=(0,100,0), scale=2)
    camera = rep.create.camera(position=(0,600,1500), rotation=(-22,0,0), focus_distance=200,f_stop=8)
    distance_light = rep.create.light(rotation=(-45,0,0), intensity=1500, temperature=6500, light_type="distant")
    
    # Apply the materials to the floor
    with floor:
        rep.modify.material(pbr_mat)
    
    # This is a randomizer, a special replicator way to organizer the graph logic (not a python function!)
    def randomize_texture(shader):
        with shader:
            rep.modify.attribute("inputs:diffuse_texture", rep.distribution.sequence(diffuse_files))
            rep.modify.attribute("inputs:reflectionroughness_texture", rep.distribution.sequence(roughness_files))
            rep.modify.attribute("inputs:normalmap_texture", rep.distribution.sequence(normal_files))
        return shader.node
    
    rep.randomizer.register(randomize_texture)
    
    # Create the render product and Writer
    render_product  = rep.create.render_product(camera, (1920, 1080))
    writer = rep.WriterRegistry.get("BasicWriter")
    writer.initialize(output_dir="_random_textures", rgb=True)
    writer.attach([render_product])
    
    # In the created graph, this is what executes for N number of frames
    with rep.trigger.on_frame(num_frames=10, rt_subframes=50):
        rep.randomizer.randomize_texture(pbr_shader)
    
    # Preview the scene by executing the graph once, without writing.
    rep.orchestrator.preview()

Here’s another possibility. I was also working on a way to query a local downloaded directory of the vMaterials library. This example will search through that folder, and build a json of all the keywords in all the mdl files it found. Then, you can return mdls that match your query and_search_results = [mdl for mdl in dict_list if "wood" in mdl["keywords"] and "interior" in mdl["keywords"]]

vMaterials | NVIDIA Developer

Again, not a direct answer, but it may help. We’re working on a way to connect this all nicely with replicator in the future, but I don’t have a timeline right now.

import re
import os
import ast
import json
import fnmatch

def find_mdl_files(directory):
    matches = []
    for root, dirnames, filenames in os.walk(directory):
        for filename in fnmatch.filter(filenames, '*.mdl'):
            matches.append(os.path.join(root, filename))
    return matches

# Function to parse the string array from the content
def parse_string_array_correctly(content):
    # Remove the 'string[](' prefix and the closing ')' suffix
    content = re.sub(r'^string\[\]\(', '', content)
    content = re.sub(r'\)$', '', content)
    # Split by '","' to get the individual quoted strings, and strip the quotes
    strings = [s.strip('"') for s in content.split('", "')]
    return strings

def parse_keywords_for_all_mdl_files(file_list):
    # Define the regex pattern to search for the keyword.
    pattern = re.compile(r'::anno::key_words')
    dict_list = []

    for file_path in file_list:
        # Initialize a list to store lines containing the keyword.
        matching_lines = []

        # Load the file and search for the pattern.
        with open(file_path, 'r', errors='ignore') as file:
            for line in file:
                if pattern.search(line):
                    matching_lines.append(line.strip())

        # Initialize a list to store the unique strings
        unique_strings = []

        for line in matching_lines:
            match = re.search(r'\((.*)\)', line)
            if match:
                content = match.group(1).strip()
                unique_strings.extend(parse_string_array_correctly(content))

        # Remove duplicates by converting the list to a set and then back to a list
        unique_strings = list(set(unique_strings))
        unique_strings.sort()  # Sort the list for better readability

        # Output the final list of unique strings
        existing_dict = {}
        existing_dict['keywords'] = unique_strings
        existing_dict['file_path'] = file_path
        dict_list.append(existing_dict)
    return dict_list

def dictlist_to_jsonfile(directory, file_attributes_dict):
    json_file_path = os.path.join(directory, 'mdl_keywords.json')
    with open(json_file_path, 'w') as json_file:
        json.dump(file_attributes_dict, json_file, indent=4)

# Find all the mdl files
mdl_directory = r'C:\vMaterials_2'
mdl_files = find_mdl_files(mdl_directory)
dict_list = parse_keywords_for_all_mdl_files(mdl_files)
dictlist_to_jsonfile(mdl_directory, dict_list)

# Test for finding wood materials with keywords "interior" and "wood"
and_search_results = [mdl for mdl in dict_list if "wood" in mdl["keywords"] and "interior" in mdl["keywords"]]

# Print the results from the vMat lib that meet the criteria
for mdl in and_search_results:
    print(mdl['file_path'])

Hi, first thanks for the replies!
@kit-nvidia-Jen I will try to provide my example with get_files_group by monday as I dont have my Laptop with me right now.
@pcallender thanks very much for the script example. Gives me a good orientation. My main question though is, from where did you download the textures to your local dir. I’d like to use some which come with the nucleus, ideally without downloading them but rather accsessing them directly from the nuclues while running the script.

The vMaterials library can be downloaded from here:
vMaterials | NVIDIA Developer

(I posted it above but the green on green is a bit hard to see)

I’m not familiar enough with nucleus access programmatically to answer this, but I’ll ask around.

Ahh yes it is indeed hard to see :D
Thanks for the link. A local dir will work for now.