There is a function for that in Isaac Gym (set_rigid_body_color) but to specify the colors your object has to be composed of different rigid objects (at least one for each color).
IsaacGymEnvs uses (by inspecting the files in the AllegroHand and ShadowHand examples) a 3D object (with a material file for the colors) imported as a .urdf file.
So, a good starting point is to modify the cube_multicolor.urdf (located at IsaacGymEnvs/assets/urdf/objects), particularly the mesh file (meshes/cube_multicolor.obj) and the material file (meshes/cube_multicolor.mtl). Both are human-readable files.
Create as many colors as you need in the material file by copying other colors and modifying the diffuse color (Kd) with normalized RGB values…
Then, just replace the current color names in the mesh file.
You can visualize the 3D object (the colored cube) by dragging a drop both files (cube_multicolor.obj and cube_multicolor.mtl) to the following online 3D viewer: https://3dviewer.net/