I do not know of a suitable minimal rootfs. However, the initial rootfs is purely Ubuntu 14.04LTS with no nVidia modifications. The apply_binaries.sh step (automatically done if using JetPack for flash) and flash boot loader config steps add the customizations. If you take any reduced replacement rootfs…and if files to overlay via apply_binaries.sh are compatible with that replacement…then you can use any rootfs. One possible variant is the supplied rootfs with some packages removed.
In terms of compatibility the biggest issue for other rootfs substitutes is that the Xorg server is the interface to accessing GPU functions. This means the ABI of the Xorg server must match the ABI from the nVidia GPU drivers…change the ABI and you lose all direct access. So either a substitute rootfs must use the same Xorg version, or you have to lose CUDA and hardware-accelerated OpenGL. This is a strong reason to cut down the existing sample rootfs instead of starting with a completely new rootfs unless you can use the same Xorg server version.
Should you choose to cut down the existing rootfs, what features does it currently have which you do not need? For example, do you need a graphical desktop login? If not, then the display manager and desktop window manager could be removed to save space.
Alternatively, Linux allows you to mount any partition you want to any directory as a mount point. Depending on your situation you could for example make a partition in an SD card or SATA drive to simply have more space, but I don’t know what carrier board you are using, nor do I know what your separate partitions are being motivated by. If you have a use-case description it might help to offer better advice.