Originally published at: Even Faster and More Scalable UMAP on the GPU with RAPIDS cuML | NVIDIA Technical Blog
UMAP is a popular dimension reduction algorithm used in fields like bioinformatics, NLP topic modeling, and ML preprocessing. It works by creating a k-nearest neighbors (k-NN) graph, which is known in literature as an all-neighbors graph, to build a fuzzy topological representation of the data, which is used to embed high-dimensional data into lower dimensions. …
All because of my Telescope…
I was able to get your example to work for 500k 1024 dimensional embeddings. However, my full sample is about 50M 1024 embeddings (using stella from huggingface). Do I have to be able to load the full 50M into host ram in order to run umap on this? If not, how do I do this. I have looked through the various cudf documentation, and it is not clear how to do this. Can you share the code (or just a snippet) you used for the Wiki-all subsample (since this is 50M 768 dim)?
1 Like