Fix rotation bug in nvmap dma alloc

Discovered a bug while using 16GB as nvidia,generic_carveout in Nvidia Jetson AGX Orin. The function __nvmap_dma_alloc_from_coherent (found in Linux_for_Tegra/sources/kernel/nvidia/drivers/video/tegra/nvmap/nvmap_init.c tag jetson_35.3.1) makes rotation operations of signed integers, generating a “integer overflow” bug when using big reserved memory size such as 16GB in our case. This is fixed by adding the necessary casts before the rotation.

diff --git a/drivers/video/tegra/nvmap/nvmap_init.c b/drivers/video/tegra/nvmap/nvmap_init.c
index 175ef1100..037164ebc 100644
--- a/drivers/video/tegra/nvmap/nvmap_init.c
+++ b/drivers/video/tegra/nvmap/nvmap_init.c
@@ -387,7 +387,7 @@ static void *__nvmap_dma_alloc_from_coherent(struct device *dev,
 
 	spin_lock_irqsave(&mem->spinlock, flags);
 
-	if (unlikely(size > (mem->size << PAGE_SHIFT)))
+	if (unlikely(size > ((ssize_t)mem->size << PAGE_SHIFT)))
 		goto err;
 
 	if ((mem->flags & DMA_MEMORY_NOMAP) &&
@@ -417,7 +417,7 @@ static void *__nvmap_dma_alloc_from_coherent(struct device *dev,
 	/*
 	 * Memory was found in the coherent area.
 	 */
-	*dma_handle = mem->device_base + (pageno << PAGE_SHIFT);
+	*dma_handle = mem->device_base + ((dma_addr_t)pageno << PAGE_SHIFT);
 	if (!(mem->flags & DMA_MEMORY_NOMAP)) {
 		addr = mem->virt_base + (pageno << PAGE_SHIFT);
 		do_memset = 1;
-- 

Hi,

Thanks for the sharing!
The patch looks reasonable, and we’ll look into it.