It turns out that there is a mistake in the docker-compose.yml file. The fix is detailed in Change OLLAMA_LLM_LIBRARY from cuda to cuda_v13.
After changing this, the GPU will be utilized.
# FIX: Change the line #61 in docker-compose.yml
environment:
- OLLAMA_LLM_LIBRARY=cuda_v13 # Use CUDA library