How to update Pytorch version in an Nvidia docker image (nvcr.io/nvidia/pytorch:20.03-py3)

I need to use a more recent version of Pytorch with CUDA 10.2. Release 20.03 is the most recent one with CUDA 10.02 support, however its version of pytorch (1.5.0a0+8f84ded) does not match the pytorch 1.5.0 release. I would like to update the image with a higher version of pytorch (1.5.0 or 1.5.1). Within the image, the example docker file for a patch is below. However, there are no directions on how to write the my-pytorch-modification.patch file. What should be included here to appropriately change the pytorch version?

When I run the image and attempt to update pytorch with (conda install pytorch=1.5.0 -c pytorch) it fails with a huge number of incompatibilities.

#
# This example Dockerfile illustrates a method to apply
# patches to the source code in NVIDIA's PyTorch
# container image and to rebuild PyTorch.  The RUN command
# included below will rebuild PyTorch in the same way as
# it was built in the original image.
#
# By applying customizations through a Dockerfile and
# `docker build` in this manner rather than modifying the
# container interactively, it will be straightforward to
# apply the same changes to later versions of the PyTorch
# container image.
#
# https://docs.docker.com/engine/reference/builder/
#
FROM nvcr.io/nvidia/pytorch:20.03-py3

# Bring in changes from outside container to /tmp
# (assumes my-pytorch-modifications.patch is in same directory as Dockerfile)
COPY my-pytorch-modifications.patch /tmp

# Change working directory to PyTorch source path
WORKDIR /opt/pytorch

# Apply modifications
RUN patch -p1 < /tmp/my-pytorch-modifications.patch

# Rebuild PyTorch
RUN cd pytorch && \
    TORCH_CUDA_ARCH_LIST="5.2 6.0 6.1 7.0 7.5+PTX" \
    CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
    NCCL_INCLUDE_DIR="/usr/include/" \
    NCCL_LIB_DIR="/usr/lib/" \
    python setup.py install && python setup.py clean

# Reset default working directory
WORKDIR /workspace