That keyserver is for the JetPack apt repo, which does contain packages for both Jetson/ARM and x86_64. You don’t actually need the host PC packages if you don’t want them - they are for cross-compilation for ARM (from PC) and the like. You can’t compile them from source because they are distributed as binaries.
@Andrey1984 has also had an interesting idea on upgrading to 20.04 inside the l4t-base container, and keeping it all inside there:
https://forums.developer.nvidia.com/t/jetpack-4-5-production-release-with-l4t-32-5/166473/9
So then you could install 20.04 packages inside the upgraded container, and it would help with keeping a clean environment on your Jetson.