I was able to overclock NVDEC and NVENC from 716MHz to 844MHz few weeks ago, and seems to be stable so far. I patched Linux_for_Tegra/source/public/kernel/kernel-4.9/drivers/soc/tegra/tegra210-core-dvfs.c like this (basically I just replaced 716800 with 844800 in three lines):
diff --git a/kernel/kernel-4.9/drivers/soc/tegra/tegra210-core-dvfs.c b/kernel/kernel-4.9/drivers/soc/tegra/tegra210-core-dvfs.c
index 8a33da7d2..86369d03d 100644
--- a/kernel/kernel-4.9/drivers/soc/tegra/tegra210-core-dvfs.c
+++ b/kernel/kernel-4.9/drivers/soc/tegra/tegra210-core-dvfs.c
@@ -89,9 +89,9 @@ static struct dvfs core_dvfs_table[] = {
CORE_DVFS("tsecb", 0, 1, 1, KHZ, 192000, 192000, 307200, 345600, 409600, 486400, 524800, 550400, 576000, 588800, 614400, 614400, 627200, 627200, 627200),
CORE_DVFS("c2bus", 0, 1, 1, KHZ, 192000, 192000, 307200, 345600, 409600, 486400, 524800, 550400, 576000, 588800, 614400, 614400, 627200, 627200, 627200),
- CORE_DVFS("nvenc", 0, 1, 1, KHZ, 268800, 268800, 384000, 448000, 486400, 550400, 576000, 614400, 652800, 678400, 691200, 691200, 716800, 716800, 716800),
- CORE_DVFS("nvdec", 0, 1, 1, KHZ, 268800, 268800, 384000, 448000, 486400, 550400, 576000, 614400, 652800, 678400, 691200, 691200, 716800, 716800, 716800),
- CORE_DVFS("c3bus", 0, 1, 1, KHZ, 268800, 268800, 384000, 448000, 486400, 550400, 576000, 614400, 652800, 678400, 691200, 691200, 716800, 716800, 716800),
+ CORE_DVFS("nvenc", 0, 1, 1, KHZ, 268800, 268800, 384000, 448000, 486400, 550400, 576000, 614400, 652800, 678400, 691200, 691200, 844800, 844800, 844800),
+ CORE_DVFS("nvdec", 0, 1, 1, KHZ, 268800, 268800, 384000, 448000, 486400, 550400, 576000, 614400, 652800, 678400, 691200, 691200, 844800, 844800, 844800),
+ CORE_DVFS("c3bus", 0, 1, 1, KHZ, 268800, 268800, 384000, 448000, 486400, 550400, 576000, 614400, 652800, 678400, 691200, 691200, 844800, 844800, 844800),
CORE_DVFS("vi", 0, 1, 1, KHZ, 268800, 268800, 473600, 473600, 576000, 588800, 678400, 691200, 691200, 691200, 793600, 793600, 793600, 793600, 793600),
CORE_DVFS("isp", 0, 1, 1, KHZ, 268800, 268800, 473600, 473600, 576000, 588800, 678400, 691200, 691200, 691200, 793600, 793600, 793600, 793600, 793600),
Changing c3bus clock is necessary otherwise NVDEC and NVENC maximum frequency will remain 716MHz.
It may be possible to achieve greater overclock but I did not try yet higher values than 844MHz.
UPDATE:
I decided to try to overclock NVENC and NVDEC farther:
896000 (896MHz) - seems to work well, but I did not test it for long yet.
934400 (934MHz) - occasional artifacts when decoding video, useless for encoding.
972800 (973MHz) - unstable, it does not crash the system, but there are mostly artifacts when decoding video.