Upgrading from 346.47 to 352.41 on AWS causes kworkers to lockup on boot/module load, and prevents u

Upgrading to 352.41 causes the nvidia driver to generate a massive amount of processor interrupts (from checking /proc/interrupts), and causes any process interacting with the nvidia driver to completely lock up. This issue only appears to occur intermittently. This is on AWS as well, using AWS Linux.

Bug report log application freezes when fetching information from procfs, but partial log will be attached.

Log entries from dmesg

[ 362.512087] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
[ 362.515687] Tainted: P O 3.14.48-33.39.amzn1.x86_64 #1
[ 362.519197] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[ 362.523752] kworker/0:0 D ffff8803ca98d0e0 0 4 2 0x00000000
[ 362.528080] Workqueue: events os_execute_work_item [nvidia]
[ 362.531541] ffff8803ca9b3b90 0000000000000046 ffff8803ca98cbf0 ffff8803ca9b3fd8
[ 362.536313] 00000000000127c0 00000000000127c0 ffff8803ca98cbf0 ffff8803ca4a7ae8
[ 362.541074] 7fffffffffffffff ffff8803ca98cbf0 0000000000000002 0000000000000000
[ 362.545560] Call Trace:
[ 362.546831] [] schedule+0x29/0x70
[ 362.549388] [] schedule_timeout+0x219/0x2b0
[ 362.552417] [] __down_common+0xed/0x13c
[ 362.555693] [] ? __getnstimeofday+0x20/0xc0
[ 362.559404] [] __down+0x1d/0x1f
[ 362.562326] [] down+0x41/0x50
[ 362.565108] [] os_acquire_mutex+0x3f/0x50 [nvidia]
[ 362.568883] [] _nv012533rm+0x18/0x30 [nvidia]
[ 362.572414] [] ? _nv014110rm+0x3d/0x120 [nvidia]
[ 362.576283] [] ? _nv010585rm+0x57/0xe0 [nvidia]
[ 362.580156] [] ? _nv000671rm+0x229/0xb10 [nvidia]
[ 362.583889] [] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[ 362.587736] [] ? os_execute_work_item+0x46/0x70 [nvidia]
[ 362.591499] [] ? process_one_work+0x181/0x450
[ 362.594729] [] ? worker_thread+0x121/0x400
[ 362.597813] [] ? rescuer_thread+0x400/0x400
[ 362.600932] [] ? kthread+0xd2/0xf0
[ 362.603585] [] ? kthread_create_on_node+0x180/0x180
[ 362.607046] [] ? ret_from_fork+0x58/0x90
[ 362.609938] [] ? kthread_create_on_node+0x180/0x180
nvidia-bug-report.log.gz (36.2 KB)