Hi Jerry,
Thanks for the suggestion! I skipped that test and just went ahead and moved everything to JetPack4.6 with L4T32.6.1. Everything just worked after that. No need to boost the clock
and argus_camera
worked as expected.
I guess there might be some bugs in vi5
which got fixed in newer version of L4T.
Thank you so much for your help!