When using V4L API, you would have to use flag :
--set-ctrl bypass_mode=0
The bypass mode is used by argus, and you may fool it if not disabling it.
Note that if it gets wrong you can restart its daemon:
sudo systemctl restart nvargus-daemon.service