UVC gadget not handling control request errors properly

Autonomous Machines Jetson & Embedded Systems Jetson Orin Nano
1

I am trying to have my Jetson Orin Nano send frames as if it were a UVC Camera. It is going to get frames from a proprietary application and send them over the streaming interface. The application exposes settings that can be changed and I want to manipulate them as if they were UVC controls as laid out in the UVC specification. I am using a patched version of Linux’s UVC gadget driver (kernel version 5.10.104) that allows for me to set the camera and processing controls in configfs and set up extension units. The device is able to stream frames and handle successful control requests. However, the device seems to be having trouble with handling failed control requests. When a UVC control request cannot be completed, the specification states:

(UVC 1.5 Specification section 2.4.4 “Control Transfer and Request Processing”)

The device shall use protocol STALL (not function stall) during the Data or Status stages if the device is unable to complete the Control transfer (see section 8.5.3.4 of the USB Specification
Revision 2.0). Reasons for protocol STALL include unsupported operations, invalid target entity, invalid control selector, unexpected Data length, or invalid Data content. The device shall update the value of Request Error Code Control, and the host may use that control to determine the reason for the protocol STALL (see section 4.2.1.2 “Request Error Code Control”). The device must not NAK or STALL the SETUP transaction.

I see two different things occurring depending if I have an error in a UVC_SET_* request or a UVC_GET_* request. If I have an error during a get request, the host is notified that an error occurred but any subsequent control requests will hang. Sending a request to the VC_REQUEST_ERROR_CODE_CONTROL to check which error occurred also hangs. If I have an error during a set request, the host is not notified that the request couldn’t be completed. Sending a subsequent request to the VC_REQUEST_ERROR_CODE_CONTROL will contain the error code for the error that occurred.

I did some tests with UVC controls on a Raspberry PI Zero 2W device with a separate uvc-gadget application (uvc-gadget.git - UVC gadget userspace sample application) modified to return errors on control requests. The host was notified of an error, I was able to check the VC_REQUEST_ERROR_CODE_CONTROL for the error, and I was able to submit subsequent control requests.

Both the Pi and Orin Nano were using the Linux UVC gadget driver. However, the Orin Nano uses the tegra-xudc device controller and I believe the Pi uses the dwc2 device controller. Does the tegra-xudc device controller have any issues with the Linux UVC gadget driver in handling control requests?

The command “cat /etc/nv_tegra_release” prints out “# R35 (release), REVISION: 3.1, GCID: 32827747, BOARD: t186ref, EABI: aarch64, DATE: Sun Mar 19 15:19:21 UTC 2023”.

Attached is the UVC 1.5 specification for reference and the dmesg log file. I made sure to rebuild my kernel with “CONFIG_DYNAMIC_DEBUG=y” and I ran the command “echo ‘file tegra-xudc.c +p’ | sudo tee /sys/kernel/debug/dynamic_debug/control” so kernel messages in the tegra-xudc.c file would be printed in dmesg. I also ran “dmesg -n 8” to enable kernel messages.

I performed the following steps from a test program that uses libuvc to communicate with the gadget:

I start the gadget user space program (dmesg.log:1)

Opened device on host using libuvc library (dmesg.log:821)

I start sending controls (dmesg.log:1055)

Sent SET_CUR request to CT_ZOOM_ABSOLUTE_CONTROL to change wObjectiveFocalLength to 63
-uvc_get_ctrl has no error on host side

Sent GET_CUR request to VC_REQUEST_ERROR_CODE_CONTROL to retrieve bRequestErrorCode
-uvc_get_ctrl has no error on host side
-bRequestErrorCode = 0 (No error) on host side

Sent GET_CUR request to CT_ZOOM_ABSOLUTE_CONTROL to retrieve wObjectiveFocalLength
-uvc_get_ctrl has no error on host side
-wObjectiveFocalLength = 63 on host side

Sent GET_CUR request to VC_REQUEST_ERROR_CODE_CONTROL to retrieve bRequestErrorCode
-uvc_get_ctrl has no error on host side
-bRequestErrorCode = 0 (No error) on host side

Sent SET_CUR request to VC_VIDEO_POWER_MODE_CONTROL to change bDevicePowerMode to 0
-This control doesn’t exist so uvc_get_ctrl should have a pipe error for a protocol STALL occurring
-uvc_get_ctrl actually has no error on host side

Sent GET_CUR request to VC_REQUEST_ERROR_CODE_CONTROL to retrieve bRequestErrorCode
-uvc_get_ctrl has no error on host side
-bRequestErrorCode = 6 (Invalid control) on host side

Sent GET_MIN request to CT_AE_MODE_CONTROL to retrieve bAutoExposureMode
-This is an invalid request as CT_AE_MODE_CONTROL does not have a minimum value
-uvc_get_ctrl has a pipe error for a protocol STALL occurring

Sent GET_CUR request to VC_REQUEST_ERROR_CODE_CONTROL to retrieve bRequestErrorCode
-uvc_get_ctrl never returns

I stop the gadget user space program (dmesg.log:1135)

dmesg.log (112.3 KB)
UVC 1.5 Class specification.pdf (2.1 MB)

3

hello hayes1,

Orin Nano do support USB cameras, please see-also Topic 108585.
hence, please review your driver implementation.

4

I did not look closely at this, but I want to add that in device mode Jetsons do not support isochronous transfer (realtime). Device mode emulated USB cameras will only work with bulk transfer mode due to clock issues (other limitations may still apply). If you are not using isochronous mode, then there may be other limitations or successes, but 100% of isochronous will fail.

6

Thanks for the responses JerryChang, linuxdev,

For further clarification, the Orin Nano is in gadget mode and is using the Linux UVC Gadget Driver (Linux UVC Gadget Driver — The Linux Kernel documentation). The Orin Nano is not using an external USB camera. I do know about the isochronous/bulk endpoint issue and I have already applied a patch so the UVC gadget driver will create bulk endpoints. My gadget has no issue streaming frames. My problem has to do with control requests and when I need to notify the host of a request error.

When I call the Linux UVC Gadget Driver’s UVCIOC_SEND_RESPONSE ioctl at the end of my event loop, I set the ‘length’ parameter of the uvc_request_data struct to ‘-EL2HLT’ if there is a request error. For UVC_GET_* requests, the host is notified of a protocol stall but subsuquent requests from the host will hang forever. For UVC_SET_* requests, the host is not notified of a protocol stall despite their being an error.

I looked at the kernel source code and the driver should be calling the ‘usb_ep_set_halt’ function for ep0 (uvc_v4l2.c - drivers/usb/gadget/function/uvc_v4l2.c - Linux source code v5.10.104 - Bootlin Elixir Cross Referencer). This function should then dispatch to tegra-xudc’s device controller implementation (tegra-xudc.c - drivers/usb/gadget/udc/tegra-xudc.c - Linux source code v5.10.104 - Bootlin Elixir Cross Referencer). I think setting halt on ep0 should trigger a protocol stall, and it should be cleared by the device afterwards. Does tegra-xudc have issues with calling the ‘usb_ep_set_halt’ function for ep0 from the UVCIOC_SEND_RESPONSE ioctl?

My host computer is an x86_64 machine with Ubuntu 22.04 OS. It is using the libuvc (libuvc: libuvc: a cross-platform library for USB video devices) library (v0.0.7) to send control requests to the Orin Nano gadget and I did not start streaming.

7

I’m not able to answer your question, but if you look at the USB from the Jetson with another Linux machine as host to the device mode Jetson, then what shows up from a verbose lsusb?

Note: You will get a manufacturer and product ID which can be used to limit any USB query and avoid query of other devices. For example, the NVIDIA manufacturer is always 0x955, which is represented like this in a verbose query of everything from that manufacturer (note the colon “':'” is required between manufacturer and device IDs):
sudo lsusb -d '0955:' -vvv
(sudo is mandatory for fully verbose)

You can get a log of this on your Linux host PC:
sudo lsusb -d '0955:' -vvv 2>&1 | tee log_lsusb.txt

I’m curious as to how many devices and interfaces the USB side itself sees. More precisely, I’m interested in seeing what is shown about the control versus sensor (I know it isn’t a real sensor, but the host PC does not know anything other than what the Gadget settings report to the host PC). If the control is a separate device (so far as USB knows), then you only need to debug that. Perhaps it is considered an i2c device over USB, in which case one would be debugging what the i2c driver sees.

EDIT: I forgot to mention that depending on how your gadget is set up, then the ID might change, so adjust for that.

8

Hi linuxdev,

Attached is the result of my lsusb command. In addition to the uvc function, my gadget setup also uses the acm and ecm functions.

lsusb_gadget.log (15.7 KB)

9

I see an old style modem listed (which is for talking to the virtual camera). Here is an excerpt:

    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)

The old telephone modems used a common serial UART, but there are basically certain key combinations (“escape sequences”) which have a control context. Imagine an old telephone modem for something like a 56k modem going over a telephone line. These use the “AT” commands on top of otherwise ordinary serial comms. The very popular application minicom was (and perhaps still is) the king of talking manually to such devices. I have other serial programs I like using for serial logins and such, but if you run minicom, then you will see it has an ability to script “AT commands”.

I don’t know much about actual cameras; what I know about mostly is USB. Perhaps someone else will comment on whether or not the camera control itself is “standardized” in such a way that the “AT-commands” device is what most cameras use. If this is not the common interface, then your virtual camera device has its Gadget setup incorrect. I just don’t know.

You previously included a dmesg log, but what I’d like to see is that if your device is not plugged in, and if you monitor “dmesg --follow”, then what is the exact log information specifically and only occurring as a result of plugging in the USB cable? I am referring a log from the host machine which sees the USB cable as a device. Do you have another camera, which works on that same Linux host PC? Do the same with that, and then compare the two logs of the plug-in event.

FYI, there are other virtual interfaces on this as well. For example, I see:

    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         2
      bInterfaceCount         2
      bFunctionClass          2 Communications
      bFunctionSubClass       6 Ethernet Networking

However, since you are connecting the Jetson as a camera device to the host PC over USB, you can ignore this. I mention it because there are all kinds of interfaces possible, and it may not be the camera itself which needs debugging. I think probably it is the USB control end which is involved (but I cannot say just from this). If a control command (from the host PC) to a camera (to the Jetson’s virtual camera) requires a response, and all you have is a “stub” which claims to be that type of communications, it is likely it will never respond and can probably confuse either the Jetson or the host PC. In that case it is a case of you needing to implement something to link the control to the camera and actually respond. I just don’t know; maybe the camera half of it is actually working and not the real issue.

10

Hi linuxdev,

I redid the gadget setup so that my gadget only uses the uvc function. I attached the result of the lsusb command in this case. When only using the uvc function for my gadget, I still having the same issues with my control requests.

log_lsusb.txt (10.5 KB)

11

Hi linuxdev,

Attached is the dmesg log from my host machine when I connect the Jetson gadget to my machine. Also, I attached a dmesg log of when I connected a separate camera to my machine and the verbose lsusb log of that camera’s USB descriptors.

log_host_dmesg.txt (619 Bytes)
log_host_dmesg_microdia.txt (1.7 KB)
log_lsusb_microdia.txt (35.2 KB)

12

Just commenting as I read a few things. This may not be in order.

Your device is found, it has the customized IDs, and other information. I’m not suggesting you make custom udev rules for this, but you could do so since you have a way to uniquely identify the specific device.

This is found to be a UVC device, and so it should automatically load the UVC driver and bind it to this hardware. The USB side is done, and is now only a pipe. What remains is how the UVC driver interacts with the “camera”.

If you interact with this camera, then it is up to your configuration of the gadget on the Jetson as to what will happen. Until you back up that stub interface of gadget with some actual function, then it is just a placeholder.

When using one of the device special files, e.g., the UVC file resulting in “/dev/”, then some parts of it are just normal file read and/or write. Constrol signals are via ioctl() calls, and those functions are custom to every driver. I have not built a gadget camera, and so I don’t know what those calls are. If the gadget has been filled in correctly, then at minimum probably some ability to use this as a camera exists. If any particular command is missing actual function, then it will probably fail in unexpected ways.

Your microdia device log is similar. However, it also has the serial device for control. A camera driver gets commands from the ioctl() calls, but the camera itself would get commands over the serial port. The combination allows manipulating both driver and camera as a whole. The microdia device has a ch341 UART in it which is a very common UART and the driver was in fact found and associated with it. The result was creation of file “/dev/ttyUSB0”. Serial communications to that device will control the camera. The virtual camera (not microdia) does not have this, and so you could only manipulate the driver, not the camera.

If one manipulates a driver and this results in requiring a response from the camera, then lack of a response or invalid response would have unpredictable results. For the gadget camera, perhaps the way this would be set up is by forwarding commands and responses to some real camera, or to a virtual camera which has had the functions created in code. I don’t know.

Can you say more about what this gadget camera is actually using to create the content it is supposed to output? Is it from an actual camera? Is it from some program creating image data?

13

Hi linuxdev,

I use a program to create the image data that is queued with the VIDIOC_QBUF ioctl. I do not have any issues with the streaming interface.

14

What is the control request which is not being handled, and where/how is that request generated? Is it the VIDIOC_QBUF ioctl which is failing, or some other ioctl? Perhaps more important, what is the recipient of the request which is not working correctly?

15

Hi linuxdev,

I am using libuvc on my host machine to send control requests to the gadget. I only having issues with control requests that cannot be fulfilled. If the gadget receives a request it cannot handle, it sets the length parameter of the uvc_request_data struct to -EL2HLT and sends it using the UVCIOC_SEND_RESPONSE ioctl.

My host program uses libuvc’s uvc_get_ctrl function to send a UVC_GET_MAX request to the gadget’s UVC_CT_AE_MODE_CONTROL. It also uses libuvc’s uvc_set_ctrl function to send a UVC_SET_CUR request to the gadget’s UVC_VC_VIDEO_POWER_MODE_CONTROL. Both of these requests are invalid and the gadget responds accordingly with the UVCIOC_SEND_RESPONSE ioctl.

On the host side, the uvc_get_ctrl function returns UVC_ERROR_PIPE, the host program hangs on any subsequent request, and the gadget never receives another v4l2_event in its event loop. The uvc_set_ctrl function returns UVC_SUCCESS and the program continued normally. The UVC_ERROR_PIPE error code indicates a stall so seeing a UVC_SUCCESS return code here is incorrect behavior.

When I sent these requests to my microdia device, in both cases the functions returned UVC_ERROR_PIPE and my host program continued normally.

16

The ioctl call goes to the driver. The driver in turn will have some expectation from the hardware it works with if the ioctl involves talking to the hardware itself. I can’t say for sure, but it is highly likely your failing ioctl wants a hardware response. The gadget sets up a specification for what it is you are emulating through a USB pipe, but gadget itself only sets up basics of the actual device being emulated. In other words, the ioctl failing is unlikely due to the driver or gadget layer, and most likely requires a more detailed editing of the actual device. I don’t have the expertise to tell you what exactly needs editing, but you will need to go into the nitty gritty details of the gadget code to know where that ioctl routes and where it fails.

One good approach is that if you see gadget or driver code which would respond or see the ioctl to simply add printk statements. Hopefully, that code is entirely in kernel modules, and so you’d simply rebuild the module against the existing kernel configuration (including a matched CONFIG_LOCALVERSION), and load your modified version instead of the stock module. This would get you closer to where the ioctl fails, or how far into the driver or gadget the ioctl gets before failing. This is what you could post on for help, but there is a high chance that this is not specific to a Jetson. You might end up asking on the kernel site (kernel.org) how to deal with the ioctl.

It is still possible that what you find will be specific to the Jetson itself, but without finding the failure point of the ioctl it won’t be possible to answer that. Either way it is likely that you will end up implementing some sort of minimal handler for that specific ioctl.

17

Hopefully, that code is entirely in kernel modules

I don’t have a module. The UVC functionality is built into the kernel.

there is a high chance that this is not specific to a Jetson

I also have a Raspberry Pi Zero 2 W that I set up as a UVC Gadget. It doesn’t have issues dealing with invalid control requests.

18

you will need to go into the nitty gritty details of the gadget code to know where that ioctl routes and where it fails

I did some inspection of the kernel source and the UVCIOC_SEND_RESPONSE ioctl will call usb_ep_set_halt for ep0 which dispatches to the device’s callback for usb_ep_ops::set_halt. The Raspberry Pi 2 W’s dwc2 will eventually go to the the dwc2_hsotg_stall_ep0 function which states:

DxEPCTL_Stall will be cleared by EP once it has taken effect, so no need to clear later

I do not see something similar in the __tegra_xudc_ep_set_halt function in Jetson’s tegra-xudc. Unhalting ep0 is done at the start of tegra_xudc_handle_ep0_setup_packet.

However, looking at tegra-xudc.c, I am not sure ep0 is getting unhalted. The tegra_xudc_handle_ep0_setup_packet function is only called from tegra_xudc_handle_ep0_event if the setup_state variable is set to WAIT_FOR_SETUP or from tegra_xudc_handle_transfer_event if there is a sequence number error. I see that tegra_xudc_handle_ep0_setup_packet function can halt ep0 if the setup function fails and change the state back to WAIT_FOR_SETUP. But, uvc_function_setup function doesn’t return a failure. It copies the usb_ctrlrequest to its v4l2_event_queue and returns 0 (success). The control request is handled and the UVCIOC_SEND_RESPONSE ioctl is callled in a separate thread. I don’t see any special code that would change the setup_state variable to WAIT_FOR_SETUP after ep0 is halted during a call to usb_ep_set_halt. I also never saw a sequence number error when I had kernel messages enabled for the dmesg.log in my first post. I will rebuild the kernel with more printk’s in these functions to get more information on what is happening here.

EDIT: The setup_state variable can also be changed to WAIT_FOR_SETUP in the tegra_xudc_ep0_req_done function called by tegra_xudc_handle_transfer_completion. I don’t see the transfer function being called in the dmesg.log either.

19

I don’t know if the response is intended to occur from within the gadget code, or perhaps it is passed on to something else and needs an adapter for the case of a gadget (perhaps it requires a user space to monitor and reply to an event intended to halt ep0. Gadget itself is just a function interface, and I’ve never examined the video part of gadget (I tend to deal with HID). However, this is enough of a “call stack” (not really a call stack, but it is a good description of function calls) which seems to end at the tegra_ equivalents of the stock gadget code that @JerryChang might be able to ask the driver people to answer that question in your reply #15. You might need to create code to respond to that, or it might be that the gadget code is incomplete. It is something I can’t answer.

20

hello hayes1,

please refer to below for some basic v4l2_ioctl_ops has implemented in the VI driver.
you may add/extend your control operations in the drivers for your own use-case.
i.e. $public_sources/kernel_src/kernel/nvidia-oot/drivers/media/platform/tegra/camera/vi/channel.c

static const struct v4l2_ioctl_ops tegra_channel_ioctl_ops = {
        .vidioc_querycap                = tegra_channel_querycap,
        .vidioc_enum_framesizes         = tegra_channel_enum_framesizes,
        .vidioc_enum_frameintervals     = tegra_channel_enum_frameintervals,
        .vidioc_enum_fmt_vid_cap        = tegra_channel_enum_format,
        .vidioc_g_fmt_vid_cap           = tegra_channel_get_format,
        .vidioc_s_fmt_vid_cap           = tegra_channel_set_format,
        .vidioc_try_fmt_vid_cap         = tegra_channel_try_format,
        .vidioc_reqbufs                 = vb2_ioctl_reqbufs,
        .vidioc_prepare_buf             = vb2_ioctl_prepare_buf,
        .vidioc_querybuf                = vb2_ioctl_querybuf,
        .vidioc_qbuf                    = vb2_ioctl_qbuf,
        .vidioc_dqbuf                   = vb2_ioctl_dqbuf,
        .vidioc_create_bufs             = vb2_ioctl_create_bufs,
        .vidioc_expbuf                  = vb2_ioctl_expbuf,
...