TX2i, headless on custom carrier card, vulkan not working

jan.kristian · December 10, 2020, 4:00am

Hi,

We are running a TX2i on our own custom carrier card (and sw tuned to match) (width out any output hdmi/dp) and are having problems getting vulkan based applications, such as ‘vulkan-smoketest’ running.
We are able to get X11, and OpenGL based applications (such as glxGears), to run. (Verified by capturing display :0, using gstreamer, and streaming it to another computer.)
But when running ‘vulkan-smoketest’ it core-dumps.

When running ‘vulkaninfo’ it seems to have no presentable surfaces:
Presentable Surfaces:
=====================
GPU id : 0 (NVIDIA Tegra X2 (nvgpu))
Surface type : VK_KHR_xcb_surface
Formats: count = 0
Present Modes: count = 0

(These counts are 2 and 3 when running on a SDK from Nvidia, where vulkan works fine).

When doing a ‘strace vulkaninfo’ on the SDK, I can see that vulkaninfo opens ‘/dev/tegra_dc_0’ and ‘/dev/fb0’.
These devices are not present on the TX2i running on out carrier card.

Does vulkan depends on these two devices, or is there something else that is preventing vulkan from working?

DaneLLL · November 2, 2020, 11:36pm

Hi,
Since we don’t have thecustom board, is it possilbe to reproduce the failure on default board so that we can check and investigate?

Honey_Patouceul · November 30, 2020, 9:55am

Sorry, I’ve poor knowledge about Vulkan on Jetson, but my understanding so far is that it would expect wayland.
Someone more skilled with this may correct me (I’d be happy to learn more).

jan.kristian · November 30, 2020, 10:15am

Hi,

I have been able to reproduce this using a TX2i on a Nvidia developmentboard (P2597)

I first flashed the TX2i using the jetson-tx2i config.
Then I modified the dtb from that config (tegra186-quill-p3489-1000-a00-00-ucm1.dtb) and disabled hdmi.

Setting:
nvdisplay@15210000/status = “disabled” sor1/status = “disabled” sor1/hdmi-display/status = “disabled” dpaux@15040000/status = “disabled”

Saved this new dts into tegra186-no-hdmi-p3489-1000-a00-00-ucm1.dts, and compiled it to dtb.

Created my own config by copying the jetson-tx2i.conf into no-hdmi-tx2i.conf.
Edited this new conf-file so that DTB_FILE and TBCDTB_FILE now points to tegra186-no-hdmi-p3489-1000-a00-00-ucm1.dts.

Then I flashed only the kernel device tree on the TX2i using
sudo ./flash.sh -r -k kernel-dtb no-hdmi-tx2i mmcblk0p1

The TX2i on the developmentboard now behaves as I described in my first post.

Any thoughts or insight into this would be greatly appreciated.

WayneWWW · December 4, 2020, 3:06am

Hi,

Looks like vulkan-smoketest is even not able to run with headless case on devkit. Issue is not related to your dtb.

jan.kristian · December 4, 2020, 11:21am

Hi,

Good to hear that this is not related to my dtb. As I understand it I can keep it as-is?

But the problem still remains, how to get vulkan based sw to run in a headless configuration.
Any thoughts on how I can fix that?

WayneWWW · December 4, 2020, 5:49pm

Hi,

I am not quite sure about the request. Vulkan smoketest is a application that would render something on display, right? And now you want it to run with headless case?

jan.kristian · December 4, 2020, 8:08pm

Hi,

The confusion is understandable.
Yes, I want to run vulkan based application on a headless system. I will then use gstreamer to capture the output and stream it to another system to be displayed.
This works fine for X11 apps (such as gdm3, xeyes etc) and for OpenGL based application (sunch as glxGears), but not for Vulkan based applications.

WayneWWW · December 7, 2020, 2:39am

Hi,

What would happen if you enable the nvdisplay again in dtb and run this application with export DISPLAY=:0?

WayneWWW · December 7, 2020, 2:58am

I just tried this app on my nvidia devkit with no monitor connected.

Under such case, xrandr still give us a 640x480 mode after we export DISPLAY variable and app can still run. fb0 node exists on our board too. Thus, I would like to know if you can see fb0 if you add nvidsplay node back to dtb again.

However, if fb0 cannot be generated on your device due to some hardware limitation, please try to add a virtual screen to your xorg.conf and see if you can see gdm3 enabled with this virtual screen.

Example:

Section “Screen”
Identifier “Default Screen”
Monitor “Configured Monitor”
Device “Default Device”
SubSection “Display”
Depth 24
Virtual 1280 800
EndSubSection
EndSection

jan.kristian · December 7, 2020, 12:31pm

Hi,

Yes, this is also what I have found, running the devkit with no monitor leaves the /dev/fb0 (and /dev/tegra_dc_0) in place.
And by enabling nvdisplay on the devkit (i.e. flashing with the jetson-tx2i config) the /dev/fb0 appears.

When trying to enable the nvdisplay on our carrier card the kernel will not complete its start-up sequence. Probably due to missing hdmi hw support on our custom carriercard.

By following the description in my post from Nov 30 I am able to recreate the exact same situation we have on our custom carrier card.

I tried your xorg.conf on our carrier card, but it gave the same result, smoketest still crashes and vulkaninfo shows no surfaces. (no /dev/fb0 either)

I have been using a xorg.conf (similar to your example).
The modifications I have used to get X11 to run is :

/etc/X11/xorg.conf:
Section “Screen” Identifier “MY-SCREEN” DefaultDepth 24 SubSection “Display” Virtual 1280 1024 Depth 24 EndSubSection EndSection

/etc/gdm3/custom.conf:
[daemon] AutomaticLoginEnabled=true AutomaticLogin=nvidia

To find the Xauthority file i run:
$ps -aux | grep Xauth
and look for the path after -auth

Then set the env variables DISPLAY and XAUTHORITY:
export DISPLAY=:0 export XAUTHORITY=/run/user/1000/gdm/Xauthority

Now I can run applications such as xeyes and glxgears, and have them displayed using gstreamer.
But not vulkan based apps.

It seems to me that vulkan need some “surface” to draw on, and it is not able to find, or create this.

WayneWWW · December 8, 2020, 3:31am

Hi,

Could you share me the dmesg to check what gets stuck in your kernel?

I think bypassing and give you a fake fb0 node would be possible and faster as workaround.

jan.kristian · December 8, 2020, 7:44am

Hi,

I was not able to do a “dmesg” since the kernel crashes/does not completely boots.
But I have attached the bootlogbootlog-custom-cc-with-hdmi.txt (68.3 KB).

I modified the dtb for the custom carrier card by enabling the following:
nvdisplay@15210000/status = “okay” sor1/status = “okay” sor1/hdmi-display/status = “okay” dpaux@15040000/status = “okay”

And flashed the kernel device tree.

Hopefully you will find something in the log to help me along.

WayneWWW · December 8, 2020, 8:00am

Actually, I am not sure why your board seems able to get some fake edid while you said there is no HDMI port on your board…

[ 11.104951] edid invalid
[ 11.108098] tegradc 15210000.nvdisplay: hdmi: tmds rate:25200K prod-setting:prod_c_hdmi_0m_54m
[ 11.118070] tegradc 15210000.nvdisplay: hdmi: get RGB quant from EDID.
[ 11.124622] Unable to handle kernel read from unreadable memory at virtual address 00000063
[ 11.132981] Mem abort info:

[ 11.312293] [] tegra_edid_is_rgb_quantization_selectable+0x4/0x10
[ 11.319954] [] tegra_hdmi_controller_enable+0x218/0xac8
[ 11.326735] [] tegra_dc_hdmi_enable+0x3c/0xb0
[ 11.332648] [] tegra_nvdisp_head_enable+0x4a4/0xe18
[ 11.339083] [] _tegra_dc_enable+0xf8/0x118
[ 11.344737] [] tegra_dc_probe+0x101c/0x14e8

WayneWWW · December 9, 2020, 10:52am

Another method is enabled the fake EDID in edid.c and see if it can bypass that error.

set use_fallback = true; # in tegra_edid_get_monspecs() under edid.c.

#Edit: please ignore my previous comment about changing the sor in dts. Only try this one should be sufficient.

jan.kristian · December 9, 2020, 10:52am

Hi,
It works.
Thank you for your assistance in solving this.