GT640 (GK208) doesn't work on Kayla Platform

Accelerated Computing CUDA CUDA Setup and Installation
1

So I get two kayla boards and two GT640 cards as suggested by NVIDIA on https://developer.nvidia.com/content/kayla-platform. Sometimes if I was lucky enough, one of the card works perfectly. But in most time it turns out none of the card works on either kayla board. Dmesg gives me something like this:

[ 13.551232] NVRM: failed to copy vbios to system memory.
[ 13.552338] NVRM: RmInitAdapter failed! (0x30:0xffffffff:711)
[ 13.552364] NVRM: rm_init_adapter(0) failed

I am booting from the on board eMMC and didn’t change the factory default rootfs too much. My change includes:

  1. Install PAPI/TAU/PDT under /opt and update ld.so.conf.

Those are tools/libs for performance analysis. See links below:

PAPI: Performance Application Programming Interface
TAU/PDT: TAU - Tuning and Analysis Utilities -

  1. Kayla comes with nVidia driver v313.24, but I have also installed/tried v325.15 and v319.32 (of cause I uninstalled old driver before install the other one).

I tried to mount those two GT640 cards on my PC (x86_64, Debian SID) and it works like a charm with NVIDIA driver, so I guess this should be a software / configuration issue.

What should I do next to debug this issue? Also please kindly let me know if this is not the right forum to post such question.

Thanks!

2

My guess would be a defective Kayla board? Do you have any other GPU besides the 2 GTX 640s to test it with? The fact that you tested the cards on your other PC and they work fine rules out the cards. Other than a defective board, the only other reference I found was to some kernel incompatibility. Maybe if you can back up/restore the eMMC contents easily you could try a newer/older kernel and see if the problem goes away?

The fact that it’s intermittent seems to suggest a hardware issue, at least to me anyway.

3

Try a different power supply.
We had good luck with gold certified power supplies.

4

I have a GT240 which was installed on my PC. It works perfect on Kayla.

I am running the factory default kernel:

uname -a

Linux kayla 3.1.10-carma #1 SMP PREEMPT Wed Jul 10 12:53:50 CEST 2013 armv7l armv7l armv7l GNU/Linux

This kernel is based on Linux for Tegra project distributed by NVIDIA. Maybe I should try mainline kernels on kayla. Definitely I will lose bunch of Tegra features, but let’s see if GT640 works with newer kernel.

5

Are you running GT640 (GK208) on Kayla? Glad to know that you didn’t have any trouble. Are you running the factory default system?

I have two PSUs, one is 650W, the other is 350W. GT640 works with none of them. What a shame…

6

To me it sounds like you’ve eliminated PSU and graphics card as a root cause. Does that board have some newer BIOS/firmware update that you can install? I’m a bit perplexed it would with with an older card but not the one it was supposedly designed to work with. The only other thing I can think of is perhaps subtle differences in GT640 card BIOS’? Do you have the ASUS one they linked to, or did you buy another GT640 GK208 from another NVIDIA vendor?

7

I don’t think so. We don’t have any traditional BIOS since this is an ARM platform. Nevertheless, I downloaded the rootfs from SECO (kayla mITX board vendor) website and tried to boot from that, no luck.

Yes, I bought the ASUS one they linked (newegg). Actually later after I got GT640 I noticed that GK208 is not on the support list on SECO’s manual for the kayla mITX board:

http://shop.seco.com/media/downloadable/Getting_started_with_GPU_DEVKIT.pdf

However, nVidia claims that their official ARM Linux driver supports GT640 and has been tested on kayla platform:

So I just doubt whether GT640 (GK208) was well tested on Kayla. My experience is that it’s very unstable. Some times it works good, but most times it’s broken with various symptoms. I have observed following in those days:

  1. Driver complains: “NVRM: failed to copy vbios to system memory.”. We can see the card in lspci.

Note that I have tried to modify nouveau (the open source nVidia driver, which doesn’t support GK208 officially), and I can bring up 2D function of the GT640 in this condition. I didn’t try nouveau in following conditions yet.

  1. Driver complains: “NVRM: No NVIDIA graphics adapter found!”. lspci won’t show the device.
  2. Driver complains: “NVRM: GPU at 0000:01:00.0 has fallen off the bus”. We can see the card in lspci.

This really looks to me that there are some subtle hardware compatibility issue between the mITX board and GT640. But maybe my experience is just a special case. I wish there are some other guys who are using this configuration could confirm their status here…

8

I was looking into buying a Kayla devkit and I had that same question.

On the SECO website the PDF you cited DOES NOT mention GK208. However their spec webpage DOES (http://shop.seco.com/gpudevkit/gpudevkit-detail.html).

The NVIDIA webpage is also misleading https://developer.nvidia.com/content/kayla-platform
They only mention GeForce GT640/GDDR5 with a first link to only an ASUS card. At the end of the webpage they have two links to ASUS and Gigabyte, but still only GT640.

So maybe someone from NVIDIA or SECO could clarify this ?

Edit: Modified quoted user :-)

9

btw, you misquoted, I didn’t say that blurb in the last post, haha.