Detectnet_v2 inference: numerous incorrect error alerts

pddarrell · February 21, 2023, 3:22pm

• Hardware (RTX 3080)
• Network Type (Detectnet_v2)
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
v3.22.05-tf1.15.5-py3
• Training spec file
detectnet_v2_inference_kitti_tlt.txt (1.6 KB)

I am attempting to run section 9 of the Detectnet_v2 Jupyter Notebook and have had a series of error alerts. I have rebooted and screen grabbed material that relates to the first two.

Despite exporting my key and then using “$KEY”, I get the -k/ --key: expected one argument alert. This happens with “TAO detectnet_v2 train” to start the notebook and then again at section “TAO detectnet_v2 inference”. I can get round this by pasting the actual key over “$KEY”, but clearly this should not be necessary. If you need to see the key in the CLI as a screen grab I will blur the actual key.

OSError: Specfile not found at: /detectnet_v2_inference_kitti_tlt.txt?
As far as I know, detectnet_v2_inference_kitti_tlt.txt is the Specfile and:

It is located in the string specified in the notebook cell.

Normally this kind of stuff goes away on a reboot, but not this time. Do you have any thoughts?

Morganh · February 22, 2023, 2:16pm

You need to run the cell again where the key is set to env.

It is related to the mapping file. ~/.tao_mounts.json.
Please make sure the mapping is valid and correct.

pddarrell · February 22, 2023, 5:46pm

Ha! That’s a neat trick. Thanks for that.

It is related to the mapping file. ~/.tao_mounts.json.
Please make sure the mapping is valid and correct.

Here’s my ~/.tao_mounts.json. It looks to me like lines 8 and 9 are pointing to the right place.

Am I missing something here?

Morganh · February 23, 2023, 1:37am

Can you insert a new cell to check the files under $SPEC/DIR
! tao detectnet_v2 run ls $SPEC/DIR

I am afraid you need to run again where the $SPEC/DIR is set to env.

pddarrell · February 23, 2023, 3:32pm

I hope I understood correctly. Here is the destination you requested, followed by the source:

Is the issue located inside the container?

Morganh · February 24, 2023, 2:33am

From your log, $SPEC is not set.
Please find in the notebook where to set $SPEC.
Then run the corresponding cell.

pddarrell · February 24, 2023, 11:23am

Hi Morganh,

I can find no mention of “$SPEC” in the notebook.

There is:
%env SPECS_DIR=/workspace/tao-experiments/detectnet_v2/specs
and:
$LOCAL_SPECS_DIR

Please advise

Morganh · February 25, 2023, 12:55pm

Sorry, should be SPECS_DIR .

Can you insert a new cell to check the files under $SPECS_DIR
! tao detectnet_v2 run ls $SPECS_DIR

pddarrell · February 25, 2023, 4:52pm

“tao-experiments” is in $SPECS_DIR.

I noticed that in the next cell the command is:
“!tao detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_tlt.tx”

“detectnet_v2_inference_kitti_tlt.tx” is not in $SPECS_DIR

Is “Specfile” detectnet_v2_inference_kitti_tlt.tx?

Is line 9 of my current tao_mounts.json correct?

Morganh · February 25, 2023, 5:02pm

Not correct. It may conflict with line5. Suggest you to set a different path in line9.

pddarrell · February 25, 2023, 5:33pm

I reset it back to this:

and re-ran all relevant cells and got a much better result for your additional cell:

but when I run the next cell the container stops with this:

This is the exact alert I got at the very beginning before I rebooted and started screen grabbing meticulously.
“AssertionError: Pretrained model not found at /home/peter/TAO_toolkit/jp_detectnet/detectnet_v2/experiment_dir_pruned/weights/resnet50_detector_pruned.tlt”
What exactly is it looking for?

Morganh · February 26, 2023, 1:18am

It is still the error due to wrong mapping.
Please check the ~/.tao_mounts.json file.

A simple solution is to set

the same path in line5 and line4.
the same path in line9 and line8.

pddarrell · February 27, 2023, 6:31pm

Hi again

This is what worked.

(I had to use “$KEY” instead of “env” in the “-k” line, because it stopped recognising the"env" key)

I have noticed that it is tending to draw the bboxes for “damage” quite a lot larger than they need to be. Is there any way to influence this?

Thanks for all your help.

system · March 13, 2023, 6:31pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.