Scenario Detection?


So far I am very happy with the object detection using the Jetson TX 2.
Now I want to detect “scenarios”. What do I mean by that?

If the Jetson TX2 sees a landing field getting bigger and the runway is straight, the scenario “landing” should be detected.
If an obstacle is getting bigger, the scenario “crash” is activated.

It is enough if someone would give me a direction (links to tutorials or so)
Oh and btw: Any idea how to detect speed + distance from a detected object? (Is that possible?)

Hi J0eBl4ck, although I’m not sure of a turn-key solution, essentially what you are describing is recognition network + RNN/LSTM for the temporal component. It is similar to behavior recognition from the research. Here are a few potential resources on the topic to look into about it:

Thank you for the links. I will take a look.

Do you think SegNet can help me? My idea:

Train the “segments” (eg.: runway of an airport, sky, tree, etc.)
If the airport segments gets “bigger” -> fine
If the sky segment gets bigger -> fine
If the tree segment gets bigger (within a certain rate) -> crash warning.

How can I check if a segment got bigger in real time? (or video)

What you are asking sounds partly like open research problems to me.

However, if you detect segments or objects, and want to know if they are getting closer, then there are a few options.
You can calculate the bounding box of the segment, and keep the bounding box over time, and calculate a fit of the estimates over time to detect a trend.
You can also use a distance estimation function (there are some good networks for this for driving on roads; haven’t seen any for aerospace.) You can then use the distance out of these models.
You can use a stereo camera, and use the distance that comes out of the disparity algorithm.

Beware that, at some point, your segment/object will hit the edge of the camera, which means it will crop, and won’t be getting “bigger” in that direction.

So, I’m not sure that your focus is at the right level right now. You want to distinguish “crash” from “landing.” It’s not clear to me that “ability to detect things getting bigger” is the best way to go about that.

Doing the task for aerospace was just an idea (My goal is in general to detect scenarios).

Another example:

Lets say I have a single camera on a car and I want to know if in front is a left curve/turn or a right curve/turn -> scenario would be “left curve”, “right curve”. Or is this still “object detection”?

How can I achieve that? (on a road with nice lane markers)
Is SegNet right for that?

I just need a point to start. :)

It feels to me as if you’re trying to solve a research problem, but asking us what the solution is, as if a perfect answer already existed in the back of the book.
Nobody knows for sure. Some approaches are better in certain situations, other approaches are better in other situations.

In general, you want to create a model with some stages. The first stage might be “shape recognition,” the next stage may be “change estimation,” and the last stage would be “scenario inferrence.”
Then you need to build the appropriate parameters of each of these models, and create the appropriate training sets, and then you can train them.
I’d probably start with something like a CNN for stage 1, some classical regression for stage 2, and a RNN for stage 3. But nothing says this is the only, or best, or even a workable solution. Only real trials will tell.

Btw, 90% of the hard work in machine learning is coming up with the appropriate parameterization of your problem, and creating enough training data based on that parameterization that you can actually build and test a high quality model. Despite the model itself getting all the headlines, that’s not the part that needs the most elbow grease.

Thanks for the answer and the direction. Nice to see that there are many ways to find a solution.

I will try some things out and see what works! :)