Create a PC assistant.

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

Excuse me for the machine translation.

The idea came because the time to create a mobile robot is not small, and the creation of an AI module is possible from the beginning of the acquisition and preparation of the Jetson Nano. Before making a full mobile machine, I would like to try to make an assistant to work with PC. Of course it will be more of a student than an assistant. What I would like from him would be:

  1. so that it connects to the PC via USB,
  2. that it has its virtual mouse and virtual keyboard on the PC,
  3. so that he can see the computer screen, let via HDMI;
  4. could receive voice commands from the microphone of the camera connected to the Jetson Nano.

Example of operation:
Create work - “cleaning documents,” create procedure - “opening folder”
beginning
Read and announce the folder name, date of creation, size;
Receive confirmation that the file or directory has been opened or continued (delete or go to the next or inward).
And so on.

The point is not to create some garbage cleaner on the PC, but to learn to create some network that can do different specific works. Of course, I would like Jetson Nano to have its web browser through which it can surf and read content, process it, and show the formed knowledge base through an interface similar to that designed to work with nods.

Everything seems like an incredibly huge volume job, but just one question - how difficult and real is it?

2

hello hretgear,

I think there’re two approaches for your first step, to communication with your host-PC.

  1. the quickest way would be using Nano platform to remote login your PC environment, then you might also grant the access permission of the file system.
  2. another way would be more complex, you’ll need a HDMI2CSI bridge for connection, let the host-PC as Nano’s camera sources frames. you should also handle the processing to emulate the virtual mouse sending controls.
3

Thank you very much for the answer. Tell me, don 't you plan to create a robot class like Java? Only in Java the robot works on the computer on which it is running - moves the cursor and so on, and here the task is more difficult of course. Of course it is not an independent project, but very interesting as an experiment for training the network.

Although… what difference does it make, even though Jetson Nano will be the computer on which the robot class is launched… there is no difference… Of course, open applications will cost RAM, but nevertheless, let the titmouse in hand than the crane in the sky.