Weird error when attempting to ssh into nano

hey everyone,

I plugged in my nano this morning and opened a terminal on my host PC in the hopes that I would finally be able to play with the AI fundamentals course materials but instead, right after I typed in “ssh username@[IP ADDRESS]” i got this very dire sounding error:
“@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:sv8NUL2AHjz59P8/G6Z+K+5uIT6WPghkYx37jGVHE4M.
Please contact your system administrator.
Add correct host key in /home/blackout/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/blackout/.ssh/known_hosts:1
remove with:
ssh-keygen -f “/home/blackout/.ssh/known_hosts” -R
ECDSA host key for IP ADDRESS has changed and you have requested strict checking.
Host key verification failed.”

I copied and pasted "ssh-keygen -f “/home/blackout/.ssh/known_hosts” -R “IP ADDRESS” into the terminal and hit enter. Things began looking more normal after it asked if I really wanted to connect though it couldn’t verify it’s authenticity. I said ‘yes’ and hit enter, whereupon I was asked for my password for my nano. Once I input that password and hit enter I got the error message:

“permission denied. please try again”

Not sure how or what happened but if anyone has some advice it’d be greatly appreciated! Thanks!

Some information on ssh may help.

When you contact a remote host it has keys for the host itself at “/etc/ssh/”. The IP address or alias for an address used for the first session is memorized, and changes are reported as possible “man in the middle” problems.

Let’s say you flashed the unit again, and the keys changed. This would trigger man in the middle warnings.

Let’s say you have several embedded systems on the network, and that you’ve used ssh login on all of them. Now pretend that the units are booted in some other order than normal and for whatever reason Jetson “A” now has the address of Jetson “B”, and Jetson “B” now has the address of the old Jetson “A” (relative to when the first ssh session memorized those details). In that case you’d get a man in the middle warning.

How do you name the address of the Jetson? Do you use a dotted-decimal address, e.g., “192.168.55.1”? Did this change? Have you flashed something and possibly ended up with a different host fingerprint?

1 Like

Hey,

What makes this feel strange to me is that I hadn’t really DONE anything-at least that I know of…

I opened a terminal like always and tried ssh’ing into my nano in exactly the same one as I’ve done all week. To your question, I had been using a dotted decimal address. My command in terminal would always be:

“ssh username@[IP ADDRESS]”

This method had been working forever until this morning. To your second question-I haven’t re-flashed the card in my nano for over a week, maybe two? I only re-flashed it those couple of weeks ago when I hooked up a monitor to it and it never booted up. I re-flashed it and everything was fine again…

Does this provide any more clues?

Are you certain the dotted-decimal address has not changed? Any router which issues DHCP addresses can and will change the address going to a specific device without notice. On my local development environment I always go into the admin page of my router and bind an exact address to every MAC address expected (and deny access to anything without a listed MAC). It is quite possible that if the address changed you really are reaching a different device.

When it says the host key changed, then this tends to say actual content of keys in “/etc/ssh/” has changed. This would imply either an actual compromised system, or else an entirely different computer. Are your systems touching the internet or an uncontrolled network not protected from outside logins via router or firewall?

It is nice to know that if you do copy your key to a compromised system that it is only a public key and might be used to fool you, but that key would be useless to get back into your original system. If you have access to the end point system, then what do you see for timestamps in:
ls -ltr /etc/ssh/*
…if those files are ancient (well, dating back to about when you installed), then perhaps nothing is wrong. If the keys are altered such that it was recent, then perhaps something really is wrong (a good attacker would change timestamps anyway, but it is worth looking).

You might also want to describe your network in more detail, e.g., all using a router and nothing automatically forwarded from the internet (safer), versus touching the internet (perhaps your Nano is somewhere else in the world?).

1 Like

I suppose I’m ultimately not certain that the dotted decimal address hasn’t changed. I guess I’m also struggling to understand how I might have affected that change myself.

I’m not sure that my router has a firewall-likely not? If it does then I didn’t set it up myself. What do you mean by end-point system? My host PC? Or the nano?

I opened a terminal and input ls -ltr /etc/ssh/*. There were randomly 3 entries from march which is bizarre considering I only purchased my nano this fall and then every other entry was from december 26th. See below:
“-rw-r–r-- 1 root root 3264 Mar 4 2019 /etc/ssh/sshd_config
-rw-r–r-- 1 root root 1580 Mar 4 2019 /etc/ssh/ssh_config
-rw-r–r-- 1 root root 553122 Mar 4 2019 /etc/ssh/moduli
-rw-r–r-- 1 root root 338 Dec 26 10:42 /etc/ssh/ssh_import_id
-rw-r–r-- 1 root root 424 Dec 26 10:42 /etc/ssh/ssh_host_rsa_key.pub
-rw------- 1 root root 1675 Dec 26 10:42 /etc/ssh/ssh_host_rsa_key
-rw-r–r-- 1 root root 204 Dec 26 10:42 /etc/ssh/ssh_host_ecdsa_key.pub
-rw------- 1 root root 227 Dec 26 10:42 /etc/ssh/ssh_host_ecdsa_key
-rw-r–r-- 1 root root 124 Dec 26 10:42 /etc/ssh/ssh_host_ed25519_key.pub
-rw------- 1 root root 444 Dec 26 10:42 /etc/ssh/ssh_host_ed25519_key”

I’m very much a newb and trying to make sense of all of your questions and of all of these outputs but it doesn’t strike me as abnormal.

And yes, my nano is physically in my room on my desk and I’m using the wifi from my own router within my apartment.

quick update,

I plugged my nano back into the monitor and brought up a terminal window. I then input ‘ifconfig’ and saw that the IP was now suddenly different than what what it had been previously! I went back to my host pc and attempted to ssh into my nano with the new IP address and it worked good as new! Thanks again for all your help.

hey,

As a weird postscript to last night’s findings, I tried to ssh again into my nano this morning but was thwarted again. I hooked up my nano up once more to the monitor and keyboard and saw that the IP had changed yet again! Is this going to be my fate? That everytime I want to ssh into my nano I’ll need to first look at the ip locally and find that it’s changed?

Is there a way to lock it in on the nano so that I don’t have to do this extra step each time? Thanks!.

Rebooting can change the IP address. Nothing the end user can do if the router itself wants to change an address. This is why I use my own router and bind IP address to MAC address for the development network. This is how you lock it in.

Routers indirectly remove incoming traffic which is not a reply to something the inside device did not initiate. This is just the nature of a system where several inside devices share the same outside IP address…the router knows a reply to something a specific inside device initiated, but if the outside world initiated an attempt to go to the inside world, then it won’t know which device to route to unless it was manually set up. Some routers (such as Comcast) may automatically pass through broadcast traffic since broadcast does not need to be a response to a specific device.

Is the router yours? Can you enter an admin page and bind a MAC address to a specific IP address? If not, then you will probably need to use either a local terminal or serial console to find the output of “ifconfig” to know for certain if your address is correct.

If you happen to have assigned the IP address to your host PC, then it won’t matter what your Jetson’s IP is, it will always log in from the PC to the PC. You absolutely must have an IP address on your Jetson which is unique and not assigned to the PC itself (the address would be in one of the PC’s subnets, but the address would not be one the PC itself actively considers “its own”).

The file permissions show these files were modified on Dec. 26:

Dec 26 10:42 /etc/ssh/ssh_import_id
Dec 26 10:42 /etc/ssh/ssh_host_rsa_key.pub
Dec 26 10:42 /etc/ssh/ssh_host_rsa_key
Dec 26 10:42 /etc/ssh/ssh_host_ecdsa_key.pub
Dec 26 10:42 /etc/ssh/ssh_host_ecdsa_key
Dec 26 10:42 /etc/ssh/ssh_host_ed25519_key.pub
Dec 26 10:42 /etc/ssh/ssh_host_ed25519_key

Well, it isn’t actually a guarantee of change, but it does say something “touched” those files on that date and time. If there was merely a timestamp change, then your ssh connect would not have detected any change (ssh depends on content, not timestamp). The evidence of timestamp in combination with ssh client being upset says the files creating the host “fingerprint” have changed.

There are reasons why those files might have changed and not be due to malicious reasons, but it would be hard to actually say what the cause is. If your system is behind a router which does not allow and does not forward incoming requests (aside from a response to an outgoing request), then odds are it was something non-malicious (but I’ll say more about that below). If your Jetson was visible to the entire outside world, then it might have indeed been malicious, but most malicious hacks would want to preserve the host fingerprint files, and would not want to change them. Two Jetsons swapping IP address might give the illusion of change.

In terms of isolation from the outside world via a router, consider too that any other computer on the inside which is compromised could be used to defeat any kind of security a router provides. Even so it would be an idiot malicious attacker that changes the host fingerprint keys…purposely making it look like the host was compromised doesn’t have much of a purpose.

There are commands to generate keys. I have never seen normal updates change keys, but if you’ve manually run a command for key generation, then I suppose it is possible to have manually changed them on Dec. 26 at 10:42. Highly unlikely though, it is a bit like tripping on the ice of a sidewalk, sliding over to the nearby convenience store, and then accidentally stumbling and buying a winning lottery ticket. :P

I don’t know if you have your timezone set to local or not, but was there anything you were doing with the Jetson at 10:42 AM on Dec. 26?