11. Common issues

11.1. Cannot connect with ssh (REMOTE HOST IDENTIFICATION HAS CHANGED!)

In some situations, particularly after the HPC system has been updated, some computers may fail when trying to use ssh to login to the cluster with an error message that looks as follows:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:JLdDrqe1QVYYPv4eJoFA3JGG4r8aFirVr0nA4AvGw+I.
Please contact your system administrator.
Add correct host key in /home/USER/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/USER/.ssh/known_hosts:69
Host key for mpsd-hpc-login1 has changed and you have requested strict checking.
Host key verification failed.

What this message ultimately means is that the computer you are trying to log into seems to be a different computer than the one you previously accessed.

This is expected after a system update on the HPC, as we redeploy the login nodes. MPSD-provided Linux laptops have their configuration managed remotely and we try and deploy the new “host keys” in advance so that this doesn’t happen. If your machine is not able to receive this update, however, you may run into this situation.

11.1.1. Mismatched host key quick fix

Tip

This is the short version. A more detailed explanation of the problem is available on the next section. You can skip right to it if you prefer.

The message will tell you the fingerprint of the remote machine’s “host key”:

The fingerprint for the ED25519 key sent by the remote host is
SHA256:JLdDrqe1QVYYPv4eJoFA3JGG4r8aFirVr0nA4AvGw+I.

This fingerprint needs to match the ones in Current host key fingerprints for MPSD HPC Login nodes.

Make sure it is correct.

Another line in the error message will point you to a specific line in a file:

Offending ECDSA key in /home/USER/.ssh/known_hosts:69

In our example, this is in line 69. Open the file with your text editor of choice and delete the offending line as indicated by your own error message.

Connect again to the node. You will get a message letting you know about the new node. Double check the fingerprint matches. If so, type yes and then hit Enter.

The authenticity of host 'mpsd-hpc-login1 (131.169.141.83)' can't be established.
ED25519 key fingerprint is SHA256:5XFHNYSygZNgJmba0IQXu9kOcoIj5iu7Y439dGIfGcM.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

You should now be able to log in normally.

11.1.2. Step-by-step explanation and correction of the issue

When you get the error message, it will tell you the “fingerprint” of the “host key”. We can use this to identify whether you are indeed connecting to the right system:

The fingerprint for the ED25519 key sent by the remote host is
SHA256:JLdDrqe1QVYYPv4eJoFA3JGG4r8aFirVr0nA4AvGw+I.

That first line tells you the type of key being used, usually ED25519 will be the default. You can then check this fingerprint against the actual ones we have listed in this page. You can find these in Current host key fingerprints for MPSD HPC Login nodes.

These aren’t functionally different, just use different types of cryptographic algorithms so there is no “right or wrong” key here. Don’t worry if your system is listing a different one, just look at that table instead.

If the fingerprint you get doesn’t match with the one on the table, something has gone wrong. Please reach out to the SSU-CS either through an e-mail support ticket or through Zulip which we constantly monitor.

If they DO match, all this means is that your computer’s information needs to be updated.

11.1.3. Updating the local configuration

The error message will also let you know where ssh has the old information stored:

Add correct host key in /home/USER/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/USER/.ssh/known_hosts:69

This known_hosts file is where ssh stores the fingerprints it knows about. Each line of this file contains the identification of a host and the host key associated with it. The file can be a bit intimidating if you’ve worked with many different computers but it’s nothing to worry about too much:

mpsd-hpc-login2 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKq/5NYgZBdsGPMHG3wTM7m2MvkBFMWF+/58/Bp8zCFb
mpsd-hpc-login2 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDL5mtfe3FpUOQCVbDp2v7sYByKj/yTgMk4iVYwsSJqM2gF8OfS2Ih4la+QYjIy+xG8RUtorgpzUh662NTHJq9Fmy234yx6/7w8dyNBmkFz4+LnXnq61SNDMbK9UIi2JfgDv7kY3fwVTLRmcrXJ/+ODsd+l1KykV7ABpRKZIihmdATYh4dhfPFMNVZ+46iiNGEjTGVREMbm2ruqMvTE69DHHLkuoT1h7BUoaCFb18v4MTcl4WOfiPsMlpFWCfhNCrAFkA8Fj8RoUHwP6pRI70uZjSXv8d4Qr5Xf9OrZNoKft0bJ2Tu6sCjOzYnyX/gww5vdpsAJNEE7IyGdfBcR72x
mpsd-hpc-login2 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEHIH8Ozezx2SKHJ8ymkmCc6tts5gr7qKeXELi3Qw/+dok+ltHVACn79jquhN/jyZkISu98pjajD7hYFJ5j4Ewo=
mpsd-hpc-draco-031 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILXE8KX93DHjW0FhyPM5+yaSTnB3zHiL1tjRCgAgm9ou
mpsd-hpc-draco-031 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp/lBwM+ntu4MsAQKm9eaHWVCNmM/+fY0qhK7NV6cgiipgtghL7w5l8CVmCCFliM6Fx8Eb3thKHLDAKm5hsHGjFMKzD6Rotoce/8nlwBUoWDkLqcH52LG3PLUT4qdIcuCnRWK+pCKcDKZZc/MDReqkamyaQb40k5QxcXCxO5/PPgO7u1DXyG5VAZ76fqyDS2/pQRBFfnwycr4Y2o3sCAI9fZfOAmvh281z+CpmpLjT4FYKC5sir+YIKDu03n41HJZGDh6NWVy4lVsZQq8ldk9hEp2tkrCH5Bm1uHYTUUk7lcgKBrevncBmKRwXkCwgHrg3yIvpwUWMydqqap7uQM7KdxUzoIT4NemTP5qEXO7nthYLcz+588XafSW082NV8zN0liHQf6X2jWkti4CEERvN5d0fVYJFg0AYaImsdVjrfjkLnv3J14sAw7+E3yPh07Eemx3vI0x9yhK780U+rvgUPO7J0qZou6KwgWPQa50w5t4fo1Pa6nBQvr9qn3XuAwE=
mpsd-hpc-draco-031 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHPMC2ZDc2RAHf86l2b+kzUE+FhDikyO8Bnj0WaZ861qmW2N7cESohSgnKgy4c++Tru0KmfnG+ZuoFV4XS4jimM=
mpsd-hpc-draco-002 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEqdwaxFxAt0dI4iNP3p8BuKGOYVSmy1DLL3xTRxn66i
mpsd-hpc-draco-002 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDn1LwPOgVwDz6oc0Ua/E7Gege2/1wP+P84l9yUUZV6LRS/7DhtIqVHaTuXVx1U72Wz4OAF3foIFvc4A9pp0p0Dy5etOSQ3R+kYLulsgVRfWhGV3c6zyH6RRUxInVJHsajdCHXctuazplbdyhr22Hj1zaiUnbCr1EhwYcFjTdTp8UVNtzTjY0Bqrr/ccjSrF/L/e0NuWktuUE1yxzPCNvErd9M6ZMZjrhzGlOp0luan98rNFMvnXsSuELg9JhOL29FTexE7L5jMnJhQnpekIOQSCW9C5Ys2tY4lewLzFirkDbWBvPvexTJRFks1xHiznGK2PTfzy+GEqXPjhjgDuGVHm+Ilh9tu5Ix24CPJqzFjePqekzbHwSngwvEnqUeGi2oVufT3HHjVQ5DA7F3Y9AYOb4IeQ0JgD6mCwsQt0yL49hYcbsg7yHsGvZ8XUgaxCXN5NpUW73Oajqp/QjYzyVdqBYMNBJp8hDPPD1TzkUZS/wrmqWoMDaNmLf6lObI2GAs=
mpsd-hpc-draco-002 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM6azKzEEEctE08wy7baVLt5RRnINhTtAU3t0z5/pz1WLuV38+PawIPXQYXAkFilNWbRyNZH34t3mkWnYKZRqE0=

In our example here, the old key is written on line 69 of this file. The easiest way to go about this is to simply delete the line and then connect again.

Once you do, ssh will then tell you this is a new system. If you knew this system by any other names such as if you accessed the new node during the testing phase, ssh will also tell you about this:

The authenticity of host 'mpsd-hpc-login1 (131.169.141.83)' can't be established.
ED25519 key fingerprint is SHA256:5XFHNYSygZNgJmba0IQXu9kOcoIj5iu7Y439dGIfGcM.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:2313: mpsd-hpc-login1-new
Are you sure you want to continue connecting (yes/no/[fingerprint])?

In this example, all the way on line 2313 of my known_hosts file, I have the same fingerprint for a machine named mpsd-hpc-login1-new. This is a name we gave the login node during the testing phase and it is, indeed, the same machine we want.

After double-checking the fingerprint against the table, typing yes and pressing Enter will register the new information and you can then log in normally.