Connection Refused

An interesting thing happened on my network over the last few days. It was easily remedied, but if you’ve never seen the issue before, it can be a little confusing. Looking through the backup job report in Solarwinds I noticed a device failure:

That’s odd. I’ve successfully backed this router up for months and now it’s failing? Time to troubleshoot.

Troubleshooting: Isolation

All of our network devices are managed via SSH or SSL if there is a web front-end. Launching SecureCRT and connecting to the offending device was unsuccessful using my normal host entry. Rather than receive the “connection refused” message, the router sent back a little more detail.

The client has disconnected from the server. Reason: The server's host key failed to verify. This could mean that the server you are connected to is impersonating the server it claims to be. A connection could not be established.

Again, an odd issue with the SSH key since I had been in that router as early as the day prior and hadn’t had any issues connecting to the device. I know this ISR4321 is up and running because it responds to pings and is obviously sending back the message to my SSH client. That means there’s a key issue on the router, but how can I get a connection to the terminal to fix the problem? Checking the properties of this host in SecureCRT yielded nothing out of the ordinary. Using SSHv2, password, and all the key exchange settings were correct. I removed the router’s SSH key from my known_hosts directory, tried again, and received the same message.

Hmmm. Let’s dumb down the transport.

Setting the terminal properties down to SSHv1, and try again. This time I’m able to connect to the router and can really start to troubleshoot. First, let’s check out the crypto key used for SSH encryption.

Troubleshooting: Details

There are a few items to notice. First, there is a general purpose key, and second, there is a temporary encryption key. None of these keys may be exported and they really don’t have to be. But there is one item that is curiously missing. The key containing the router hostname; time to create one.

Now that the crypto key is generated I can set the proper version and key for SSH access.

I then modify my SecureCRT host configuration back to SSH2 and run a quick connection test to this router. Viola! The router is now properly responding to the SSH connection! A quick  write memory to save the running-configuration to startup, resubmit the failed Solarwinds backup script, and have a cup of coffee to celebrate!

Actually, the celebration may be premature. Remember, this device was operating normally for months. So what changed? Why did SSH go sideways? To start the investigation I needed to check some basic details. The output of this first command was a head scratcher.

This router has only been up for a day, and looked like it was caused by a reload! A quick check through email reminded me that this router was replaced to address the Cisco clock signal bug in the ISR router family. At least the router replacement activity explains the reload notices in the logs!

Hope this helps in the future.

Leave a Reply