Connection Refused

Nov 1, 2018
Brian Gleason

An interesting thing happened on my network over the last few days. It was easily remedied, but if you’ve never seen the issue before, it can be a little confusing. Looking through the backup job report in Solarwinds I noticed a device failure:

ERROR: Cannot download Running config : Connection Refused by 192.168.150.5

That’s odd. I’ve successfully backed this router up for months and now it’s failing? Time to troubleshoot.

Troubleshooting: Isolation

All of our network devices are managed via SSH or SSL if there is a web front-end. Launching SecureCRT and connecting to the offending device was unsuccessful using my normal host entry. Rather than receive the “connection refused” message, the router sent back a little more detail.

The client has disconnected from the server. Reason: The server’s host key failed to verify. This could mean that the server you are connected to is impersonating the server it claims to be. A connection could not be established.

Again, an odd issue with the SSH key since I had been in that router as early as the day prior and hadn’t had any issues connecting to the device. I know this ISR4321 is up and running because it responds to pings and is obviously sending back the message to my SSH client. That means there’s a key issue on the router, but how can I get a connection to the terminal to fix the problem? Checking the properties of this host in SecureCRT yielded nothing out of the ordinary. Using SSHv2, password, and all the key exchange settings were correct. I removed the router’s SSH key from my known_hosts directory, tried again, and received the same message.

Hmmm. Let’s dumb down the transport.

Setting the terminal properties down to SSHv1, and try again. This time I’m able to connect to the router and can really start to troubleshoot. First, let’s check out the crypto key used for SSH encryption.

ROUTER1#sho crypto key mypubkey rsa 
% Key pair was generated at: 10:39:28 UTC Oct 30 2018
Key name: CISCO_IDEVID_SUDI
Key type: RSA KEYS
 On Cryptographic Device: act2 (label=act2, key index=24)
 Usage: General Purpose Key
 Key is not exportable.
 Key Data:
  < truncated >
% Key pair was generated at: 10:39:31 UTC Oct 31 2018
Key name: CISCO_IDEVID_SUDI.server
Key type: RSA KEYS
Temporary key
 Usage: Encryption Key
 Key is not exportable.
 Key Data:
  < truncated >
ROUTER1#

Troubleshooting: Details

There are a few items to notice. First, there is a general purpose key, and second, there is a temporary encryption key. None of these keys may be exported and they really don’t have to be. But there is one item that is curiously missing. The key containing the router hostname; time to create one.

ROUTER1#config t
ROUTER1(config)#crypto key generate rsa
The name for the keys will be: ROUTER1.company.com
Choose the size of the key modulus in the range of 360 to 4096 for your
  General Purpose Keys. Choosing a key modulus greater than 512 may take
  a few minutes.

How many bits in the modulus [512]: 1024
% Generating 1024 bit RSA keys, keys will be non-exportable...
[OK] (elapsed time was 0 seconds)

ROUTER1(config)#end
ROUTER1#

Now that the crypto key is generated I can set the proper version and key for SSH access.

ROUTER1#config t
ROUTER1(config)#ip ssh version 2
ROUTER1(config)#ip ssh rsa key ROUTER1.company.com
ROUTER1(config)#end
ROUTER1#

I then modify my SecureCRT host configuration back to SSH2 and run a quick connection test to this router. Viola! The router is now properly responding to the SSH connection! A quick write memory to save the running-configuration to startup, resubmit the failed Solarwinds backup script, and have a cup of coffee to celebrate!

Actually, the celebration may be premature. Remember, this device was operating normally for months. So what changed? Why did SSH go sideways? To start the investigation I needed to check some basic details. The output of this first command was a head scratcher.

ROUTER1#show version
Cisco IOS XE Software, Version 03.15.03.S - Standard Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(2)S3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Fri 25-Mar-16 16:59 by mcpre

< TRUNCATED >

ROM: IOS-XE ROMMON

ROUTER1 uptime is 1 day, 1 hour, 19 minutes
Uptime for this control processor is 1 day, 1 hour, 20 minutes
System returned to ROM by reload at 10:35:25 UTC Tue Oct 30 2018
System image file is "bootflash:isr4300-universalk9.03.15.03.S.155-2.S3-std.SPA.bin"
Last reload reason: Reload Command

< TRUNCATED >

This router has only been up for a day, and looked like it was caused by a reload! A quick check through email reminded me that this router was replaced to address the Cisco clock signal bug in the ISR router family. At least the router replacement activity explains the reload notices in the logs!

Hope this helps in the future.

Leave a Reply