Do you remember the first access control list you ever added to a Cisco router? Was probably something simple like blocking IP subnets on your Internet facing routers from known hostile countries like China and Russia. Any business-to-business communications would normally come through a firewall via IPSEC or TCP socket termination on a DMZ host. If you need deeper security measures other than port-filtering the firewall is the way to go.
But…
CORP Router
What happens when you have a scenario that should require better security, but your architecture recommendations are overruled. Despite your misgivings, that third party business partner has direct MPLS connections into your data center, rather than drop them into a DMZ? Other than documenting your concerns to your management there isn’t much you can do except salute and carry out the order. This is where a more complex ACL comes into play. You’ll really need to think about how TCP works with the SYN-ACK connection setup and which direction the ACL should be applied.
Here’s the scenario you need to deal with. You need to set up a new MPLS network for a third-party your company does business with. Unfortunately, VPNs aren’t appropriate because this company is based in China, and the Chinese government has a nasty habit of blocking encrypted traffic on a moments notice. The company you work with manages your shipments into and out of the APAC region, so you’ll need to prepare an ACL that allows the remote hosts access to systems like SAP and label printers. You’ll also need to have access to the remote routers (since you manage them) using SSH, SNMP, ICMP, and TFTP.
This gets a little more complicated than you would think! In this blog, as with others, I’ve included a sample configuration and the network diagram. The rest of this article will talk through some initial configuration, troubleshooting, and final configuration. I will assume that basic connectivity testing is verified; ICMP reachability between the PE and CPE routers, appropriate routes in the route table, etc. In this article, network engineers should know how to write an ACL, but when they don’t work, how do you figure out the problem? Especially when the syntax looks correct and you see some hits on the policy! Let’s start with the configuration.
Applying the extended ACL in a “normal” way, you’d no doubt create the object-groups for the TCP ports, host, and subnet objects. You would then apply that ACL to the inbound MPLS-facing interface. The common idea would be that any traffic originating on 192.168.0.0/16 would be allowed through the LAN side of the router (Gi0/0/1) and traffic from 10.133.3.0/24 would only be allowed if it matched the ACL applied to the WAN interface (Gi0/0/0). However, when you apply this to the interface, something unexpected happens. From a host on the Corporate network you can get full reachability when the ACL is not applied, but all traffic to the remote side drops when it’s applied. What will really throw you for a loop is if you allow ICMP, for example, in both of the applied ACLs. If you start pinging from one of the hosts on either side, you’ll get replies but if you SSH from one of the 192.168.0.0/16 hosts, the traffic will fail. When you’re trying to troubleshoot everything will look correct, so, here are a couple of steps that will help you decipher the crazy behavior.
CEF Check
Step one—after checking your ACL syntax—is to verify how Cisco Express Forwarding is interpreting the traffic flow, assuming it is enabled. Even though this seems like a stupid step and you and the router should “obviously” agree how to send the data, you’re actually not trying to verify forwarding. What you really want to do is verify that the ACLs are applied to the proper interfaces, and in the correct direction. You’ll take this information and use it in the next step.
CORP#show ip cef exact-route 10.133.3.4 192.168.200.49 10.133.3.4 -> 192.168.200.49 =>IP adj out of GigabitEthernet0/0/1, addr 192.168.250.33 CORP#show ip cef ex 192.168.200.49 10.133.3.4 192.168.200.49 -> 10.133.3.4 =>IP adj out of GigabitEthernet0/0/0, addr 192.168.1.1 CORP#
In the above output you see how each route is learned and that CEF forwarded along the proper interface. Compare that information to the direction and syntax of your ACL. It should look correct if you followed that traditional approach. You should see the inbound ACL on Gi0/0/1 CORP-REMOTE receiving hits for the remote network. The weird thing will be that you’ll get no reply. It’ll be easy to overlook a simple fact that source-destination addresses will change in the TCP header. Since you have two access control lists, and the headers change, does your ACL allow for the reply traffic?
TCP/IP Sources
Do you remember all the time you put into learning the OSI Model and how TCP packets were formed? This was all fundamentals and you thought you’d hardly ever have to think about frames and packet headers again. Did you promptly forgot about it when you realized you never had to dive that deep into a packet after a couple years in the enterprise? Well, here’s where that information pays off. When troubleshooting your ACL you need to remember the direction of the packet and the TCP ports that are being used in the flow. In this scenario for SSH management traffic to the customer network you would see this kind of flow:
| Inbound Interface | Source Address | Source Port | Outbound Interface | Destination Address | Destination Port |
| Gi0/0/1 | 192.168.40.130 | 1065 | Gi0/0/0 | 10.133.3.4 | 22 |
| Gi0/0/0 | 10.133.3.4 | 22 | Gi0/0/1 | 192.168.40.130 | 1065 |
With this flow you need to ask yourself if the ACL you’ve applied matches the proper interfaces? Does it match up with the CEF logic? Because you’re limiting your traffic on two different interfaces, you need to think of a proper ACL as two different lists. One for received traffic and one for sent. This fact is what makes troubleshooting connectivity through a router’s control list so frustrating. It’s not like the CCNA book. You have to think through the flow and make your decision based on that. Now, this is a bigger issue with TCP rather than UDP for an obvious reason; TCP is connection oriented whereas UDP isn’t. TCP is “knock knock, are you there?” “Yes, I’m here” “Good, I’m going to chat right now.” “Okay, let’s chat. But, is this what you actually said?” “Yep. You’re picking up what I’m dropping.” UDP is a shout at a person across the room. Hopefully he heard it and is forwarding that packet up the stack to the application.
The problem is if you add a rule on the opposite interface for the return traffic you may inadvertently open up your internal systems to attack. Look at this access control list. ACL 101 is not the same as ACL 102 for the reply traffic when applied to Gi0/0/1 and Gi0/0/0, respectively. In fact this would allow all WAN-inbound traffic on destination port TCP 22 to any host on 192.168.0.0/16. It would also allow any host on 192.168.0.0/16 to 10.133.3.0/24 over TCp22. But, anything off the 192.168.0.0/16 subnet would fail if it were destined to anything other than 10.133.3.0/24.
ip access-list extended 101 5 permit tcp 192.168.0.0 0.0.255.255 any eq 22
ip access-list extended 102 5 perm tcp any 192.168.0.0 0.0.255.255 eq 22
So what do you do about this? I can honestly say as I was troubleshooting it was incredibly frustrating until I started thinking through TCP, ACL application, and forwarding interfaces. I can’t open up the inbound and ruin the security posture, but I also need to open up that reply traffic so I can actually get connectivity. See the rub?
Packet Capture
If it’s still a mystery, try a packet capture. Where supported, Embedded Packet Captures (Cisco EPC) on the router are an option. This Cisco guide covers IOS-XE v16 if you want to look at this in more detail. A simple way to do that is to follow the example in these steps. Make changes as appropriate for you envirionment and be very careful to enable this on a busy router. The captures will hit the CPU, so don’t introduce an outage while troubleshooting your ACL. That could be a way to brush up your resume!
- Configure a capture ACL for the traffic you want to inspect.
ip access-list extended <ACL_NAME>
<seq_number> permit <protocol> <source_address> <wildcard_mask> <destination_address> <wildcard_mask> eq <destination_port>
ip access-list extended 101 5 permit tcp 192.168.0.0 0.0.255.255 host 10.133.3.4 eq 22 10 permit tcp host 10.133.3.4 192.168.0.0 0.0.255.255
2. Configure the data capture parameters
enable monitor capture mycap start monitor capture mycap access-list mycap monitor capture mycap limit duration 1000 monitor capture mycap interface gi0/0/0 both monitor capture mycap buffer circular size 10 monitor capture mycap start monitor capture mycap export bootflash:mycap.cap monitor capture mycap stop end
3. Copy the finished packet capture to a TFTP server
CORP#copy bootflash: tftp: Source filename []? mycap.cap Address or name of remote host []? 192.168.254.27 Destination filename [mycap.cap]? !! 5160 bytes copied in 0.393 secs (13130 bytes/sec) CORP#
4. Inspect the output of the capture in Wireshark, text editor, or other PCAP reader. This output is shown in text format but should be easy enough to read.
No. Time Source Destination Protocol Length Info
1 0.000000 192.168.200.49 10.133.3.4 TCP 74 60964 → 22 [SYN] Seq=0 Win=16384 Len=0 MSS=536 WS=1 TSval=399307709 TSecr=0
Frame 1: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0), Dst: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98)
Internet Protocol Version 4, Src: 192.168.200.49, Dst: 10.133.3.4
Transmission Control Protocol, Src Port: 60964, Dst Port: 22, Seq: 0, Len: 0
No. Time Source Destination Protocol Length Info
2 0.237039 10.133.3.4 192.168.200.49 TCP 58 22 → 60964 [SYN, ACK] Seq=0 Ack=1 Win=4128 Len=0 MSS=536
Frame 2: 58 bytes on wire (464 bits), 58 bytes captured (464 bits)
Ethernet II, Src: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98), Dst: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0)
Internet Protocol Version 4, Src: 10.133.3.4, Dst: 192.168.200.49
Transmission Control Protocol, Src Port: 22, Dst Port: 60964, Seq: 0, Ack: 1, Len: 0
No. Time Source Destination Protocol Length Info
3 0.237039 192.168.200.49 10.133.3.4 TCP 54 60964 → 22 [ACK] Seq=1 Ack=1 Win=16616 Len=0
Frame 3: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0), Dst: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98)
Internet Protocol Version 4, Src: 192.168.200.49, Dst: 10.133.3.4
Transmission Control Protocol, Src Port: 60964, Dst Port: 22, Seq: 1, Ack: 1, Len: 0
No. Time Source Destination Protocol Length Info
4 0.238031 192.168.200.49 10.133.3.4 SSHv2 80 Client: Protocol (SSH-2.0-OpenSSH_6.2 PKIX)
Frame 4: 80 bytes on wire (640 bits), 80 bytes captured (640 bits)
Ethernet II, Src: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0), Dst: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98)
Internet Protocol Version 4, Src: 192.168.200.49, Dst: 10.133.3.4
Transmission Control Protocol, Src Port: 60964, Dst Port: 22, Seq: 1, Ack: 1, Len: 26
SSH Protocol
No. Time Source Destination Protocol Length Info
5 0.475018 10.133.3.4 192.168.200.49 SSHv2 74 Server: Protocol (SSH-1.99-Cisco-1.25)
Frame 5: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98), Dst: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0)
Internet Protocol Version 4, Src: 10.133.3.4, Dst: 192.168.200.49
Transmission Control Protocol, Src Port: 22, Dst Port: 60964, Seq: 1, Ack: 1, Len: 20
SSH Protocol
No. Time Source Destination Protocol Length Info
6 0.475018 10.133.3.4 192.168.200.49 SSHv2 422 Server: Key Exchange Init
Frame 6: 422 bytes on wire (3376 bits), 422 bytes captured (3376 bits)
Ethernet II, Src: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98), Dst: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0)
Internet Protocol Version 4, Src: 10.133.3.4, Dst: 192.168.200.49
Transmission Control Protocol, Src Port: 22, Dst Port: 60964, Seq: 21, Ack: 27, Len: 368
SSH Protocol
No. Time Source Destination Protocol Length Info
7 0.476026 192.168.200.49 10.133.3.4 TCP 590 60964 → 22 [ACK] Seq=27 Ack=389 Win=16248 Len=536 [TCP segment of a reassembled PDU]
Frame 7: 590 bytes on wire (4720 bits), 590 bytes captured (4720 bits)
Ethernet II, Src: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0), Dst: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98)
Internet Protocol Version 4, Src: 192.168.200.49, Dst: 10.133.3.4
Transmission Control Protocol, Src Port: 60964, Dst Port: 22, Seq: 27, Ack: 389, Len: 536
No. Time Source Destination Protocol Length Info
8 0.476026 192.168.200.49 10.133.3.4 SSHv2 422 Client: Key Exchange Init
Frame 8: 422 bytes on wire (3376 bits), 422 bytes captured (3376 bits)
Ethernet II, Src: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0), Dst: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98)
Internet Protocol Version 4, Src: 192.168.200.49, Dst: 10.133.3.4
Transmission Control Protocol, Src Port: 60964, Dst Port: 22, Seq: 563, Ack: 389, Len: 368
[2 Reassembled TCP Segments (904 bytes): #7(536), #8(368)]
SSH Protocol
No. Time Source Destination Protocol Length Info
9 0.713013 10.133.3.4 192.168.200.49 TCP 54 22 → 60964 [ACK] Seq=389 Ack=563 Win=4128 Len=0
Frame 9: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98), Dst: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0)
Internet Protocol Version 4, Src: 10.133.3.4, Dst: 192.168.200.49
Transmission Control Protocol, Src Port: 22, Dst Port: 60964, Seq: 389, Ack: 563, Len: 0
No. Time Source Destination Protocol Length Info
10 0.914006 10.133.3.4 192.168.200.49 TCP 54 22 → 60964 [ACK] Seq=389 Ack=931 Win=3760 Len=0
Frame 10: 54 bytes on wire (432 bits), 54 bytes captured (432 bits)
Ethernet II, Src: JuniperN_c5:c9:98 (4c:16:fc:c5:c9:98), Dst: Cisco_dd:df:d0 (68:ca:e4:dd:df:d0)
Internet Protocol Version 4, Src: 10.133.3.4, Dst: 192.168.200.49
Transmission Control Protocol, Src Port: 22, Dst Port: 60964, Seq: 389, Ack: 931, Len: 0
Now that you have the connectivity, routes, CEF path, and packet capture to analyze, you should be seeing the solution. Even if you don’t know the syntax, hopefully it’s becoming clear.
ACL Resolved
Given the CEF entries, TCP flow, and the packet capture information it’s pretty clear you need to make the router do a rudimentary form of stateful packet inspection. What do I mean? Modern firewalls keep track of the data flow from one interface to another. Simply, if the router sees a rule-matching SYN, it will expect a replied ACK. In the hypothetical I’ve outlined, we have to manually build that functionality into the applied ACLs. The way to handle this traffic is to break it up into outbound/inbound plus replies. Look at this configuration snippet and then we’ll break it down.
CORP#show object-group
Network object group CORP-HOSTS
host 192.168.40.17
host 192.168.40.19
host 192.168.40.40
host 192.168.40.42
host 192.168.40.43
host 192.168.40.45
host 192.168.40.46
host 192.168.40.47
host 192.168.40.48
host 192.168.40.91
host 192.168.40.92
host 192.168.40.93
172.16.33.0 255.255.255.0
Service object group CORP-REMOTE-RX
icmp
icmp echo-reply
tcp source eq 22
udp source eq snmp
udp source eq snmptrap
udp source eq tftp
Service object group CORP-REMOTE-TX
icmp
icmp echo-reply
tcp eq 22
udp eq snmp
udp eq snmptrap
tcp eq 69
Service object group REMOTE-CORP-RX
tcp source eq 9100
tcp source eq www
tcp source range 3200 3299
tcp source range 3300 3399
tcp source range 3600 3699
tcp source range 4800 4899
tcp source range 8000 8001
tcp source eq 8010
tcp source eq 8080
tcp source range 8100 8199
tcp source eq 8443
tcp source range 50000 59914
tcp source eq 443
tcp source eq 22
icmp
icmp echo-reply
Service object group REMOTE-CORP-TX
tcp eq 9100
tcp eq www
tcp range 3200 3299
tcp range 3300 3399
tcp range 3600 3699
tcp range 4800 4899
tcp range 8000 8001
tcp eq 8010
tcp eq 8080
tcp range 8100 8199
tcp eq 8443
tcp range 50000 59914
tcp eq 443
tcp eq 22
icmp
icmp echo-reply
CORP#show ip access-list
Extended IP access list CORP-INBOUND
5 permit ospf any any
10 permit object-group CORP-REMOTE-TX 192.168.0.0 0.0.255.255 any
15 permit object-group REMOTE-CORP-RX object-group CRUS-HOSTS any
20 permit object-group CORP-REMOTE-TX 172.16.33.0 0.0.0.255 any
Extended IP access list REMOTE-INBOUND
5 permit tcp host 192.168.1.1 host 192.168.1.2 eq bgp
10 permit object-group CORP-REMOTE-RX any 192.168.0.0 0.0.255.255
15 permit object-group REMOTE-CORP-TX any object-group CRUS-HOSTS
20 permit object-group CORP-REMOTE-RX any 172.16.33.0 0.0.0.255
CORP#show run interface gi0/0/0
Building configuration...
Current configuration : 204 bytes
!
interface GigabitEthernet0/0/0
description WAN
bandwidth 25000
ip address 192.168.1.2 255.255.255.252
ip access-group REMOTE-INBOUND in
media-type rj45
negotiation auto
end
CORP#show run interface gi0/0/1
Building configuration...
Current configuration : 169 bytes
!
interface GigabitEthernet0/0/1
description LAN
ip address 192.168.250.34 255.255.255.252
ip access-group CORP-INBOUND in
shutdown
negotiation auto
end
CORP#
When you create a service object-group, the thing that will put your head into the swirling toilet bowl is the syntax of the permit/deny statements in the ACL. Normally you would say <seq_num> <permit/deny> <protocol> <source_address/host> <destination_address/host> <protocol/application> . If you follow this syntax with a service object-group you’ll have issues. It doesn’t work, and Cisco doesn’t let you do configure the rule that way in IOS/IOS-XE. The syntax changes with object-groups and the destination port moves to the front of the string.
In the above object-groups we’ve created the allowed protocols—which show up just after the host/subnet entry on the ACL to define the destination port—to be used in the rules. One item you’ll notice in the object-group is the keyword source . This tells the object-group that the source port is to equal the given value. The ACL applied to the opposite, inbound, interface needs to have this command. If you look back up at the chart under the TCP/IP Sources heading, this makes some sense. When the SYN packet goes out, it uses a random port higher than 1024 and connects to the well-known destination port on the server. But, when the ACK (and subsequent reply traffic) is sent back through that first interface, the source TCP port will be the well-known destination port. The ACL must reflect this conversation. If you’re having a tough time thinking through the conversation flow, it helps to write it out on a whiteboard like it’s illustrated in the chart above. The chart helps visualize the conversation and often times it is extremely helpful to see it written out.
The source keyword doesn’t change what you need to do for the rest of the ACL. I mean, you definitely create the filter for, in this case, the inbound traffic (based on the interface position). Again, the helpful hint in all of this is to remember that a reply needs to be sent from the host and must be allowed on the interface closest to that host. This leads me to a best practice discussion. It’s a REALLY good idea to install extended access-lists onto the inbound interface closest to the source being filtered. Why? Since extended ACLs filter on more specific packet details than standard ACLs, you only affect the specific host or subnet prior to consuming additional network resources. In this scenario, we’d normally place the ACL on the remote router. The issue with that is it will introduce additional management overhead because you’d need to manage each remote host individually, rather than the MPLS head-end “choke point” shown in this topology. ACL-inbound is still best because it keeps things standard…place the extended ACL on the interface closest to the network needing the filtering.
The ACL creation is pretty self explanatory. Going into its creation is beyond the scope of this article but just remember what was said before. Think sequence number, protocol, source, destination, protocol number. The order is important within an ACL since it looks for a match from top to bottom. If you say ’10 ip permit any any” all IP traffic will be processed prior to ’25 tcp permit 192.168.1.0 0.0.0.255 any eq http’. Pretty much breaks your security posture.
Once you’re finished completing the ACL and object-group and are testing, a quick show ip access-list will give you some information about which rules in the ACL are actually being hit. Be aware that the log keyword on a rule will hit the router/switch processor. If the processor is busy, best leave that off. In any case, you should see hits on your rules. Now, something I wish Cisco would do would be to add a hit counter to the individual objects in the object-list. If you get hits on the ACL, you won’t know in particular what rule is being triggered.
CORP#show ip access-list
Extended IP access list CRUS-INBOUND
5 permit ospf any any (455 matches)
10 permit object-group CORP-REMOTE-TX 192.168.0.0 0.0.255.255 any (3408 matches)
15 permit object-group REMOTE-CORP-RX object-group CRUS-HOSTS any (42 matches)
20 permit object-group CORP-REMOTE-TX 172.16.31.0 0.0.0.255 any (12 matches)
Extended IP access list JSI-INBOUND
5 permit tcp host 192.168.1.1 host 192.168.1.2 eq bgp (124228 matches)
10 permit object-group CORP-REMOTE-RX any 192.168.0.0 0.0.255.255 (2936 matches)
15 permit object-group REMOTE-CORP-TX any object-group CRUS-HOSTS (16 matches)
20 permit object-group CORP-REMOTE-RX any 172.16.31.0 0.0.0.255 (29 matches)
CORP#
The output here shows we’re getting traffic through the router’s ACL, on both interfaces. At this point you’d want to have the application owners test their connectivity and adjust as required.
Wrapping Up
The key to making this scenario work is to recognize the two-way nature of the client/server communication, and, how the ACLs need to be applied to the closest inbound interface. Understanding how to troubleshoot the flow into and out of the router only helps when you remember the basics of TCP/IP and ACL creation. Hopefully this will clarify some trouble spots in your own network and you can apply it.