Those Pesky ACLs

Do you remember the first access control list you ever added to a Cisco router? Was probably something simple like blocking IP subnets on your Internet facing routers from known hostile countries like China and Russia. Any business-to-business communications would normally come through a firewall via IPSEC or TCP socket termination on a DMZ host. If you need deeper security measures other than port-filtering the firewall is the way to go.


Configuration Files
CORP Router

What happens when you have a scenario that should require better security, but your architecture recommendations are overruled. Despite your misgivings, that third party business partner has direct MPLS connections into your data center, rather than drop them into a DMZ? Other than documenting your concerns to your management there isn’t much you can do except salute and carry out the order. This is where a more complex ACL comes into play. You’ll really need to think about how TCP works with the SYN-ACK connection setup and which direction the ACL should be applied.

Here’s the scenario you need to deal with. You need to set up a new MPLS network for a third-party your company does business with. Unfortunately, VPNs aren’t appropriate because this company is based in China, and the Chinese government has a nasty habit of blocking encrypted traffic on a moments notice. The company you work with manages your shipments into and out of the APAC region, so you’ll need to prepare an ACL that allows the remote hosts access to systems like SAP and label printers. You’ll also need to have access to the remote routers (since you manage them) using SSH, SNMP, ICMP, and TFTP.

Diagram: Application flow

This gets a little more complicated than you would think! In this blog, as with others, I’ve included a sample configuration and the network diagram. The rest of this article will talk through some initial configuration, troubleshooting, and final configuration. I will assume that basic connectivity testing is verified; ICMP reachability between the PE and CPE routers, appropriate routes in the route table, etc. In this article, network engineers should know how to write an ACL, but when they don’t work, how do you figure out the problem? Especially when the syntax looks correct and you see some hits on the policy! Let’s start with the configuration.

Applying the extended ACL in a “normal” way, you’d no doubt create the object-groups for the TCP ports, host, and subnet objects. You would then apply that ACL to the inbound MPLS-facing interface. The common idea would be that any traffic originating on would be allowed through the LAN side of the router (Gi0/0/1) and traffic from would only be allowed if it matched the ACL applied to the WAN interface (Gi0/0/0). However, when you apply this to the interface, something unexpected happens. From a host on the Corporate network you can get full reachability when the ACL is not applied, but all traffic to the remote side drops when it’s applied. What will really throw you for a loop is if you allow ICMP, for example, in both of the applied ACLs. If you start pinging from one of the hosts on either side, you’ll get replies but if you SSH from one of the hosts, the traffic will fail. When you’re trying to troubleshoot everything will look correct, so, here are a couple of steps that will help you decipher the crazy behavior.

CEF Check

Step one—after checking your ACL syntax—is to verify how Cisco Express Forwarding is interpreting the traffic flow, assuming it is enabled. Even though this seems like a stupid step and you and the router should “obviously” agree how to send the data, you’re actually not trying to verify forwarding. What you really want to do is verify that the ACLs are applied to the proper interfaces, and in the correct direction. You’ll take this information and use it in the next step.

In the above output you see how each route is learned and that CEF forwarded along the proper interface. Compare that information to the direction and syntax of your ACL. It should look correct if you followed that traditional approach. You should see the inbound ACL on Gi0/0/1 CORP-REMOTE receiving hits for the remote network. The weird thing will be that you’ll get no reply. It’ll be easy to overlook a simple fact that source-destination addresses will change in the TCP header. Since you have two access control lists, and the headers change, does your ACL allow for the reply traffic?

TCP/IP Sources

Do you remember all the time you put into learning the OSI Model and how TCP packets were formed? This was all fundamentals and you thought you’d hardly ever have to think about frames and packet headers again. Did you promptly forgot about it when you realized you never had to dive that deep into a packet after a couple years in the enterprise? Well, here’s where that information pays off. When troubleshooting your ACL you need to remember the direction of the packet and the TCP ports that are being used in the flow. In this scenario for SSH management traffic to the customer network you would see this kind of flow:

Inbound Interface Source Address Source Port Outbound Interface Destination Address Destination Port
Gi0/0/1 1065 Gi0/0/0 22
Gi0/0/0 22 Gi0/0/1 1065

With this flow you need to ask yourself if the ACL you’ve applied matches the proper interfaces? Does it match up with the CEF logic? Because you’re limiting your traffic on two different interfaces, you need to think of a proper ACL as two different lists. One for received traffic and one for sent. This fact is what makes troubleshooting connectivity through a router’s control list so frustrating. It’s not like the CCNA book. You have to think through the flow and make your decision based on that. Now, this is a bigger issue with TCP rather than UDP for an obvious reason; TCP is connection oriented whereas UDP isn’t. TCP is “knock knock, are you there?” “Yes, I’m here” “Good, I’m going to chat right now.” “Okay, let’s chat. But, is this what you actually said?” “Yep. You’re picking up what I’m dropping.” UDP is a shout at a person across the room. Hopefully he heard it and is forwarding that packet up the stack to the application.

The problem is if you add a rule on the opposite interface for the return traffic you may inadvertently open up your internal systems to attack. Look at this access control list. ACL 101 is not the same as ACL 102 for the reply traffic when applied to Gi0/0/1 and Gi0/0/0, respectively. In fact this would allow all WAN-inbound traffic on destination port TCP 22 to any host on It would also allow any host on to over TCp22. But, anything off the subnet would fail if it were destined to anything other than

So what do you do about this? I can honestly say as I was troubleshooting it was incredibly frustrating until I started thinking through TCP, ACL application, and forwarding interfaces. I can’t open up the inbound and ruin the security posture, but I also need to open up that reply traffic so I can actually get connectivity. See the rub?

Packet Capture

If it’s still a mystery, try a packet capture. Where supported, Embedded Packet Captures (Cisco EPC) on the router are an option.  This Cisco guide covers IOS-XE v16 if you want to look at this in more detail. A simple way to do that is to follow the example in these steps. Make changes as appropriate for you envirionment and be very careful to enable this on a busy router. The captures will hit the CPU, so don’t introduce an outage while troubleshooting your ACL. That could be a way to brush up your resume!

  1. Configure a capture ACL for the traffic you want to inspect.

ip access-list extended <ACL_NAME>
<seq_number> permit <protocol>  <source_address> <wildcard_mask> <destination_address> <wildcard_mask> eq <destination_port>

2. Configure the data capture parameters

3. Copy the finished packet capture to a TFTP server

4. Inspect the output of the capture in Wireshark, text editor, or other PCAP reader. This output is shown in text format but should be easy enough to read.

Now that you have the connectivity, routes, CEF path, and packet capture to analyze, you should be seeing the solution. Even if you don’t know the syntax, hopefully it’s becoming clear.

ACL Resolved

Given the CEF entries, TCP flow, and the packet capture information it’s pretty clear you need to make the router do a rudimentary form of stateful packet inspection. What do I mean? Modern firewalls keep track of the data flow from one interface to another. Simply, if the router sees a rule-matching SYN, it will expect a replied ACK. In the hypothetical I’ve outlined, we have to manually build that functionality into the applied ACLs. The way to handle this traffic is to break it up into outbound/inbound plus replies. Look at this configuration snippet and then we’ll break it down.

When you create a service object-group, the thing that will put your head into the swirling toilet bowl is the syntax of the permit/deny statements in the ACL. Normally you would say  <seq_num> <permit/deny> <protocol> <source_address/host> <destination_address/host> <protocol/application> . If you follow this syntax with a service object-group you’ll have issues. It doesn’t work, and Cisco doesn’t let you do configure the rule that way in IOS/IOS-XE. The syntax changes with object-groups and the destination port moves to the front of the string.

In the above object-groups we’ve created the allowed protocols—which show up just after the host/subnet entry on the ACL to define the destination port—to be used in the rules. One item you’ll notice in the object-group is the keyword  source . This tells the object-group that the source port is to equal the given value. The ACL applied to the opposite, inbound, interface needs to have this command. If you look back up at the chart under the TCP/IP Sources heading, this makes some sense. When the SYN packet goes out, it uses a random port higher than 1024 and connects to the well-known destination port on the server. But, when the ACK (and subsequent reply traffic) is sent back through that first interface, the source TCP port will be the well-known destination port. The ACL must reflect this conversation. If you’re having a tough time thinking through the conversation flow, it helps to write it out on a whiteboard like it’s illustrated in the chart above. The chart helps visualize the conversation and often times it is extremely helpful to see it written out.

The  source keyword doesn’t change what you need to do for the rest of the ACL. I mean, you definitely create the filter for, in this case, the inbound traffic (based on the interface position). Again, the helpful hint in all of this is to remember that a reply needs to be sent from the host and must be allowed on the interface closest to that host. This leads me to a best practice discussion. It’s a REALLY good idea to install extended access-lists onto the inbound interface closest to the source being filtered. Why? Since extended ACLs filter on more specific packet details than standard ACLs, you only affect the specific host or subnet prior to consuming additional network resources. In this scenario, we’d normally place the ACL on the remote router. The issue with that is it will introduce additional management overhead because you’d need to manage each remote host individually, rather than the MPLS head-end “choke point” shown in this topology. ACL-inbound is still best because it keeps things standard…place the extended ACL on the interface closest to the network needing the filtering.

The ACL creation is pretty self explanatory. Going into its creation is beyond the scope of this article but just remember what was said before. Think sequence number, protocol, source, destination, protocol number. The order is important within an ACL since it looks for a match from top to bottom. If you say ’10 ip permit any any” all IP traffic will be processed prior to ’25 tcp permit any eq http’. Pretty much breaks your security posture.

Once you’re finished completing the ACL and object-group and are testing, a quick  show ip access-list will give you some information about which rules in the ACL are actually being hit. Be aware that the  log keyword on a rule will hit the router/switch processor. If the processor is busy, best leave that off. In any case, you should see hits on your rules. Now, something I wish Cisco would do would be to add a hit counter to the individual objects in the object-list. If you get hits on the ACL, you won’t know in particular what rule is being triggered.

The output here shows we’re getting traffic through the router’s ACL, on both interfaces. At this point you’d want to have the application owners test their connectivity and adjust as required.

Wrapping Up

The key to making this scenario work is to recognize the two-way nature of the client/server communication, and, how the ACLs need to be applied to the closest inbound interface. Understanding how to troubleshoot the flow into and out of the router only helps when you remember the basics of TCP/IP and ACL creation. Hopefully this will clarify some trouble spots in your own network and you can apply it.

Brian Gleason is a full-time Lead Network Engineer for an Austin, Tx company and is currently pursuing the Cisco Certified Internetwork Expert, Data Center certification. He also teaches firearms in his spare time after being a husband to his wonderful wife and father to his three awesome kids. Brian was also selected as a delegate to Network Field Day 20 held in San Jose, CA.

Leave a Reply