Beagles are the best. Friendly. Personable. Very stubborn. And alert. If you own one, you understand the struggle. If anything walks by a window, the howling and alerting of “bad things” happening can be attributed to your beagle. I have two. The crescendo is deafening. Reminds me of my network operations days when things went bad and I didn’t know about the bad things.

A friend of mine over at PathSolutions reached out not too long ago. He was having some issues monitoring Fortinet FortiSwitches that were managed by a FortiGate and since I’m an SE at Fortinet, maybe I could help. Apparently, the beagles weren’t howling at problems. It seemed that all the SNMP responses were limited to whatever was configured on the FortiGate. I love Fortinet’s security, the fabric, and shared information along the Forti-Path. I wasn’t surprised, however, that there were troubles in paradise from Fortinet partners.

Configuring the FortiGate and FortiSwitch to allow for SNMP monitoring isn’t very difficult, on the global side. But if you want it to work cross-platform, it involves some CLI work, especially to get the results you expect to see. I’ve known PathSolutions TotalView (check Tech Field Day NFD20) for a few years, and they are a heavy hitter in SNMP monitoring. Tim Titus, Steve Winter, and the team are Johnny on the Spot for operations teams. The configuration settings to help PathSolutions shown here, will work for other SNMP-based monitoring platforms, in most instances, but I love TotalView, so I’ll use them as the test case.

The Problem

The issue we’re tackling in this article is pretty straight forward. When you enable SNMP on a FortiGate, you’ll see all the physical and logical interfaces configured on the box. The problem comes when you want to monitor the interfaces on a FortiSwitch managed by the FortiGate; you never see the switch in the monitoring platform. It just doesn’t display. There are a few reasons this happens, but, I’ll get to that.

PathSolutions uses standard, or general, OIDs for polling. Anything vendor-specific may cause a breakage concern in subsequent code releases. Makes sense and is a good approach to coding…use something standard and very stable so you don’t have to create a new release with everyone else’s release. When doing the initial SNMP-polling from TotalView using the Fortinet-documented SNMP configuration on the FortiGate, again, the only response was from the FortiGate and the switch was never displayed. A MIB walk with the vendor-specific OIDs showed promising results from the switch, but we really don’t want to risk the breaking of functionality on the monitoring platform. Why weren’t we getting FortiSwitch interface results?

A packet capture during testing wasn’t very helpful, either. Seems like SNMP polls were being initiated, but nothing was returned except a ‘destination port unreachable’ message. But, I’ve configured SNMP globally, so the switch should be listening! Why does the vendor specific OID work? Why not a general OID? When using the TotalView Interface tool, you’ll see no values being returned for the FortiSwitch, as shown in the image to the right.

Even after following the FortiDocs, there wasn’t a satisfactory result. Something was missing. That issue boils down to a few items, and frankly, it may help all network engineers understand the Fortinet platform:

• There is a default security policy that is applied to Fortilink interfaces. If you recognize that every interface created on Fortinet devices can have management capability, you’ll scratch your head wondering why the Fortilink interface doesn’t show up in the GUI. At some point you have to enable certain management protocols, but, Fortilink doesn’t give you the option. It’s a CLI-only option. This happens a LOT with FortiOS.
• You have to enable SNMP on the FortiGate managed controllers, not just the global SNMP setting. Every FotiSwitch, via the FortiGate controller, must have SNMP enabled. The global setting you configure in SYSTEM | SNMP doesn’t work for downstream Fortinet devices.
• Finally, create a firewall policy, bi-directionally, to give your SNMP host access to the FortiSwitches and FortiAPs. Remember, FortiGates are stateful, which is TCP-based. If you’re dealing with fire-and-forget-UDP, you’ll need a bi-directional rule.

Security Focused Fabric

If you’re a data center focused network engineer, to understand Fortinet you have to adjust your thinking. Fortinet is a 100% security focused company that “does” networking. That sounds like a pejorative, but it’s really not. Fortinet is very good at network technologies, but you have to change your thinking as a network person. Think security, then, passing packets. Think security, then network overlays. Think security, then how interfaces are created. With Fortinet, everything is an interface, and everything has an applied security policy, even if you can’t find the document that says, “HEY HERE’S THE APPLIED POLICY”. I can’t be more blunt. Everything on core-Fortinet is an interface.

To enable SNMP monitoring of attached FortiSwitches, you first have to change the default FORTILINK policy. There is a non-GUI’d configuration you must edit. It enables access to the Fortilink interface and it’s this interface that provides access to the managed FortiSwitch. Again, everything is an interface. If you want to touch it, you’d better have a policy and a layer-3 interface enabled. Remember that SNMP is on UDP 161. For a stateful firewall, there’s no state to remember. When creating your firewall rule for SNMP, you’ll need to add a reverse rule; the host makes the request and the target answers…not a reply in the TCP sense.

That also means you need to create a subnet for FortiGate managed devices. Fortinet uses the expanded RFC for 169.x.x.x address space. The issue is Microsoft uses that same address space when a client fails a DHCP request. If you open up network reachability to the 169-address, it could screw up your client community. Best thing would be to carve out an IPv4 subnet, that is reachable, for your Fortilink/FortiSwitch management.

config firewall policy
edit <ID_NUM>
    set name "FORTILINK SNMP ACCESS"
    set srcintf "<SUBNET_OBJECT>"
    set dstintf "<FORTILINK_SUBNET_OBJECT>"
    set action accept
    set srcaddr "<SUBNET_IP_OBJECT>"
    set dstaddr "FORTILINK"
    set schedule "always"
    set service "SNMP" "ALL_ICMP"
    set logtraffic all
    set comments "SNMP ACCESS TO FORTILINK-CONNECTED DEVICES"
  next
edit <ID_NUM>
    set name "FORTILINK SNMP ACCESS RVRS"
    set srcintf "<FORTILINK_SUBNET_OBJECT>"
    set dstintf "<SUBNET_OBJECT>"
    set action accept
    set srcaddr "FORTILINK"
    set dstaddr "<SUBNET_IP_OBJECT>"
    set schedule "always"
    set service "SNMP" "ALL_ICMP"
    set logtraffic all
    set comments "SNMP ACCESS TO FORTILINK-CONNECTED DEVICES - Reverse"
    next
  exit
exit

Once the policy allowing access to the FORTILINK subnet is enabled, now it’s time to start applying the SNMP configurations. If you look through the Fortinet documentation you’ll not likely find what you’re looking for. As I was looking, I checked the SNMP configuration docs for both the FortiGate and FortiSwitch but nothing worked. I still have not figured out what the difference between the ‘global’ and ‘local’ SNMP configuration is. I mean, if it’s global, why doesn’t it apply globally? Well, neither here nor there…just add these settings to the switch-controller branch.

The first section addresses the local security policy. This is the security applied to the switch controller and any devices that are connected for end-point connectivity. Poorly documented, never ignored. This will get you a step closer to monitoring your devices. Looks just like the management access of all your other interfaces, right?

config switch-controller security-policy local-access
    edit "default"
      set mgmt-allowaccess https ping ssh snmp
      set internal-allowaccess https ping ssh snmp
    next
end

The above snippet may add a few extra management protocols, so be certain you match your corporate security posture to the needed protocols. If you don’t need direct SSH access to a managed FortiSwitch, then there’s no reason to enable this on the allowaccess configuration. Remember your security focus…only enable protocols that are required.

The next part is to enable SNMP access on your downstream devices. Ignore the docs that talk about global because, well, whatever. Already mentioned that. Adding SNMP configuration to the FortiOS switch-controller is the red-pill of visibility.

config system snmp community
    edit <ID_NUM>
        set name "<COMMUNITY_NAME>"
        config hosts
            edit <ID_NUM>
                set ip <SUBNET> <NETMASK>
                set host-type query
            next
            edit <ID_NUM>
                set ip <HOST_IP> 255.255.255.255
                set host-type query
            next
         end
         set events cpu-high mem-low log-full intf-ip vpn-tun-up vpn-tun-down ha-switch ha-hb-failure ips-signature ips-anomaly av-virus av-oversize av-pattern av-fragmented fm-if-change fm-conf-change bgp-established bgp-backward-transition ha-member-up ha-member-down ent-conf-change av-conserve av-bypass av-oversize-passed av-oversize-blocked ips-pkg-update ips-fail-open faz-disconnect wc-ap-up wc-ap-down fswctl-session-up fswctl-session-down load-balance-real-server-down device-new per-cpu-high
     next
end
config switch-controller snmp-sysinfo
    set status enable
    set description "<DESCRIPTION>"
    set contact-info "<CONTACT_INFO>"
end
config switch-controller snmp-community
    edit <ID_NUM>
      set name "<COMMUNITY_NAME>"
      config hosts
          edit <ID_NUM>
              set ip <SUBNET> <NETMASK>
              set host-type query
          edit <ID_NUM>
              set ip <HOST_IP> 255.255.255.255
              set host-type query
      next
end
set events cpu-high mem-low log-full intf-ip ent-conf-change
next
end

There are some options in this configuration that may not be required for your environment; namely, the events. The events options are for SNMP traps, and many times, they aren’t needed. I’ve shown them here to show you can leverage them as required. I’m also assuming you’ll be able to figure out which objects are created in the firewall so you can call them out for source/destination networks and devices.

Final Testing

Once you have added these parameters, it’s a good idea to test SNMP connectivity. You’ll notice that prior to these settings you’ll have the FortiGate-only, but after these changes, you get the FortiGate, FortiSwitch, and if you go further, FortiAPs. Shameless plug for TotalView is that after I got the SNMP polling operational, I found an issue on one of my FortiSwitch interfaces. Apparently a workstation had negotiated to 10Mbps Half-Duplex. Easy fix when you know about the problem.

After all of these changes you should be able to do a MIB walk with your favorite stroller against all things FORTILINK. It’s obviously important that your monitoring solutions be able to see everything on your network. Not only see it, but also alert and/or take automated action against the problem source. Many devices are transitioning to telemetry and API’s, but, within the transition, we still have SNMP.

Use the tools as needed to keep your devices up and running.