Fortinet Security POC - SD-WAN & SD-Branch

Fortinet Security

SD-WAN/SD-Branch Testing (Part 1 & 2)

Introduction

As SD-WAN/SD-Branch solutions have matured, we often see typical requests from customers for solutions they are researching. A holistic SD-WAN/SD-Branch solution will provide support for the loss of possible data centers, branch underlay failures, and also a robust solution for brown out scenarios and intelligent path selection by defined cost and several network performance metrics.

In addition to having flexible data center designs with possible primary/primary or primary/secondary data centers for their SD-WAN solution, customers are looking for a solution that also affords the ability to manage and monitor their SD-Branch environment from a single pane of glass.

Fortinet's SD-WAN/SD-Branch solution can provide this result by utilizing SD-WAN features of their FortiGate Firewalls. The Fortinet solution allows for a centralized management, logging and monitoring solution with the use of FortiManager to fully manage the environment. Fortinet's solution is truly 'Controllerless' and will continue to operate normally should any or all SD-Branch devices lose connectivity with FortiManager.

Fortinet Secure SD-WAN leverages and provides: 

• Path failover moving flows from an under-performing transport to a transport that performs better.
• Remote destination monitoring and steering detecting issues at a remote location and re-routing traffic through optimal paths.
• Link aggregation taking advantage of multiple WAN transports.
• Active path performance metrics viewing WAN underlay metrics and trends.
• Application performance improvement.
• Improving user experience by using a variety of techniques, such as Forward Error Correction, Packet Duplication, QoS, and WAN Optimization.

Fortinet SD-WAN determines which path best meets performance expectations or service-level agreements (SLAs) for a particular application and assigns application flows to that WAN path.

The team at the WWT Advance Technology Center (ATC) tested the core functionality and usability of Fortinet's solution and in this two-part series we will look at the following items:

• Part 1
  • SD-Branch Provisioning
  • Routing and Site-to-Site Auto-Discovery Virtual Private Network (ADVPN)
  • SD-WAN Features – Traffic Flow Rules and SLA's
• Part 2
  • SD-WAN Failover
  • SD-WAN Automation
  • SD-WAN Monitoring and Reporting

In part one of testing, we first look at SD-Branch provisioning utilizing Fortinet's ZTP (Zero Touch Provisioning) process of building an entire SD-Branch. FortiManager is the central pane of glass to achieve this, and we will take a closer look at building device templates for common configuration items, device blueprints and meta-variables for ease of provisioning and grouping. Next, we take a closer look at the use of ADVPN for dynamically creating secure tunnels automatically between sites when site-to-site traffic patterns exist. We also look at how ADVPN routing will change automatically should SLAs not be met on a primary ADVPN link. Finally, we look at the use SD-WAN Rules and performance SLAs to influence traffic steering for applications.

Part two consisted of taking a closer look at failover scenarios, from brownouts due to induced latency, Hub HA Failover, to entire Hub connectivity loss to both provider circuits. Second, we look at a few of the API calls available into FortiManager to verify usability and then take a quick look at FortiSoC automations in FortiAnalyzer. Lastly, we review the SD-WAN specific reporting and monitoring capabilities of both FortiManager and FortiAnalyzer.

Testing Environment

This lab was deployed on a hypervisor allowing for a virtual environment with multiple virtual machines and Fortinet products and the addition of three physical devices that were deployed as Branch-3. Branch-3 consisted of a FortiGate Firewall, a FortiSwitch and a FortiAP. From the lab topology diagram below, you will see that we have 2 data centers that were configured as Active/Passive Hubs along 3 Remote Branches consisting of two Branches with Virtual Machine (VM) configurations and one Branch that is all physical hardware. The DCs had a DCI (Data Center Interconnect) between them as well for a backup path between DCs should both provider circuits fail.  Also, you can see that the FortiManager and FortiAnalyzer were both hosted in DC1.

Every FortiGate was connected to a simulated MPLS cloud and connected to a simulated Internet cloud that had direct internet access. Both simulated clouds were deployed on WWTs own proprietary software, TrafficJam. TrafficJam allows full BGP peering across the MPLS cloud and affords us the ability to introduce impairments such as latency, jitter and packet loss on any link. Ixia Breaking Point was used to create application specific flows throughout the environment providing the opportunity for FortiGate Application Control Engine to identify the applications in the flows.

Testing Components

The Fortinet Secure SD-WAN consists of several components: 

• FortiGate NGFW, which runs FortiOS, is the core of Secure SD-WAN.
• FortiManager for the orchestration and management plane.
• FortiAnalyzer for advanced analytics and automation.

Fortinet Secure SD-WAN solution can be extended to Secure SD-Branch. SD-Branch consists of the following components: 

• FortiSwitch to provide security on the wired LAN edge.
• FortiAP to provide WiFi access to users.

The following image shows the components of the Fortinet Secure SD-WAN architecture, including the items mentioned above, along with additional Fortinet solutions that extend beyond SD-WAN. This provides complete coverage across the branch to further build out the Fortinet Security Fabric, providing broad visibility, integration and automation for organizations.

Testing Highlights

SD-Wan Provisioning

In SD-WAN Provisioning we wanted to review the templates and settings used to create our Active/Passive Hub and Remote Branch topology. We start by taking a close look at the SD-WAN Overlay Template and how it is used to create a few other auto-generated templates such as IPSEC Tunnel Templates, SD-WAN Templates and BGP Templates. The Overlay Template is where we define items such as the Loopback Interface and Overlay Network Address spaces and BGP-AS Number we want the SD-WAN topology to use and whether we want to enable ADVPN to allow for dynamic tunnels to be built between Remote Sites when Site-to-Site communications are required.

Working through the Overlay Template, we also configure which devices are the hub site devices and which hub is primary and secondary in our case for Primary/Secondary Dual Hub design. As we continue to fill out the Overlay Template we also define which interfaces are part of the Underlay. For the Branch side we show the use of meta-variables to define the underlay interfaces. This is due in part to branches that may have many different models deployed depending on capacity requirements for particular sites. This allows us to create CSV files that provide device Blue Prints of information relevant to the remote site such as, but not limited to IP Addresses, Site Name, Interface Identification, Device Serial Number, etc. that are all used to build out the remote site during the ZTP (Zero Touch Provisioning) process. 

Once we have built our base configuration for the SD-WAN Topology using the Overlay Template, we walk through the Templates we used to configure the rest of the configuration items we want pushed to the onboarding devices during ZTP. We show both CLI Script based and Jinja Script based templates used and how we used the meta-variables to populate items for that branch and also how Jinja Scripts can include Logical and Mathematical Operations when parsing information to build configurations. In our testing we created many smaller templates to do specific things instead of creating one large template making it easier to find where you may want to change certain items instead of deciphering and searching long templates to find an item you may want to modify.

After building all the necessary templates to build a site we then look at building the CSV file used to define parameters for the Remote Site that are used in many of the templates. Importing this information is very fast and convenient and makes it much easier to define and deploy one to many sites at the same time. As our testing shows, there we a couple of caveats we found when trying to deploy Remote Sites using ZTP. Our VM FortiGate did not match the Model Device provided within FortiManager itself when it came to the Logging Disk. Our VM had a logging disk while the built-in FortiManger Model device was not expecting a log disk. While not ideal we identified this as a configurable item on the model device when you edit the device in the Device Manager area. You can toggle the CLI Option "Has Log Disk" as some VMs will have a log disk while some may not as this is a configurable option. 

For the Physical Device Branch, we had an issue around order of operations when trying to configure the FortiGate, FortiSwitch and FortiAP in a single ZTP push. We were able to provision the FortiGate, pre-authorize and provision the FortiSwitch, but unable to pre-authorize and provision the FortiAP at the same time. The FortiAP would not provision due to the fact FMG tries to provision the "System WiFi SSID Interface" before defining the "Wireless-Controller vap." This led us to a scenario in which we provisioned the FGT and FortiSwitch using ZTP and pushed the FortiAP configuration after that. Fortinet has indicated that this should be resolved in version 7.2.5 of FortiManager and all subsequent releases.

The below image shows all the different CLI Templates that were used for the Physical ZTP deployment. Notice that we added a 3-digit leading number to the templates to define the order we want them to execute in as each template will execute from the top down as shown and order will matter in certain cases. Also notice we have some CLI scripts with "Jbranch" in the name just to indicated for reference that it is a Jinja CLI Script while just "Branch" indicates a CLI Script. Fortinet has indicated that Jinja Script will be the preferred method moving forward to allow for more flexibility and versatility on possible script logic.

Although we had a couple unexpected issues, there was always a way to remediate the items we ran into, which shows the flexibility of FortiManager. This may even be a preferred way to deploy in your environment by using ZTP to deploy a very basic configuration on the Remote FortiGate just to ensure it is online and connected to both providers. Once the basic configuration is verified, then push the rest of the configuration items to the site to completely build out routing, underlay, overlay, policies and provision other devices such as FortiSwitch and FortiAPs.

Routing and Site-to-Site ADVPN

In Test Case 2, we show the simple configuration and use of ADVPN (Auto-Discovery VPN) for Remote Site-to-Site communications. ADVPN will dynamically build an IPSEC Tunnel between two sites should there be traffic between two sites. Please note that ADVPN is not a requirement and can simply be turned on or off from the first dialog box in the SD-WAN Overlay Template used to define our SD-WAN topology. If you choose to allow ADVPN for Site-to-Site communications, only the first few packets destined from one Remote Site to another will flow through the DC/HUB. Once the HUB device sees that it is traffic between two Remote Sites it will have the two sites build their own IPSEC Tunnel between them and offload that traffic to the new tunnel so that the traffic will not have to traverse the HUBs.

When the ADVPN Tunnel is created, a new Performance SLA is created between the two tunnel endpoints that monitors ICMP performance. Since we have two providers there will actually be two new tunnels created across each provider. Should the Performance SLA of the Primary path not meet SLAs, the traffic will automatically be pushed to the alternate, better performing, ADVPN path between the sites. While the configuration to enable ADVPN is simple, you will still need to insure your Firewall Policies account for site-to-site communications and monitoring of the tunnel endpoint IPs as shown in our testing.

SD-WAN Features – Traffic Flow Rules and SLA's

This test case looks at how to use SD-WAN Rules and Performance SLAs to influence traffic steering of application traffic across preferred path for the defined traffic types. This is one of the features that is a draw to SD-WAN as traffic can be identified in many ways and influenced to traverse the path you want it to. You may want highly sensitive traffic, VoIP traffic, etc., to traverse your MPLS Network for security, reliability and speed, while other non-critical traffic uses a lower cost provider circuit or allow Direct Internet Access without traversing a DC/HUB.

To identify applications, we need to use FortiGate's Application Control feature and also deep inspection for traffic that is encapsulated in an Encrypted HTTPS Session. Once we can identify the applications, we then can build out our SD-WAN rules that define the application and criteria used to direct traffic to the path you choose. Each rule will also use a defined Performance SLA that is used to monitor the performance of a path to ensure it is meeting the requirements needed to have reliable communications for the application.

The Performance SLAs used to monitor traffic paths allow you to set all the parameters required to have reliable traffic using different parameters. You can define active or passive probes, what protocol such as ping is used, destination IP to monitor as well as thresholds for Latency, Jitter, Packet Loss, etc., needed to be met for that traffic.

While we will be doing further failover testing, we did show in this test how introducing latency, jitter or packet loss affects where traffic flows as defined by the SD-WAN Rules and Performance SLAs we created. We show in the GUI as well as the CLI how you can verify the path traffic is taking pre and post of introducing an issue or failure of a provider link.

SD-WAN Failover

This test starts by taking a closer look at failover scenarios and how traffic can survive degradations in circuit quality by monitoring SLAs, DC HA Pair Failovers and a Full DC Failover when configured as Active/Passive Hubs for the SD-WAN environment. We started by showing how a FortiGate HA pair can failover between the two units and continue to service already established sessions as long as Session Pickup was enabled for the HA configuration.

Next, we continued our failover testing by introducing Latency or Jitter to our circuits to show how multiple paths can be configured for traffic to traverse. We show how the SD-WAN rules define the preferred order of paths for a defined traffic type to take, while Performance SLAs define the criteria required to mark each path as valid or not.

We also show how traffic can flow through the Secondary DC/HUB should both provider circuits become unavailable at the Primary DC/HUB. While there are many ways to ensure resiliency between DCs, we chose to implement a very common DCI Circuit scenario. You should always consult with your Fortinet SE to develop the network design to best fit your environment. Once all routing converged though our SD-WAN environment we could see that traffic was able to Ingress and Egress DC1 via the DCI while both provider circuits were down. The reverse is also true once the provider circuits came back up that traffic returned to the normal preferred paths.

Lastly, we wanted to show that FortiGate's SD-WAN solution is truly a "controllerless" solution by making FortiManager unavailable to the Remote FortiGate's. We show how SD-WAN rules can still be modified on the remote device itself without using FortiManager to direct traffic to a different path by changing the preferred path order in the rule. A major differentiator from other SD-WAN vendors, Fortinet Secure SD-WAN offers a controllerless-based architecture where each FortiGate device maintains control-plane autonomy at the branch edge. In other words, the solution does not require a centralized or cloud-based controller to provide control-plane operations for application steering. Instead, each FortiGate edge device operates independently to evaluate available path efficacy and choose the most appropriate path for applications to traverse the WAN, whether the selected link be an overlay interface (IPsec) or an underlay interface (MPLS, DIA).

The FortiManager acts as a single pane of glass to simplify operations. The following image demonstrates how each FortiGate edge device communicates with centralized components but maintains all control-plane functionality at the edge. Transport-agnostic link support, SD-WAN core capabilities and services, and NGFW services are all delivered throughout the enterprise without dependency for control-plane input from an external device.

SD-WAN Automation

In this test case we take a simplified look at making API calls to FortiManager using PostMan. While PostMan is not an enterprise solution for automation such as Ansible Tower, the API calls it makes still demonstrate the abilities of making API calls to FortiManager. To start we show how to create an API User Profile as well as the API User Account itself. Afterwards we show the very important API call to login to FortiManager and obtain a session key that will be used for the subsequent API calls that will take place before the API User logs out or is timed-out by the system.

Once the API user is logged in, we look at interacting with the Devices and Groups area in FortiManager. We perform a few simple API calls list devices and details about a single device. You will see that API calls tend to use an HTTPS POST base URL of "https://<host_ip>/jsonrpc" to access the system. Each API call then has the "method" of the call such as GET, ADD, DELETE, UPDATE, etc. along with the extended "URL" of the information it is trying to access.

Once we verified we could access information within the Devices and Groups area, we wanted to further verify we could make changes in other areas of FortiManager such as FortiSwitch. In the FortiSwitch area, we wanted to verify that not only could we list items already present but add new items as well. In our case we wanted to create a new VLAN policy after listing the current VLAN policies to show that the item we were adding did not already exist. Once we created a new VLAN Policy we re-ran the List VLAN Policy API call to prove that the new policy was indeed added.

After we created a new VLAN Policy we added yet another one and then performed a Delete VLAN Policy API call to insure we could not only create new items but delete them as well using APIs. Finally, we review the logs and highlight that you can see the API calls made. While we only made a few API calls out of the hundreds available we successfully verified FortiManager's robust APIs. For reference, you can find the API references for Fortinet devices at the following URL with appropriate access:

https://fndn.fortinet.net/index.php?/fortiapi/1-fortios/

Alternatively, you can download the document at this link:

https://www.fortinet.com/content/dam/fortinet/assets/data-sheets/Fortinet_Developer_Network.pdf

Once you have followed the link above you can get to the other device APIs by exposing the "FortAPI" dropdown as shown below. This area will give you example code for every API call you can make. Be sure to select your desired code version as API calls have changed over time and as new features become available.

SD-WAN Monitoring and Reporting

The last test we performed was to verify that FortiManager's and FortiAnalyzer's built-in SD-WAN Monitoring Views and Reports were providing valuable information as expected. We start by looking at FortiManagers built-in views that show SD-WAN Sites and Links. FortiManager Views are short term views that can show the health of SD-WAN Tunnels including latency, jitter and packet loss, the amount traffic that has traversed those tunnels, traffic growth, etc. We are also providing a map and table views of each SD-WAN Site and associated SD-WAN Interfaces. Most items can be drilled down into to provide more information about those items or perform actions such as bringing a tunnel up or down.

Once we covered the monitoring views available within FortiManager, we moved on to the views and reports provided by integrating FortiAnalyzer into the environment. FortiAnalyzer provided views and reports that have longer retention times and detail than those available on FortiManager itself. In this case we looked specifically at the FortiView and Reports options available for SD-WAN in FortiAnalyzer.

FortiView has a couple of nice SD-WAN Monitors (Secure SD-WAN Monitor & SD-WAN Summary) that we review. These two views provide extensive details around SD-WAN Interface information, SD-WAN Bandwidth and SD-WAN Performance, SD-WAN Rule Utilization, SD-WAN Utilization by Application and MOS Scores just to point out a few. These views are instrumental in verifying your traffic is utilizing the environment as you intended and you are not over/under utilizing paths, sending traffic over unreliable paths, etc.

Another area of interest in the FortiView area is the VPN Monitoring capabilities. Again, we can get detailed information about the SD-WAN IPSEC Tunnels themselves and how they are performing over time. Again, like a lot of informational panes, shown below we can drill-down (red arrow) into an item for more detail.

Finally, we take a look at the Reports section of FortiAnalyzer and a couple of the built-in SD-WAN Reports that are available. We first must note that these reports are located in the Network Reports folder. We show that when you run reports the resulting report becomes available in the "Generated Reports" menu item after they run. We demonstrate how to configure reports to run on a schedule as well as the ability to email the reports after they run.

FortiAnalyzer is a robust monitoring platform and has many other reporting capabilities outside of just SD-WAN but we wanted to focus on those items related to SD-WAN since this testing effort was around the SD-WAN capabilities of the Fortinet solution.

Conclusion

Secure SD-Branch follows the Fortinet security-driven networking philosophy, which integrates networking services into the security infrastructure through FortiOS, the FortiGate operating system. Key benefits include a common management platform and integrated security, enabling Ethernet switch, WLAN interfaces, and LTE/5G backhaul to be controlled with the same level of enforcement as firewall interfaces. FortiGate uses "FortiLink" technology to communicate with switch and wireless platforms and requires no additional licenses. SD-Branch capabilities are included as part of the FortiOS running on every FortiGate.

Fortinet SD-Branch enables customers to converge their security, WAN, and LAN, extending the benefits of the Fortinet Security Fabric to their distributed branches. Secure SD-Branch is comprised of FortiGate Secure SD-WAN, FortiSwitch, FortiAP, and FortiExtender to deliver consolidation of branch services. To extend protection to the device edge, built-in Zero Trust features in their FortiOS operating system allows for secure onboarding of devices. 

Fortinet's Secure Networking strategy integrates an organization's network infrastructure and security architecture. This approach provides consistent security enforcement across flexible perimeters by combining a next-generation firewall with advanced SD-WAN networking capabilities all within the FortiOS/Fortigate that can be deployed as physical devices and virtualized into private and public cloud deployments, all with the same FortiOS and SD-WAN capabilities. This provides both security and resiliency for business-critical and cloud-based applications survivability and optimal path selection.

While the FortiGates can be controllerless, FortiManagers addition to the Fortinet SD-WAN environment provides templatized procedures, device configurations for FortiGates as well as FortiSwitches and FortiAPs and central policy repository to build your environment. This provides the ease of ensuring that devices throughout the environment are configured correctly and maintain compliance against the policies defined within FortiManager.

FortiManager offers all the necessary tools to manage and orchestrate Fortinet Secure SDBranch and SD-WAN solutions. You can quickly deploy thousands of edge locations, trigger changes to entire groups of devices, and consistently define security, SD-WAN policies, Fortiswitch and FortiAP configurations throughout your environment. FortiManager reduces administration and workload costs with smart features, such as device discovery, device group creation by administration domain, audit and management of complex SD-WAN architecture.

The FortiAnalyzer offers rich and detailed monitoring and reporting of the environment  with the ability to define and meet historical reporting requirements. FortiAnalyzer allows administrators or business owners to generate automatic security events and SD-WAN reports targeted to executive management. These reports provide immediate information to assess for any given timeframe, which is a benefit of the SD-WAN solution, while at the same time aggregating critical security information. While the highlights are listed in a convenient executive summary report, each section provides a more detailed view. This includes a set of recommended actions at the end of the report, plus actionable steps an organization may take to optimize their network for DIA, protect their organization from external/branch office threats, and ultimately reduce expenditures and save money.