The Time for OT Architects to Consider Using SD-WAN is Now

Today, most utilities have migrated their Wide Area Network (WAN) environments from SONET to some form of packet network, such as IP-MPLS, MPLS-TP, carrier ethernet, and now, segment routing. The two primary drivers for this transition were SONET obsolescence and the need to support a growing list of emerging grid applications that used native IP.  The utility WANs and the grid applications they support continue to evolve, so a growing densification of routing will occur.  More routers can mean more network complication. 

It has been said that it's less about the growth in traffic and more about the fact there are more directions for traffic to travel that challenges network operators. This begs an interesting question: "Is it time for OT architects to consider using Software Defined Wide Area Networking (SD-WAN)?" In doing so, they could leverage network controllers to perform the complex work of network control versus increasing their staffs to manually make ongoing changes.  

This post focuses on exploring the rationality of adopting SD-WAN technology — in particular, private SD-WAN in support of various grid operations traffic. Much like MPLS, the term SD-WAN can apply to public services offered by carriers or to the technology itself. But here, we seek the technical features SD-WAN offers that can be implemented by and totally under the control of an electric utility. 

Here, we cover SD-WAN technology and its market adoption, and point out challenges utilities face and the special considerations mission critical operators must make when using SD-WAN. For these purposes, SD-WAN as being complementary to existing transport networks, including private MPLS and private 4G/5G networks. And it discusses how SD-WAN implementation can support both greenfield as well as brownfield situations. 

SD-WAN is a mature technology. Market acceptance of Software Defined Networks (SDN), which includes both Software Defined Access (i.e., LANs) and WANs, experienced annual growth rates of approximately 25% in 2022 and is projected to continue at approximately 10.1% annually, according to market intelligence experts at IDC. SD-WAN can be made secure. The U.S. Department of Defense's (DoD)'s "Thunderdome Initiative," will utilize SD-WAN to support its cybersecure Defense Information System Network (DISN). 

Most service providers have been offering SD-WAN as an alternative to their public MPLS services, which unfortunately has caused some to perceive SD-WAN as just a public cloud-based service. Many utilities are implementing SD-WAN for their enterprise environment, whether a public or an on-premises, privately implemented solution. Now, some utilities are considering using private SD-WAN for their OT environment. What is it about SD-WAN technology that is so valuable that electric utilities should be considering its use in the OT environment? And are there special considerations utilities should be making regarding OT support that might be different than their Enterprise environments? Does SD-WAN supplant a utility's investment in MPLS or some other WAN technology? These and other OT-related questions are what we intend to address in this post.

SD-WAN, a multifaceted resolve

It can help to view SD-WAN as an overlay services technology. It functions best when there is an underlying connectivity technology that provides much of the routing, but SD-WAN can add a routing layer that benefits its users in a way the underlay wasn't necessarily designed. As such, SD-WAN is a complimentary technology to existing utility WAN environments, presuming they already incorporate routing capabilities. 

Note: MPLS-TP and carrier ethernet are underlay technologies that emphasize transport features and don't necessarily provide routing. Furthermore, some utilities whose IP-MPLS simply offers Ethernet 'pipes' may not be configured to offer the needed routing capabilities. In these scenarios, SD-WAN is still an option, but would require additional technical considerations.

When SD-WAN was developed its greatest strength was to separate the WAN's data plane (that which carries the grid application user traffic) from its control plane (the internal signaling that occurs to appropriately administer and direct user traffic to its destination). One of the advantages of data plane/control plane separation is that it allows implementation of virtual networks (which historically was a complex but very important task) over diverse underlying networks. Additionally, SD-WAN implemented smart controllers that took on the job of continually evaluating the network and automatically making changes to accommodate for any adverse anomalies and to optimize traffic flows. Automation reduces manual efforts and helps reduce various risks associated with an ever-changing network. As SD-WAN technology evolved, valuable security features, which could also be virtualized and pushed to the edge of the network were added. This is referred to as Secure Access Service Edge or SASE-based SD-WAN.  

As utility OT networks evolve, private versions of SD-WAN and SASE become increasingly strategically important. In the past, utilities built their own microwave and fiber networks and implemented some form of packet technology. Why would they need SD-WAN/SASE if they have private networks? First, most utility private networks don't reach 100% of their grid endpoints, which today are predominantly substation-based applications. Many substations require some form of public service for connectivity. Leased T1 and DS0 circuits provided adequate connectivity and security on 'non-routable circuits' was acceptable. But TDM discontinuance and SONET equipment discontinuance are forcing utilities to transition to various packet offerings, such as public ethernet, internet and public 4G wireless services. Routers and firewalls now dominate the OT telecommunication environment. Unless some form of automated network overlay provides the necessary traffic segmentation and security, utilities will need to continually modify their manually administered packet systems.  

Rising challenges, no match for SD-WAN

This (routing and security) problem will grow exponentially as utilities experience the explosion of grid control devices supporting grid modernization. However, the aspiration of 100% private networks is rapidly becoming an unattainable goal. Even if the utility could expand their private WANs and their emerging private wireless systems (such as Private LTE) everywhere, they still will be faced administering an increasingly complex network of networks. New traffic patterns, better reliability and more extensive security are outpacing grid operators' abilities to keep up. Increasingly, blended public-private solutions are being implemented. SD-WAN/SASE technologies which have been proven in the enterprise space can also be deployed in the OT space to create a unified overlay with security pushed all the way to the edge all remaining under the utility's control.  

Increased network complication, new traffic patterns and better reliability are very challenging issues to deal with. Where is relief likely to come from? It is unlikely that conventional networks will somehow produce simplification. And the need for a more modern grid with rich sensory data and control features will accelerate new traffic patterns. To achieve reliability (much less sustain it), the networking community is turning to automation, which incorporates artificial intelligence and machine learning (AI-ML) to monitor network health, automatically adjust application flows according to defined policies, issue intelligent alerts for faster remediation and supply projected forecasts to enable optimal system planning. 

AI-ML is the foundation of next generation software defined networks, such as SD-Access, SD-WAN and 5G/6G. For electric utilities, SD-WAN should be seen as a strategic technology that has moved much farther than simply taking advantage of third-party transport or for directing traffic across optional uplinks. Every device supporting grid operations can benefit from SD-WAN — even constrained devices which are common in OT environments.  

Private SD-WAN/SASE for mission critical OT requirements will require some architectural adjustments not typical in enterprise environments. Foremost will be to support the quantity of endpoints that the SD-WAN controller will administer. While enterprise environments might support hundreds or thousands of SD-WAN endpoints, electric utility OT environments must support hundreds of thousands of endpoints. Therefore, during the planning phase it is important to evaluate how a vendor's SD-WAN solutions might scale and perhaps how it can eventually become segregated.  

Another issue to consider is how private SD-WAN solutions might support legacy traffic and interfaces which are common in utility OT environments. Some legacy applications such as SCADA, which use RS-232 and RS-485 serial interfaces, can be adequately supported using adaptation methods such as raw socket. However extremely low-latency applications such as teleprotection are better served using other networking technologies.   

By no means should SD-WAN be precluded at sites requiring teleprotection. Most of these sites are connected using utility-operated fiber underlay systems, which can support proper emulation or adaptation services used for teleprotection. However, it is important to recognize that the majority of utility traffic is becoming IP-based and will utilize some form of REST-based protocol along with Transport Layer Security (TLS). Distributed Energy Resources (DER), EV support systems and even AMI2.0 are some examples. Therefore, overlay solutions, such as SD-WAN, should be duly considered for accommodating IP needs while the underlay directly support teleprotection.  

This leads us to another consideration that many utilities are facing: Should the underlay be uniform or can it be heterogeneous? The problem is that many utilities have diverse WAN transport infrastructure (underlay) consisting of SONET, MPLS, native IP and other protocols. Most of these sub-networks use different vendor technologies having diverse network management systems. One of the advantages of SD-WAN is that as an overlay technology, it can fully operate over heterogeneous underlay systems: even having diverse routing configurations. And with SD-WAN's advantages in traffic segmentation, it is arguable that the underlay can even become simplified by not needing to support so many Virtual Routing and Forwarding (VRF) sub-networks. Additionally, using SD-WAN's inherent performance optimization, application performance can even be improved by adding alternate and heterogeneous underlay sub-networks. Therefore, a private SD-WAN overlay creates flexibilities for utility telecom teams to refresh their diverse underlay networks according to actual business requirements such as capacity increases or equipment end-or-life replacement rather than for the sake of vendor or technology homogeneity.   

The discussion above shows that private SD-WAN should be regarded as a complementary technology rather than being competitive with SONET, MPLS-TP, IP/MPLS, Segment Routing and IP-only networks. Can SD-WAN co-exist with wireless networks such as 4G/5G (public and private)?  

It turns out that SD-WAN is complementary with wireless access for various reasons. From a reliability standpoint, SD-WAN's inherent micro-segmentation capabilities can help utilities simplify the virtualization and segmentation of diverse applications, where for security reasons traffic should not be inter-mixed. Further, endpoints having multiple radios can benefit by SD-WAN's ability to dynamically direct traffic over the best performing uplink. Diverse routing of wireless endpoints was rarely considered historically due to the applications' relaxed performance requirements. However, in the situation where distributed energy resources will power the grid, uplink reliability, supported by diverse routing, will become very critical. Although conventional routing supports this feature without SD-WAN, it is very complex to operate this at scale, which is exactly the emerging situation utilities are beginning to face. Using policy-based routing managed in central controllers, SD-WAN can also help contain costs when one or both uplinks use public 4G/5G services. Therefore, private SD-WAN endpoints can dynamically improve reliability of utility's mission critical traffic while optimizing its associated telecommunication costs.  

When private SD-WAN is combined with SASE capabilities, the advantages are even greater. SASE allows security functions to be pushed to the network edge. This can be very important for 4G/5G systems where network access is combined with network backhaul by virtue of its use of Generic Tunnel Protocol (GTP) tunnels. These tunnels exist between the radios in the field and the often centralized gateways (PGW and UPF) where traffic emerges from the 4G/5G networks. Since these networks can span the utility's entire service territory, measures to enhance security at the edge helps prevent improper use of network resources and in turn helps optimize the performance of all legitimate traffic. Edge security becomes even more valuable for REST/TLS-based traffic which as stated above will become the utility's' dominant form of traffic.  

But not every utility grid endpoint will be capable of supporting SD-WAN. Indeed, this is true for most of the IOT industry where endpoint resources need to conserve power, bandwidth, and costs. One SD-WAN vendor has addressed this challenge by interworking the SD-WAN control plane with that of the 4G/5G system. As a result, the 4G/5G system becomes the first hop access component that automatically directs diverse traffic from every attached User Equipment (UE) to its appropriate micro-segmented virtual network. The importance to electric utilities is that this solution furthers their conformance with NIST's Zero Trust Architecture (ZTA) as outlined in Special Publication 800-207 (SP 800-207). Rather than protecting some network location (which too often is performed centrally at the detriment of IOT devices), ZTA enabled private SD-WAN/SASE and 4G/5G seeks to protect the individual asset regardless of its capabilities.  

Putting ideas into practice, practice into play

Lastly, how should utilities think about private SD-WAN roll-out? One of the most significant challenges utilities have is the time it takes to evolve their infrastructures. Implementations and refreshes can take years to complete and outage windows can be seasonal to obtain. Greenfield environments don't necessarily bring such challenges; however, brownfield environments are expressly challenging. While SD-WAN brings administrative simplicity, improves reliability, performance and security, and shifts complexity from people to machines, there is no explicit reason SD-WAN greenfield or brownfield environments cannot co-exist with legacy networking systems. In fact, in brownfield environments, baselining is a common practice where the initial SD-WAN is deployed in a generic configuration; then after detailed as-is information has been collected, devices are moved into more optimized to-be SD-WAN subnetworks.

It is true that a programmatic implementation of private SD-WAN would expedite uniformity, but one of the values of this technology is that it's rollout can be related to individual business needs versus the needs of a telecom department. For example, a video security implementation might trigger the implementation of an SD-WAN appliance installation at many substations and a Phy-Sec VRF be implemented at that site. 

Using ethernet connections into the local switching infrastructure, provisions can be made to simply add camera endpoints with their strategic/long-term IP addressing scheme, security controls and any other networking element at the time of dispatch. Other legacy devices can be re-attached to a temporary 'legacy-like' VRF overlay so as to defer time-consuming efforts. Then in the future on an as-needed basis, devices in the 'legacy-like' VRF can be moved to more proper and strategic VRFs for better segmentation. IP readdressing, proper QoS and security features can be implemented on a case-by-case basis until such time that all remaining endpoints become migrated to the improved private SD-WAN environment. This is much harder to accomplish in classic legacy environments where the control and data planes are not separated. Obviously, it is more desirable to transition all services at the same time, but this is not a luxury that is always available.  

Flexibility in deployment can be very valuable to electric utilities. However, many design details must be made to properly accommodate mission critical traffic. WWT has deep experience in SD-WAN strategy, design and deployments. WWT's Advanced Technology Center (ATC) offers capabilities for utility customers to have hands-on experience to better understand various SD-WAN vendor solutions. The SD-WAN lab environment will be integrated with WWT's new Utility Field Network lab, currently under construction. The Utility Field Network lab will offer customers that ability to evaluate how various solutions can be applied to support utility functions such as event data collection, SCADA, distribution automation, open phase detection and other grid applications. The new facility should be available in 1Q2024.