A Discussion on SolarWinds and Securing the Supply Chain

Cybersecurity is in the news every day and top of mind for many IT executives (as it should be). But if there is one cybersecurity topic that has recently dominated headlines and the mental space of IT decision makers, it is the SolarWinds incident. In many regards, the SolarWinds incident was an anomaly in state, local and education (SLED) because SLED organizations do not typically have to defend against nation states. And although an anomalous incident for SLED, it should still be used for lessons learned and as an opportunity to further educate on the importance of a comprehensive cybersecurity program.

I will first explain a few salient points about the SolarWinds incident, including a bit about why it was so difficult for organizations to detect. I will then go into what SLED organizations should have already done and what the next steps, both short-term and long-term, should be.  

While we will be discussing supply chain risk in this article, the SolarWinds incident has also made application development and code security top of mind because of how the threat actors were able to insert malicious code into the legitimate code pipeline. Because of this, my next article will be on securing your code development pipeline, so be sure to stay on the lookout for this very important and relevant topic.

Why wasn't this detected?

One of the biggest questions still largely unanswered on the SolarWinds incident is: why wasn't this caught earlier? In reality, it probably was; however, it was never identified as the large scale, advanced, persistent threat (APT) it inevitably turned out to be.

Since the attack went public, opportunities for detection have been published which describe how activity associated with the SolarWinds incident could have been realized as malicious, which has caused some organizations to question why their security teams did not alert on this attack. And while it may be easy to recognize these opportunities for detention after the fact, very few organizations were prepared for the level of sophistication employed in this attack, especially in the SLED space.

This was an anomalous event for SLED. SLED organizations do not typically have worry about the threat of an APT from a nation state level actor. Nation states are not burning their zero-day exploits on SLED organizations or going through the effort of a lengthy reconnaissance to steal information from a state system. SLED organizations should not lose sight of this and should not react by altering strategic plans too drastically. Make sure you are doing the basics well and look for opportunities to continuously improve the prioritization of your cyber activities using a risk-based approach. In the SLED reality of competing budget priorities, don't try to fight the bear if you can't keep unencrypted PII out of your DMZ. Instead, continue to focus on those things that are the highest risk based on likelihood and impact.

What should your organization have already done?

Let's briefly discuss what organizations should have already done: follow the guidance from the experts. Both FireEye and the Department of Homeland Security (DHS) Cyber and Infrastructure Security Agency (CISA) have issued comprehensive guidance on the methods of attack and remediation actions. Follow that guidance, even if it means painfully re-building systems and even if activity seems to have ceased months ago. If you fail to follow this guidance there is a good chance your systems will remain compromised. And err on the side of caution — assume your systems have been affected by this incident until/unless you can conclusively prove otherwise.  

What should SLED organizations do now?

On the topic of this being an anomalous event for SLED, it is worth owning up to the fact that this event made me a liar. I have long touted that SLED organizations do not need to be concerned with nation state level attacks and should really be focused on cyber hygiene and making sure they are doing the basics really well. Specifically, to keep the threat actors from executing an attack they need to secure their remote access protocols, patch diligently and prevent and mitigate user induced risk. Backups should also be performed and, more importantly, tested to make sure they are comprehensive and complete.

I can say from experience that many would be attackers of SLED systems will stop after a scan finds no exposed network protocols. If they are more persistent and they do manage to gain entry to your environment, up to date patching will help to prevent movement and make it more difficult for them to launch an attack. At this point, you will have thwarted the majority of would be attacks on SLED systems. And if you keep your users from doing things like clicking that link (either through controls or training) or setting their passwords to 12345, you will have a reasonably secure SLED organization. And due to the anomalous nature of the SolarWinds event, I still stand by this guidance.

Similar to my guidance on not altering strategic plans too drastically, organizations should not have a knee jerk reaction and decide to move away from SolarWinds without significant consideration. At least the SolarWinds incident has been discovered and a path for remediation seems to have been identified, but what about other tools in your network, or the tool you may choose to replace SolarWinds? There is more than a small probability that hackers sophisticated enough to pull off this type of incident can infiltrate other companies' supply chains as well. "The devil you know" can sometimes be the better choice.

Further, a knee-jerk reaction to move from SolarWinds due to the incident is probably not the best strategy because there are impacts to switching your network monitoring solution and a strategy needs to be developed to make sure that there are not capabilities being lost or visibility into parts of the network do not suffer during a transition, which could potentially allow for other attacks to be successful. Network monitoring tools such as SolarWinds can be another source of data to alert SOC teams of potentially malicious behavior via indicators such as anomalous or worrisome network traffic. And it becomes even more problematic if an organization is using multiple SolarWinds offerings such as the SolarWinds Security Event Manager (SEM) tool. Because of this, there is the potential for very real impacts to your organization if visibility is degraded or comprehensive capabilities/controls are not considered during a replacement strategy.

What if my organization has already decided to move away from SolarWinds?

If there is a desire to move away from SolarWinds, make sure you have either qualified internal resources or a partner to assist with things like identification of pros and cons of products specific to your organization's needs. WWT has unique capabilities to work with customers in this area by leveraging our Advanced Technology Center (ATC) where we can test capabilities of products in an environment mimicking your own. We can make sure you have all the information on how easily (or difficult) a product integrates with your other tools, pitfalls to watch out for and provide a capabilities comparison as it relates to the specific things your organization cares about.

If you have already decided to transition off SolarWinds, here are a few recommendations before making the leap:

• Determine if a tools rationalization workshop is required to consolidate tools, lower risk and improve cost effectiveness as you mature forward.
• Identify key areas of functionality that are required specific to your organization and prioritize them before you start to consider alternatives.
• Focus on major cost-based variables like ease of onboarding, how cohesive and integrated the software is and ease of use by various stakeholders.
• Consider other elements of selection strategy related to areas of strength or capabilities, like analytics and reporting, troubleshooting and diagnostics, scalability and agility, support agreement terms and cost.

What should SLED organizations be looking to do in the future?

Now that we appear to be largely on the other side of the SolarWinds incident, there are things that SLED organizations can be doing moving forward to help prevent similar events in the future. I have already touched upon a couple of these items such as greater communication and correlation and looking for lessons learned, but these activities alone will likely be insufficient. Organizations need a comprehensive supply chain risk management (SCRM) program with impact levels identified holistically. And organizational and legislative initiatives would also contribute to reducing the number of similar incidents we experience in the future. Here are four specific things SLED organizations should be exploring:

1. Evaluate Lessons Learned. Organizations can use the indicators of compromise (IoCs) associated with the SolarWinds incident as guidance to tune their systems and implement products to detect and perhaps respond to any similar future events. For instance, having systems that detect things like lateral movement and privilege escalation, SSL certificate generation activities and multiple simultaneous logons from people and systems will serve to protect SLED organizations from some of the more advanced threats that at some point in the future may again be leveraged against them.
2. Legislation. State and local entities should also explore legislation which could aid in setting a definitive set of standards to which technology providers should adhere. This legislation could provide for incentives for those technology providers that meet the standards, which would be largely based on transparency of security related practices. This type of endeavor will naturally receive resistance from technology providers, potentially citing that greater transparency of their security practices will put them at greater risk of an incident. This premise is, however, unfounded as general knowledge that a technology provider is adhering to security best practices would only serve to deter would be attackers.
3. Centralize IT Procurement. Centralizing procurement of IT could help lower the risk of similar incidents in the future. Centralizing IT procurements would mean fewer products in the environment, thereby effectively reducing the attack surface of the organization. Also, with centralized procurements there would be greater opportunities to do things like include security testing in product evaluations and it is much easier to account for a smaller number of products when building or maintaining a comprehensive SCRM program.
4. Develop a Comprehensive SCRM Program. On the topic of comprehensive SCRM programs, many government agencies simply do not have one. I am not picking on SLED here, far from it. In fact, the National Institute of Standards and Technology (NIST) issued information and communications technology (ICT) SCRM specific guidance in 2015, and OMB has required agencies to implement ICT SCRM since 2016, but GAO issued a report in late 2020 that cited that only 6 of the 23 agencies they audited had implemented any of the 7 basic building blocks of an ICT SCRM program and no agency audited had implemented all 7 components.

A comprehensive SCRM should quantify the level of risk posed by an organization's supply chain. Supply chain risk is often calculated similar to other risks, as a factor of probability of an event multiplied by the impact if an event does in fact occur. Many organizations are able to quantify the probability of an event, but even those organizations with decent SCRM programs often fail to fully account for the impact that would be realized if an event were to occur. 

This is because the impact can be more far reaching and have ripple effects that are often difficult for organizations to account for. The good news is that there are resources that can assist with building out a comprehensive SCRM strategy and program. The CISA ICT SCRM guidance is a great starting place for developing an SCRM program, and if you are one of the many organizations struggling with identifying all the interconnections, dependencies and ripple effects an incident could impact, the NIST Internal Report 8272 is a great place to look for guidance. 

And, of course, a trusted partner like WWT is always advisable whether you are just setting off on the journey of SCRM or looking for recommendations on how to continue to grow and strengthen your SCRM program and practices.