Thoughts on Breaking Glass (in Case of Emergency) for Backup Systems

Thoughts on breaking glass (in case of emergency) for backup systems

Introduction

World Wide Technology is an industry leader in deploying Cyber Recovery and Cyber Resiliency services. We've successfully built and deployed Isolated Recovery Environments (IRE) for some of the largest companies in the world.

Inside the IRE are all the required components to restore a subset of an organization's most valuable computing resources. The key components, and thus the first to get restored in the event of a cyber-attack, are what is commonly known as Critical Rebuild Materials. Critical Rebuild Materials include network configurations, storage configurations, intellectual property, rebuild tools, documentation and most importantly and Identity Access Management (IAM) tools such as Active Directory or LDAP.

The IRE needs to be isolated from production to ensure it remains pristine, free of malware. This enables a quick start up of Active Directory, and recovery operations continue in a timely manner.

The problem

How does an organization begin to restore from backups if they don't have an Isolated Recovery Environment and IAM tools become compromised or disabled? Assuming the backups are immutable, how do the backup administrators log into the backup servers to initiate restores?

The ability to recover critical data from backups is the last line of defense against disastrous business losses due to ransomware and other enterprise-scale destructive cyberattacks including insider threats.

To access backups when production networks and critical infrastructure are down, system administrators need an offline or isolated 'Break Glass' account to access the backup storage systems. 

Break Glass Account

A "Break Glass" account is a highly privileged, pre-staged account used in emergency situations when standard access methods fail. It allows authorized personnel to regain access and control. The concept is analogous to breaking a glass case to access a fire extinguisher in an emergency, hence the name.

Break Glass accounts should be designed for situations where regular authentication methods such as AD or LDAP are unavailable or compromised. Most importantly, they allow authorized users to access critical resources such as backup servers during outages or security breaches when regular authentication methods fail.

The process for accessing backup systems in emergency situations must be well-documented. The documentation that accompanies the account must detail how to access the backup console directly, the IP addresses of the servers and storage, how to rebuild a management server, etc. 

Privileged Access Management

Privileged Access Management (PAM) is a crucial aspect of cybersecurity that focuses on controlling and monitoring access to critical systems and sensitive information. In PAM, a Break Glass account provides emergency access when standard authentication methods fail, allowing authorized personnel to regain control of critical systems. 

Adhering to PAM principles requires the Break Glass account credentials to be stored with strong security measures such as a physical safe or a password manager accessible only by a limited number of trusted personnel. The password should be sufficiently complex that it cannot be easily memorized.

Other considerations

Two person rule

To prevent an insider attack, it may be beneficial to implement a two-person rule requiring multiple individuals to be involved in accessing Break Glass accounts.

When only one login credential is required, implementing a two-person rule is a way to implement a split password policy. Split password refers to a security practice where a highly privileged system account's password is physically split into multiple pieces and stored securely in different locations, ensuring that no single person can gain unauthorized access to the account. 

Logging and alerts

While it can be assumed that connectivity to SIEM (Security Information and Event Management) and SOAR (Security Orchestration, Automation and Response) systems have been compromised, logging should be performed and retained to allow for later diagnostics and accountability. Even with logging enabled, it may be impossible to determine who gained access. It will provide evidence of activities performed.

Multi-factor authentication

Multi-factor authentication (MFA) requires users to provide multiple attributes before being granted access to a system. This enhances security by combining something the user knows (like a password) with something the user has, such as a mobile device or token.

Some forms of MFA won't work when network access is compromised. MFA methods such as email or SMS that employ one-time codes may not be available in those cases. 

If you choose to use MFA for break glass login, Authenticator Apps, Hardware Tokens and PINs are some methods that work with backup servers that are isolated from the extended network.

Role-based access control (RBAC)

In a 'break glass' scenario for managing backup servers, role-based access controls (RBAC) need to strike a balance between strict security and emergency accessibility. 

One extreme would limit the Break Glass account to only being allowed to backup operator functions such as backups and restores. This would be the minimum access that would be effective in helping restore operations. On the other extreme, allowing administrator-level access may be required in emergencies. Regardless of which level is granted, under no circumstances should the role allow changes to existing backups, policies or retentions.

Periodic testing & review

After creating the break glass process, it's crucial to test it on a regular basis. At the very least, it's important to have trust that the account will work when needed. It's also valuable to verify the access and credentials, compliance, and security on a regular basis since security practices are constantly evolving in any complex environment.

Lastly, the procedures for break glass activation should be detailed enough for a technical administrator to understand and execute in a crisis situation. The documentation shouldn't depend on your backup administrator for execution. 

Conclusion

Restoring from backups is often the last line of defense against increasingly sophisticated attacks. However, if Identity and Access Management (IAM) systems like Active Directory or LDAP are compromised, organizations may be unable to initiate recovery, even with secure, immutable backups.

To address this, World Wide Technology (WWT) recommends the use of a 'break glass' account, a highly privileged, emergency-access credential designed for use when standard authentication methods fail. These accounts are a vital component of a resilient backup and recovery strategy, particularly in environments without an Isolated Recovery Environment (IRE). For more information, please visit WWT.com