The SolarWinds Cyber-Attack: What SLTTs Need to Know
On 13 December 2020, FireEye announced the discovery of a highly sophisticated cyber intrusion that leveraged a commercial software application made by SolarWinds. It was determined that the advanced persistent threat (APT) actors infiltrated the supply chain of SolarWinds, inserting a backdoor into the product. As customers downloaded the Trojan Horse installation packages from SolarWinds, attackers were able to access the systems running the SolarWinds product(s).
This cyber-attack is exceptionally complex and continues to evolve. The attackers randomized parts of their actions making traditional identification steps such as scanning for known indicators of compromise (IOC) of limited value. Affected organizations should prepare for a complex and difficult remediation from this attack. We have detailed a tiered set of guidance that organizations can take based on their specific capabilities and cybersecurity maturity. We’ve also provided available IOCs below.
Recent evidence shows that not all organizations with the malicious SolarWinds software were compromised by the threat actor, and that there were different stages of the attack. New information also reveals that some organizations without any SolarWinds products in their environment have been compromised with the same tactics, techniques, and procedures (TTPs) as the SolarWinds attack. This indicates that the attackers may have leveraged similar supply chain attacks against other products.
Who, What, When, Where
- Who: SLTT organizations with SolarWinds Orion Platform versions 2019.4 HF5, 2020.2 with no hotfix installed, and 2020.2 HF 1 within their environment. Note: there is evidence of organizations being compromised by this same cyber threat actor without SolarWinds products present in the network. Additional vectors are suspected and further investigation is ongoing by CISA and the FBI.
- What: A cybersecurity intrusion campaign affecting public and private organizations carried out by sophisticated APT actors. The United States government has determined that this attack poses a “grave risk to the Federal Government and state, local, tribal, and territorial governments as well as critical infrastructure entities and other private organizations.”
- When: Cybersecurity company FireEye discovered the supply chain attack against the SolarWinds products while investigating a compromise of their own network and publically announced the discovery of the SUNBURST backdoor on 13 December 2020. Confirmed compromises have occurred dating back to March of 2020. Forensic evidence has revealed files associated with this attack being compiled as far back as December of 2019.
- Where: Multiple industry verticals and government agencies across the globe. According to a recent SEC filing by SolarWinds, approximately 18,000 of their 300,000 customers were running vulnerable versions of the SolarWinds Orion platform.
The MS- and EI-ISAC understand that many SLTT organizations do not have full-time IT or cybersecurity staff, nor do they possess network monitoring tools or logging capabilities. As a result, we have provided tiered recommendations below that combine CIS guidance with that of the Federal Government; organizations can apply what is most applicable to their situation and level of expertise. For those SLTT organizations that outsource cybersecurity functions to a Managed Security Services Provider (MSSP), these recommendations can be used to coordinate a response with the MSSP.
For Organizations with Limited or No Cybersecurity Expertise
- If the organization has the versions of SolarWinds Orion Platform identified as vulnerable, isolate these systems by doing one of the following:
- Unplugging any network connectivity (e.g., Ethernet cable or Wi-Fi) from the system(s) running the SolarWinds application
- Isolating any network traffic to/from the SolarWinds system via a network device (e.g., firewall or switch)
- Completely power off the system running the SolarWinds software.
- If the organization is already a member of the MS- and EI-ISAC, contact our SOC at 1-866-787-4722, or email@example.com for further assistance
- If the organization is not a member of the MS- or EI-ISAC, but fit the criteria, they can still request and get assistance from the SOC in most circumstances.
For Organizations with Monitoring Tools and Some Cybersecurity Expertise
CISA has created three categories for organizations to use in order to determine the appropriate response and mitigation/remediation. CIS is using CISA’s methodology for consistency:
- Category 1: Organizations with SolarWinds products, but not any product listed as containing the malicious code
- Category 2: Organizations that have identified the malicious SolarWinds code in their environment, with or without internet traffic seen to the domain avsvmcloud[.]com
- Category 3: Organizations that have the malicious SolarWinds code and have confirmed that network traffic has been seen from the organization to the malicious domain of avsvmcloud[.]com and additional command and control (C2) traffic to a separate domain or IP address
Category 1 - Immediate Actions
- Follow the instructions by SolarWinds and download the latest release from their portal. Apply the latest release in the environment.
- Apply other security patches to servers running the SolarWinds application.
- Utilize CIS-CAT Pro (free to all MS- and EI-ISAC members) to scan SolarWinds system(s) and apply hardening recommendations from the CIS Benchmarks. For additional recommendations on hardening the SolarWinds Orion Platform, go here.
- Continue to monitor the environment for any malicious IOCs or other suspicious activities. Contact the MS- and EI-ISAC SOC with further questions if necessary.
Category 2 - Immediate Actions
- Examine network traffic looking for any beaconing activity to the domain avsvmcloud[.]com.
- If no traffic is seen to that domain since March 2020, follow all of the instructions listed above for Category 1 Immediate Actions.
- If traffic has been seen to avsvmcloud[.]com, look for additional unexplained network external communications from the SolarWinds systems. If no additional unexplained network traffic is located except for the beaconing to avsvmcloud[.]com, follow the steps listed above for Category 1 Immediate Actions.
- Conduct an audit of all systems looking for default credentials and new accounts created; perform an organizational-wide password/credential reset.
- If additional unexplained external network traffic is found from SolarWinds systems, go to Category 3.
Category 3 - Immediate Actions
- If external communications from the organization to avsvmcloud[.]com appear to suddenly cease on 14 December 2020 and the communication was not stopped by any action from cyber defenders, assume the environment is compromised.
- If the organization has in-house digital forensic expertise or has brought in external resources, proceed with the following steps. If the organization is an SLTT and does not have the necessary expertise, contact the MS- and EI-ISAC for assistance at 1-866-787-4722, or firstname.lastname@example.org.
- For those with expertise, do the following:
- Forensically acquire system memory and host operating systems of any system hosting all infected versions of SolarWinds Orion
- Analyze network traffic for additional IOCs
- Examine SolarWinds host systems for anomalous behavior, including new user or service accounts, new processes running, or other signs of persistence
- Upon completing the forensic acquisition and network analysis of impacted SolarWinds hosts, immediately disconnect or power down all affected versions of SolarWinds Orion from the environment
- Block all traffic at the perimeter firewall to and from all hosts outside of the environment where any version of SolarWinds Orion software has been installed (e.g., cloud instances)
- Identify and remove all threat actor created accounts and other mechanisms of persistence
- SLTT leadership may use CISA Alert AA20-245A, “Technical Approaches to Uncovering and Remediating Malicious Activity” as a guide when reviewing work done by internal or external IT and cybersecurity staff.
- Once the immediate threat has been remediated, there are a variety of technical steps recommended by CISA for complete remediation. These steps include:
- Rebuilding systems
- Restoring network infrastructure managed by SolarWinds to known good versions of firmware
- Resetting all credentials across the enterprise (users, SNMP strings, SSH keys, certificates, etc.)
- Forcing multi-factor authentication
- Additional system and configuration hardening, which can be found on CISA’s Alert AA20-352A under the heading of Mitigations
Special Note: Due to the sophistication of the cyber threat actor and the length of time this attack has been ongoing, organizations should assume that backups and virtual snapshots may also be compromised. Organizations must take special care to ensure the restoration of backups does not reintroduce the compromise to the environment. Backups should be thoroughly examined by digital forensic experts before any restoration event is completed.
Actions Taken by the MS- and EI-ISAC
This incident is fluid and the MS- and EI-ISAC are working continuously to protect our SLTT members. Upon discovery of this attack, the MS- and EI-ISAC Security Operations Center (SOC), Threat Intelligence Team, Computer Emergency Response Team (CERT), and leadership assembled a cross-functional team working around the clock and collaborating with our public and private partners to assist the SLTT community. Specific action items include:
- For SLTT members with the Albert network monitoring service, analysts are reviewing traffic for IOCs and maintaining communication with members about findings.
- For SLTT members with the Managed Domain Blocking & Reporting (MDBR) service, the MS- and EI-ISAC is reviewing logs for any evidence of communication with malicious domains associated with this attack. Note: MDBR is offered free to any SLTT organization.
- Known IOCs for this attack have been added to MS- and EI-ISAC monitoring and control platforms to alert and take immediate action as necessary.
- Assisting SLTT organizations with questions, incident response, and forensic analysis.
- Providing curated IOCs via our Threat Intelligence Platform.
Future Actions for SLTTs
This sophisticated cyber-attack is yet another example of why organizations, regardless of size, must implement cyber hygiene best practices. As SLTT leadership looks beyond the immediate tasks of the SolarWinds response, CIS has a number of longer term operational and strategic recommendations.
- Ensure cybersecurity is a conversation occurring at the highest levels of executive leadership. Cybersecurity is not an IT problem, it is an enterprise-wide risk management topic that requires attention.
- If not already taking advantage of all of the free services from the MS-ISAC and EI-ISAC, ensure those are being fully utilized. Examples include the MDBR service, threat intelligence sharing, CERT and SOC resources, and more.
- Monitor for high-risk events such as account creations, privilege escalation, new services created, security-related services disabled, changes to security posture, unusual network communications, etc.
- Deploy endpoint protection tools to all hosts and mobile devices. Depending on experience level and budget, consider solutions such as Endpoint Detection and Response (EDR), or a more inclusive Endpoint Protection Platform (EPP). If in-house resources don’t allow this, consider outsourcing to CIS or another MSSP for monitoring and administration.
- Become familiar with the CIS Controls and implement them according to the level of risk for the organization. Organizations must have a minimum level of cybersecurity to help mitigate threats like this in the future. This includes:
- Asset inventories
- Patch and vulnerability management
- Multifactor authentication
- Adoption of least-privileged accounts
- System hardening
- Implement the CIS-CAT Pro software, free to all SLTT MS- and EI-ISAC members. CIS-CAT Pro allows organizations to scan systems running a variety of operating systems and measure their compliance with the CIS Benchmarks, industry standard best practice hardening guides. By hardening systems to begin with and eliminating unnecessary risk, organizations make it more difficult for an adversary to gain access.
- Implement a risk-based vulnerability management program that includes patching timelines, accounting for the criticality of assets and of the vulnerabilities.
- Consider additional paid services from CIS that are tailored for the SLTT community, including our managed security services (MSS) and Albert network monitoring services.
- Sign up for the Cyber Hygiene (CyHy) monthly vulnerability scans conducted by DHS for another view of risk from an outsider’s perspective. Be sure to select the option to give the MS- and EI-ISAC access to the scan results so we can monitor for exploitation and understand the threat landscape.
- Ensure all staff have annual cybersecurity awareness training and that policies exist to provide administrative controls over areas that cannot be controlled with a technical solution.
- Implement monitoring and logging capabilities for endpoints and network infrastructure.
- Update (or create if none exists) the Incident Response (IR) protocol for the organization, and include organizations outside of IT such as public information, human resources, legal, executive leadership, and functional organizations. Be sure to include critical vendors and requirements for data and service restorations along with many other considerations. Practice the plan before it is needed through the use of tabletop exercises.
- Utilize CIS or another third party to perform internal vulnerability assessments and penetration testing to provide IT and leadership an unbiased snapshot of the current risks and condition of the organization’s cybersecurity posture.
Many IOCs have been made public. It is important to note that subdomains created by a domain generation algorithm (DGA) are likely unique to each victim organization and are not likely to appear in another victim’s environment.
The following resources are currently hosting publicly-available IOCs:
- FireEye Initial Analysis
- FireEye GitHub including Yara rules, Snort rules, hashes, and other IOCs
- Volexity Analysis of Sunburst Kill Chain
- Cisco Talos Analysis of Supply Chain Compromise with IOCs
- McAfee Analysis of SunBurst
- SolarWinds Security Advisory
- CloudFlare Analysis with Trend Data
- Prevasio Analysis of Sunburst Backdoor
- Huntress Security SunBurst DLL Locations (list)
- Paste of SunBurst Hashes
- Paste of SunBurst DGA Subdomains
- List of Unique DGA SunBurst Subdomains
- TrustedSec Summary and Recommendations
- TrustedSec Incident Response Playbook for Sunburst
For MS- and EI-ISAC members that have the ability to ingest threat intelligence via STIX/TAXII, contact us at email@example.com for information on how to get access to our feeds.
- CIS MS-ISAC Advisory 2020-166, “Multiple Vulnerabilities in SolarWinds Orion Could Allow for Arbitrary Code Execution”
- CISA Alert (AA20-352A), “Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations”
- FireEye Threat Research, “Highly Evasive Attacker Leverages SolarWinds Supply Chain to Compromise Multiple Global Victims With SUNBURST Backdoor”
- Microsoft Security Response Center, “Customer Guidance on Recent Nation-State Cyber Attacks”
- SolarWinds Documentation for Secure Configuration of the Orion Platform
- FireEye GitHub including Yara rules, Snort rules, hashes, and other IOCs