The SolarWinds Cyber-Attack: What You Need to Know

Arrow  Last Updated: January 7, 2021

Executive Overview

On December 13, 2020, FireEye announced the discovery of a highly sophisticated cyber intrusion that leveraged a commercial software application made by SolarWinds. It was determined that the advanced persistent threat (APT) actors infiltrated the supply chain of SolarWinds, inserting a backdoor into the product. As customers downloaded the Trojan Horse installation packages from SolarWinds, attackers were able to access the systems running the SolarWinds product(s).

This cyber-attack is exceptionally complex and continues to evolve. The attackers randomized parts of their actions making traditional identification steps such as scanning for known indicators of compromise (IOC) of limited value. Affected organizations should prepare for a complex and difficult remediation from this attack.

We have provided available IOCs as well as detailed a tiered set of guidance that organizations can take based on their specific capabilities and cybersecurity maturity. There is also a dedicated section with specific actions and support for MS-ISAC members and SLTT governments.

Who, What, When, Where

Systems Affected

  • SolarWinds Orion Platform Version 2019.4 HF 5
  • SolarWinds Orion Platform Version 2020.2
  • SolarWinds Orion Platform Version 2020.2 HF 1

For CVE-2020-10148, SolarWinds Orion Platform versions 2019.2 HF 3, 2018.4 HF 3, and 2018.2 HF 6 are also affected. Security patches have been released for each of these versions specifically to address this new vulnerability.

Updated Technical Summary

SolarWinds Orion is prone to one vulnerability that could allow for authentication bypass. API authentication can be bypassed by including specific parameters in the Request.PathInfo portion of a URI request, which could allow an attacker to execute unauthenticated API commands.  In particular, if an attacker appends a PathInfo parameter of WebResource.adx, ScriptResource.adx, i18n.ashx, or Skipi18n to a request to a SolarWinds Orion server, SolarWinds may set the SkipAuthorization flag, which may allow the API request to be processed without requiring authentication, potentially resulting in a compromise of the SolarWinds instance.

  • Who: Organizations in private industry and U.S. SLTTs with SolarWinds Orion Platform versions 2019.4 HF5, 2020.2 with no hotfix installed, and 2020.2 HF 1 within their environment. Note: there is evidence of organizations being compromised by this same cyber threat actor without SolarWinds products present in the network. Recent evidence shows that not all organizations with the malicious SolarWinds software were compromised by the threat actor, and that there were different stages of the attack.  Additional vectors are suspected and further investigation is ongoing by CISA and the FBI.
  • What: A cybersecurity intrusion campaign affecting public and private organizations carried out by sophisticated APT actors. The United States government has determined that this attack poses a “grave risk to the Federal Government and state, local, tribal, and territorial governments as well as critical infrastructure entities and other private organizations.”
  • When: Cybersecurity company FireEye discovered the supply chain attack against the SolarWinds products while investigating a compromise of their own network and publicly announced the discovery of the SUNBURST backdoor on 13 December 2020. Confirmed compromises have occurred dating back to March of 2020. Forensic evidence has revealed files associated with this attack being compiled as far back as December of 2019.
  • Where: Multiple commercial industry verticals and government agencies around the world. According to a recent SEC filing by SolarWinds, approximately 18,000 of their 300,000 customers were running vulnerable versions of the SolarWinds Orion platform.

Recommendations

The Center for Internet Security  understands that many organizations do not have full-time IT or cybersecurity staff, nor do they possess network monitoring tools or logging capabilities. As a result, we have provided tiered recommendations below that combine CIS guidance with that of the Federal Government; organizations can apply what is most applicable to their situation and level of expertise. For those organizations in private industry and SLTTs that outsource cybersecurity functions to a Managed Security Services Provider (MSSP), these recommendations can be used to coordinate a response with the MSSP.

CISA recommends the following actions be taken:
  • Apply appropriate updates provided by SolarWinds to vulnerable systems, immediately after appropriate testing.
    • 2019.4 HF 5 Update To 2019.4 HF 6
    • 2020.2 (with no hotfix installed) & 2020.2 HF 1 > Update To 2020.2.1 HF 2
    • If you are running 2019.2 HF 3, 2018.4 HF 3, or 2018.2 HF 6 and do not wish to update completely to one of the above versions, apply the security patch released by SolarWinds to address CVE-2020-10148.
  • Run all software as a non-privilege user (one without administrative privileges) to diminish the effects of a successful attack.
  • Remind users not to visit un-trusted websites or follow links provided by unknown or un-trusted sources.
  • Inform and educate users regarding the threats posed by hypertext links contained in emails or attachments especially from un-trusted sources.
  • Apply the Principle of Least Privilege to all systems and services.

Additionally, CISA has published Current Activity: CISA Releases Free Detection Tool for Azure/M365 Environment.

CISA has created a free tool for detecting unusual and potentially malicious activity that threatens users and applications in an Azure/M365 environment. The tool is intended for use by incident responders and is narrowly focused on activity that is endemic to the recent identity- and authentication-based attacks seen in multiple sectors.

CISA strongly encourages users and administrators to visit the GitHub page for additional information and detection countermeasures: https://github.com/cisagov/Sparrow

CISA has released Supplemental Guidance to Emergency Directive 21-01.

This guidance supplements the Emergency Directive (ED) 21-01 and Supplemental Guidance v1 issued on December 18, 2020.

Specifically, all federal agencies operating versions of the SolarWinds Orion platform other than those identified as “affected versions” are required to use at least SolarWinds Orion Platform version 2020.2.1HF2. The National Security Agency (NSA) has examined this version and verified that it eliminates the previously identified malicious code. Given the number and nature of disclosed and undisclosed vulnerabilities in SolarWinds Orion, all instances that remain connected to federal networks must be updated to 2020.2.1 HF2 by COB December 31, 2020. CISA will follow up with additional supplemental guidance, to include further clarifications and hardening requirements.

For reporting indications of potential compromise, contact: ​https://us-cert.cisa.gov/report​.

For general questions and inquiries, contact: ​Central@cisa.dhs.gov.

For Organizations with Limited or No Cybersecurity Expertise

If the organization has the versions of SolarWinds Orion Platform identified as vulnerable, isolate these systems by doing one of the following:

  1. Unplugging any network connectivity (e.g., Ethernet cable or Wi-Fi) from the system(s) running the SolarWinds application
  2. Isolating any network traffic to/from the SolarWinds system via a network device (e.g., firewall or switch)
  3. Completely power off the system running the SolarWinds software.

For U.S. SLTT organizations that are already a member of the MS- and EI-ISAC, contact our SOC at 1-866-787-4722, or soc@msisac.org for further assistance. For U.S. SLTT organizations that are not currently a member of the MS- or EI-ISAC, but fit the criteria, they can sign-up to be a member and request assistance from the CIS SOC in most circumstances.

For Organizations with Monitoring Tools and Some Cybersecurity Expertise

CISA has created three categories for organizations to use in order to determine the appropriate response and mitigation/remediation. CIS is using CISA’s methodology for consistency:

  • Category 1: Organizations with SolarWinds products, but not any product listed as containing the malicious code
  • Category 2: Organizations that have identified the malicious SolarWinds code in their environment, with or without internet traffic seen to the domain avsvmcloud[.]com
  • Category 3: Organizations that have the malicious SolarWinds code and have confirmed that network traffic has been seen from the organization to the malicious domain of avsvmcloud[.]com and additional command and control (C2) traffic to a separate domain or IP address

Category 1 - Immediate Actions

  1. Follow the instructions by SolarWinds and download the latest release from their portal. Apply the latest release in the environment.
  2. Apply other security patches to servers running the SolarWinds application.
  3. Scan SolarWinds System(s) and apply hardening recommendations from the CIS Benchmarks. For additional recommendations on hardening the SolarWinds Orion Platform, go here. Note: CIS-CAT Pro(free to all MS- and EI-ISAC members) can scan SolarWinds system(s) and offer recommendations.
  4. Continue to monitor the environment for any malicious IOCs or other suspicious activities.

Category 2 - Immediate Actions

  1. Examine network traffic looking for any beaconing activity to the domain avsvmcloud[.]com.
  2. If no traffic is seen to that domain since March 2020, follow all of the instructions listed above for Category 1 Immediate Actions.
  3. If traffic has been seen to avsvmcloud[.]com, look for additional unexplained network external communications from the SolarWinds systems. If no additional unexplained network traffic is located except for the beaconing to avsvmcloud[.]com, follow the steps listed above for Category 1 Immediate Actions.
  4. Conduct an audit of all systems looking for default credentials and new accounts created; perform an organizational-wide password/credential reset.
  5. If additional unexplained external network traffic is found from SolarWinds systems, go to Category 3.

Category 3 - Immediate Actions

  1. If external communications from the organization to avsvmcloud[.]com appear to suddenly cease on 14 December 2020 and the communication was not stopped by any action from cyber defenders, assume the environment is compromised.
  2. If the organization has in-house digital forensic expertise or has brought in external resources, proceed with the following steps.
    Note: If the organization is a U.S. SLTT and does not have the necessary expertise, contact the MS- and EI-ISAC for assistance at 1-866-787-4722, or soc@msisac.org.
  3. For those with expertise, do the following:
    • Forensically acquire system memory and host operating systems of any system hosting all infected versions of SolarWinds Orion
    • Analyze network traffic for additional IOCs
    • Examine SolarWinds host systems for anomalous behavior, including new user or service accounts, new processes running, or other signs of persistence
    • Upon completing the forensic acquisition and network analysis of impacted SolarWinds hosts, immediately disconnect or power down all affected versions of SolarWinds Orion from the environment
    • Block all traffic at the perimeter firewall to and from all hosts outside of the environment where any version of SolarWinds Orion software has been installed (e.g., cloud instances)
    • Identify and remove all threat actor created accounts and other mechanisms of persistence
  1. Once the immediate threat has been remediated, there are a variety of technical steps recommended by CISA for complete remediation. These steps include:
    • Rebuilding systems
    • Restoring network infrastructure managed by SolarWinds to known good versions of firmware
    • Resetting all credentials across the enterprise (users, SNMP strings, SSH keys, certificates, etc.)
    • Forcing multi-factor authentication
    • Additional system and configuration hardening, which can be found on  under the heading of Mitigations
  1. S. SLTT leadership may use CISA Alert AA20-245A, “Technical Approaches to Uncovering and Remediating Malicious Activity”as a guide when reviewing work done by internal or external IT and cybersecurity staff.

Special Note: Due to the sophistication of the cyber threat actor and the length of time this attack has been ongoing, organizations should assume that backups and virtual snapshots may also be compromised. Organizations must take special care to ensure the restoration of backups does not reintroduce the compromise to the environment. Backups should be thoroughly examined by digital forensic experts before any restoration event is completed.

Future Actions

This sophisticated cyber-attack is yet another example of why organizations, regardless of size, must implement cyber hygiene best practices. CIS has a number of longer term operational and strategic recommendations.

  • Ensure cybersecurity is a conversation occurring at the highest levels of executive leadership. Cybersecurity is not an IT problem, it is an enterprise-wide risk management topic that requires attention.
  • Monitor for high-risk events such as account creations, privilege escalation, new services created, security-related services disabled, changes to security posture, unusual network communications, etc.
  • Deploy endpoint protection tools to all hosts and mobile devices. Depending on experience level and budget, consider solutions such as Endpoint Detection and Response (EDR), or a more inclusive Endpoint Protection Platform (EPP). If in-house resources don’t allow this, consider outsourcing to CIS or another MSSP for monitoring and administration.
  • Become familiar with the CIS Controls and implement them according to the level of risk for the organization. Organizations must have a minimum level of cybersecurity to help mitigate threats like this in the future. This includes:
    • Asset inventories
    • Patch and vulnerability management
    • Multifactor authentication
    • Adoption of least-privileged accounts
    • System hardening
  • Implement a risk-based vulnerability management program that includes patching timelines, accounting for the criticality of assets and of the vulnerabilities.
  • Sign up for the  monthly vulnerability scans conducted by DHS for another view of risk from an outsider’s perspective. Be sure to select the option to give the MS- and EI-ISAC access to the scan results so we can monitor for exploitation and understand the threat landscape.
  • Ensure all staff have annual cybersecurity awareness training and that policies exist to provide administrative controls over areas that cannot be controlled with a technical solution.
  • Implement monitoring and logging capabilities for endpoints and network infrastructure.
  • Update (or create if none exists) the Incident Response (IR) protocol for the organization, and include organizations outside of IT such as public information, human resources, legal, executive leadership, and functional organizations. Be sure to include critical vendors and requirements for data and service restorations along with many other considerations. Practice the plan before it is needed through the use of tabletop exercises.
  • Utilize CIS or another third party to perform internal vulnerability assessments and penetration testing to provide IT and leadership an unbiased snapshot of the current risks and condition of the organization’s cybersecurity posture.

Actions Taken by the MS- and EI-ISAC

This incident is fluid and the MS- and EI-ISAC are working continuously to protect our SLTT members. Upon discovery of this attack, the MS- and EI-ISAC Security Operations Center (SOC), Threat Intelligence Team, Computer Emergency Response Team (CERT), and leadership assembled a cross-functional team working around the clock and collaborating with our public and private partners to assist the SLTT community. Specific action items include:

  • For SLTT members with the Albert network monitoring service, analysts are reviewing traffic for IOCs and maintaining communication with members about findings.
  • For SLTT members with the Managed Domain Blocking & Reporting (MDBR) service, the MS- and EI-ISAC is reviewing logs for any evidence of communication with malicious domains associated with this attack. Note: MDBR is offered free to any SLTT organization.
  • Known IOCs for this attack have been added to MS- and EI-ISAC monitoring and control platforms to alert and take immediate action as necessary.
  • Assisting SLTT organizations with questions, incident response, and forensic analysis.
  • Providing curated IOCs via our Threat Intelligence Platform.
  • Additional free services including threat intelligence sharing, CERT and SOC resources, and more.

Available IOCs

Many IOCs have been made public. It is important to note that subdomains created by a domain generation algorithm (DGA) are likely unique to each victim organization and are not likely to appear in another victim’s environment.

The following resources are currently hosting publicly-available IOCs:

** For MS- and EI-ISAC members that have the ability to ingest threat intelligence via STIX/TAXII, contact us at indicator.sharing@cisecurity.org for information on how to get access to our feeds.

Additional Resources

Arrow  New as of January 7, 2021: Additional Resources Added:

 

MS-ISAC_RGB_R

EI-ISAC-logo

Quick References for U.S. SLTT Organizations

CISA Alerts:

  • System and configuration hardening, which can be found on CISA’s Alert AA20-352A under the heading of Mitigations
  • SLTT leadership may use CISA Alert AA20-245A, “Technical Approaches to Uncovering and Remediating Malicious Activity”as a guide when reviewing work done by internal or external IT and cybersecurity staff.
  • CISA Alert (AA20-352A), “Advanced Persistent Threat Compromise of Government Agencies, Critical Infrastructure, and Private Sector Organizations”