Monitoring and Support During the CrowdStrike Falcon Outage

 

On July 19, 2024, a flawed update to CrowdStrike’s Falcon sensor, a cybersecurity tool widely used to protect devices like computer workstations and servers, resulted in a global IT outage. The update triggered a memory error that led to system crashes and widespread Blue Screen of Death (BSOD) issues at private and public sector organizations globally.

Learn about the details of this outage and how the Multi-State Information Sharing and Analysis Center^® (MS-ISAC^®) supported members through it.

Manual Operations and Loss of Access in the Public Sector

The CrowdStrike Falcon outage affected state and local governments' ability to deliver essential services and maintain critical operations.

• 911 Dispatch Centers: In cities like Phoenix, AZ, and Portland, OR, dispatch systems went offline, forcing operators to manually record caller information. In Alaska and Ohio, some 911 systems went down completely, prompting police departments to publish alternative emergency contact numbers.
• Law Enforcement and Fire Services: In Columbus, OH, police lost access to law enforcement databases and license plate readers, while firefighters couldn’t receive dispatch information in their vehicles.
• Municipal Services: The outage affected DMV offices, election databases, and official websites in multiple states. Some jurisdictions responded by standing up temporary websites to maintain service continuity.
• Downtime Duration: Several states reported restoring core operations in 24 hours, with full endpoint recovery taking up to a week. For example, Oklahoma restored 819 servers and resolved 10,000 endpoints by the end of the first week following the outage.
• Estimated Mitigation Costs: While exact figures for public sector losses were still emerging at the time of publication, CIO reported that total remediation costs across all sectors were estimated at $700 million based on an average cost of $82.50 per affected machine. Given the scale of public sector deployments, especially in emergency services and municipal IT, the costs of the outage likely represent a significant financial burden for this sector, particularly for smaller jurisdictions with limited IT budgets.

Effective Response Coordination by the MS-ISAC

The MS-ISAC was instrumental in helping public sector organizations respond effectively to the CrowdStrike Falcon outage:

• High-Engagement Webinar: It hosted an emergency call on July 19, arranging for the CrowdStrike CTO to directly address nearly 4,000 participants, including state and local government IT leaders, with details about the incident.
• 24x7x365 SOC Support: The CIS Security Operations Center (SOC) fielded calls throughout the day on July 19, helping members triage issues. By that evening, call volume dropped to zero, and no overnight incidents were reported, indicating strong containment.

• Timely Guidance: Members received a comprehensive email update on July 20, including CrowdStrike’s latest recovery tools and dashboards tailored for environments like Azure, AWS, and Google Cloud.
• Actionable Intelligence: The MS-ISAC’s Cyber Threat Intelligence (CTI) and Countering Hybrid Threats (CHT) teams issued intelligence reports with new Indicators of Compromise (IOCs). These reports were among the only reliable sources of intelligence available to law enforcement and emergency management partners at the time.

• Public Safety Monitoring: The MS-ISAC tracked the impact of the outage on 911 centers and ensured that backup protocols were functioning, helping jurisdictions maintain emergency response capabilities.
• Clear Communications: Updates posted to the CIS website and CIS social media channels guided members to authoritative resources. Some member organizations reported that the actionable information from the MS-ISAC was the most they received from any other source during the incident.

Lessons Learned from the CrowdStrike Falcon Outage

The CrowdStrike Falcon outage exposed the vulnerabilities of public sector systems that rely heavily on third-party cybersecurity vendors. It also highlighted the importance of robust disaster recovery plans, backup protocols, and coordinated response mechanisms.

More than that, however, the incident highlighted the vital role the MS-ISAC plays in safeguarding public sector organizations. From real-time intelligence to hands-on support, the MS-ISAC ensures that state and local governments can respond quickly, stay informed, and protect their communities during notable cyber events.

Want this same level of support during the next major security incident?