Recent Water Utility Attacks Offer a Blueprint for Resilience

Organizations cannot protect systems they don't know about. This is one of the key reasons the CIS Critical Security Controls (CIS Controls) begin with inventorying everything you own, software and hardware.

Attackers scan for, build inventories of, and specifically target internet-accessible operational technology (OT). Unfortunately, many of the owners, operators, and security teams responsible for configuring and monitoring those same OT assets are not fully aware of what they own or what is internet-facing. There are many reasons why, but none of them matter during an incident. What matters is that we know, or can know very quickly, everything we need to know about our systems in a crisis.

Visibility is critical to incident response. On the one hand, many of the impacted entities were able to respond extremely quickly and restore authorized access to affected systems before any public safety impacts occurred. On the other, the discovery of these attacks was due to operational impact, even if minor, and not from security alerts, alarms, or other visibility-related detections on the cybersecurity side. Had the adversary been more advanced, the lack of visibility could have caused much greater impacts, including public health concerns.

Takeaway: Essential OT security practices require that we establish and maintain complete asset visibility across both IT and OT environments.

Reference: CIS Control 1 (Inventory and Control of Enterprise Assets)

Even strong security programs must prepare for disruption. Preventing every cyber attack is not a realistic goal. What is realistic is that proactively building operational resilience can dramatically limit the blast radius when there is an incident. Emergency protocols, including isolating key systems and switching to manual control in near real time, preserve public safety even as a full investigation begins.

Response to the current wave of attacks on water systems across multiple states reinforced the fact that the ability to switch to manual operation is a key feature of industrial systems. Utilities that maintain documented manual procedures, keep operators trained, and regularly test their emergency response plans fare significantly better than those that wait to figure it out in the middle of a crisis. The EPA recommends regular tabletop exercises to test Emergency Response Plans and train staff on response protocols.

Operational resilience is often heavily influenced by resourcing. An incident that would be easily contained in a highly resourced utility can cause prolonged operational impacts in organizations with limited monitoring, limited segmentation, and no dedicated cybersecurity personnel. To boost resilience, organizations large and small can take advantage of community resources through the MS-ISAC, WaterISAC, and OT-CERT as well as federal resources.

Takeaway: Business continuity, emergency response, and crisis management plans aren’t nice to have items; they are operational requirements. And they require regular updates and regular exercise. Every operator must know their manual fallback procedures before a crisis.

Reference: CIS Control 17 (Incident Response Management); AWIA Emergency Response Plan requirements; CISA Water Sector Incident Response Guide

The water sector runs on vendor relationships. Instrumentation vendors, supervisory control and data acquisition (SCADA) integrators, chemical suppliers, managed service providers, and equipment manufacturers all have varying degrees of access to utility systems. And as we learned above, that access is not always well-governed. From underlying operating systems lacking reliable patch management to unsafe connection protocols to an overall misunderstanding of security best practices, the risk provided by third parties creates a complex and challenging web of interdependency.

Third-party risk in OT is different from third-party risk in IT. A vendor remoting into a billing system is a governance and data protection concern. But a vendor remoting into an industrial system is an operational safety concern. The security expectations need to reflect that distinction but often don’t. In recent weeks, we’ve learned that many PLCs are not tucked behind VPNs, rogue wireless access points are common in water districts nationwide, and sometimes vendor security advisories give adversaries the very information they need to remotely reset your devices. While security advisories contain valuable patch information for defenders, they can often act as a tutorial for those with ill intent.

To truly build the operational resilience described above, utilities must be able to both trust their third-party vendors to bake security into their products and processes and hold them accountable when they don’t. This isn’t an overnight solution, but it is required to reduce community-wide risk.

Takeaway: Third-party access to OT systems requires, at a minimum, the same rigor as direct operator access. Every vendor connection to OT should be inventoried, time-limited, monitored, and tied to a specific authorization.

Reference: CIS Control 15 (Service Provider Management); Joint Guidance on OT Remote Access

Water utilities don't make headlines when they work. They make headlines when they don't. A pressure loss, a contamination scare, a boil-water notice, and other downstream consequences of a successful cyber attack on water infrastructure aren't abstract to the public. The impacts are felt in hospitals, schools, restaurants, businesses, and homes, and it reframes the issue of cybersecurity investment from a business cost to a public service obligation.

For water utilities, and many other public services, the responsibility for reasonable security can’t rest solely on the IT department. Technology managers in water districts large and small are asked to solve a problem that Fortune 500 companies struggle with, on a fraction of the budget, with a fraction of the staff, and with infrastructure that often predates the internet. Yet in our rapidly evolving technology environment, cyber and physical security decisions increasingly affect service reliability, community trust, regulatory obligations, and operational continuity.

From incident protection to real-time detection, from proactive emergency planning to clear communication pathways in a crisis, and from preventative maintenance to manual operation during an incident or outage, these are public safety decisions supported by and supporting the technical staff that make it all possible. Anticipating emergencies, careful planning, and conducting drills before an incident reduces confusion, builds confidence, and supports faster decision-making to ensure public health and safety is not at risk.

Takeaway: Cybersecurity in the water sector is a resource and prioritization challenge as much as a technical one. Leadership at every level, from lawmakers and regulators down to the local district, has a role in closing it.

Reference: CIS Controls; EPA Cybersecurity Grant Program; WaterISAC Resource Center; Security Strategy for Critical Infrastructure

The past several weeks have been a stress test the sector didn't ask for. But it reminds us of an idiom that has long been attributed to Churchill: we should never let a good crisis go to waste. We can learn from this event since the pattern is clear: exposed devices, remote reset weaknesses, flat networks, and limited visibility. None of those are unsolvable problems nor do they require a seven-figure budget to address. Some security prioritization and leadership focus today gets us closer to operational readiness tomorrow.

Run through the basics. Complete your AWIA Risk and Resilience Assessment if you haven't already. Segment your OT network from your IT network. Establish visibility into what you own and disconnect anything openly internet-facing or put it behind proper security tools. Placing internet-facing OT assets behind a VPN is one of the highest-impact, lowest-cost steps you can take almost immediately. Build and exercise your incident response plan. Seek out the free resources that exist specifically for this sector.

Even sophisticated threat actors will go after low-hanging fruit first, so your focus should be to put as many obstacles in the way as possible.

One thing you can do today: Search your organization on Shodan. It's free, it takes five minutes, and it will show you exactly what your utility looks like to an adversary scanning the internet. If you find OT exposed that shouldn't be, then you know where to start!

Need help getting started? Join the Multi-State Information Sharing and Analysis Center® (MS-ISAC®) and take advantage of cybersecurity resources, threat intelligence, and incident response support specifically designed to help state, local, tribal, and territorial organizations strengthen resilience.

About the Author

Randy Rose
Senior Vice President, Security Operations and Intelligence

Randy has over two decades of experience across state, local, and federal government. He currently leads Security Operations at the Center for Internet Security where he is privileged to oversee the execution of the Security Operations Center (SOC), Cyber Threat Intelligence (CTI), Multidimensional Threat Intelligence (MDT), Cyber Incident Response Team (CIRT), and Red Team missions in direct support of the Multi-State ISAC. Previously, he led the largest Joint Military Security Operations Center in Europe and the Intelligence Department for the Navy's Global Cyber Operations Task Force.

Early in his career, he served in the U.S. Air Force and later was New York State’s first local government-focused penetration tester. He moonlights as an adjunct instructor at Siena University and the State University of NY at Albany. He sits on the Board of Directors for two local non-profits helping those in need of housing, food, and trauma services. He loves rock music and is willing to have deep conversations with anyone over his favorite bands.