Traditionally, rural areas handled utilities on a per-house basis with water wells and septic tanks; however, most new suburbs were built with full service water and wastewater systems handled by their cities. This resulted in utility providers handling utilities in larger areas with fewer people – and, therefore, smaller budgets.

Now, with over 50,000 utility providers across the country, many with aging pipes and control systems as well as ever shrinking budgets, cities are facing a new realm of cybersecurity.

The control systems for many utilities are increasingly networked, and some are even on the internet. Unauthorized access of these systems by bad actors is no longer a hypothetical — as we have seen by the hack in Oldsmar, Florida.

To combat this, the Federal EPA, along with the various state EPAs, have implemented new rules and regulations to encourage better cybersecurity practices. One such rule, specified in the American Water Infrastructure Act of 2018 (AWIA), mandates that all water and wastewater utilities of a certain size perform a Cybersecurity Assessment.

DMC recently had the opportunity to perform one of these assessments for a municipal water/wastewater utility. In this article, we will share some of our generalized findings.

A cybersecurity assessment is the first step in the broader cybersecurity lifecycle. It is a process of uncovering the cybersecurity vulnerabilities in a system and rating the risks associated with those vulnerabilities.

There are many guides and resources available for the cybersecurity lifecycle and cybersecurity assessments: such as the ISA 62443 standard. While these guides and standards are applicable for numerous types of systems, it is still useful to understand some of the unique risks that water and wastewater control systems face.

1. Critical Uptime

While most control system owners and operators would likely describe the uptime of their systems as critical, few systems have as much bearing on people’s lives as water and wastewater systems.

Unplanned downtime doesn’t just cost money, but it has the potential to affect the health and safety of hundreds to hundreds of thousands of people. This means that every vulnerability is likely to result in a much higher risk because the consequence of any bad act is so extreme.

Risk = Likelihood X Consequence.

It is nearly impossible to improve the cybersecurity of a system without understanding and rating its various risks. The formula above is a simple way of producing a numerical value for each risk by considering the likelihood and consequence of some action or exploit occurring.

2. Vendor Reliance

Many water and wastewater systems are municipally run and funded. This often means that there may not be any Instrument and Electrical (I&E) or control systems specialized persons on staff. Rather, many of the systems are maintained by the vendors that installed them. There may even be multiple vendors maintaining different systems at a single plant.

For each different vendor solution, you must consider how the vendors will access their system for support, the interoperability of their system with others in the plant, and the long-term support potential.

3. Geographically Disperse Locations

To provide services across an entire community, water and wastewater operations must have controls equipment miles or even dozens of miles apart. Between the water towers, pump stations, lift stations, treatment plants, offices, and meters, there may be controls equipment dozens of miles away from the main operating centers.

This not only leads to a larger attack surface but increases the amount of time and resources it takes to secure all the sites. This often results in an architecture using the internet to connect all of these locations together: which adds extra risk not present in systems with a single private network.

The Zone and Conduit Model is a method of breaking up systems by their security level and connection to other levels.

Grouping objects into similar security zones and identifying their conduits of information to the other zones can help to simplify the security of complex networks of machines and systems.

Checkout this blog by the National Cybersecurity Institute of Spain for more information on the Zone and Conduit Model.

There are common sense steps to securing a control system that likely make sense across the spectrum. For example, the US Cybersecurity & Infrastructure Security Agency (CISA) has a number of recommendations available that provide a great place for any control system owner or operator to start (https://www.cisa.gov/uscert/ics/Recommended-Practices). Below are some of the recommendations that we have found to be especially relevant to water and waste water utilities.

1.  Remove all remote access software from SCADA/Controls PC’s immediately.
   • We understand some of these PCs may also serve as an office computer, but we recommend at least removing Zoom, Teams, TeamViewer, and any other remote access applications from these PCs. Ideally SCADA and Controls PC’s would only run applications necessary to their controls functions, and all office, email, or time management applications would be run on a separate machine.
2. Keep track of all vendor/3^rd party access to your system.
   • Ensure that you understand if and how your vendor gets access to their system if they need to support it. Ideally, support would come from an in-person technician, but that may not always be feasible for time or budgetary reasons. If remote support is necessary, make sure you have a full understanding of where and how this access is granted. Request that any passwords used for access are secure and unique to your plant and discuss with the vendor the possibility of unplugging any gateways or modems used for remote access when not in use.
3. Segregate your office and controls networks.
   • The fewer devices you have on your controls network, the more secure your system will be. Ideally you would only have the bare minimum devices required to carry out your systems function (e.g. PLCs, HMIs, SCADA PCs, Flow Computers, etc.) and all other office computers, phones, or security cameras would be on a completely separate network. As many systems increasingly need the internet to function, you may not be able to completely air gap your controls and enterprise networks. In these cases, we recommend using your router or managed switches to create VLANs to virtually separate these networks.
4. Lock down and monitor all sites.
   • A physical attack may not always come in the form of an armored vehicle or an agent with a USB drive hidden in their shoe. Anyone could cause just as much damage if given unmitigated access to certain systems. For remote sites, consider implementing security cameras to monitor the access points or equipment. For plants or operations centers where staff may be moving around throughout their shifts, consider keycard systems to ensure the doors are locked at all times but personnel are not inhibited from carrying out their jobs.
5. Maintain backups of all programs and mission critical data and consider backup hardware.
   • Even without a cyber-attack, it is not uncommon for hard drives to fail or PLCs to short. For all of your controllers and PCs, consider how you would resume operations if that device or software were to fail. Consider creating PC images of your SCADA computers and keeping backups of PLC programs. If you rely on vendors for this equipment, request that your vendor keep secure backups and potentially even keep a copy at your site. Ideally you would follow Peter Krogh’s “3-2-1” rule for backups which is to keep “3 backups on 2 different types of media and at least 1 offsite”.

Cybersecurity is best achieved using an approach of layered protection. No mitigation or security measure is perfect, but when multiple different forms of protection are layered on top of each other, you can be more assured that one of the layers will impede an attack.

Water and wastewater systems in the US are in a unique position of being major targets of cyber-attacks, but they are often lacking the resources to implement major cybersecurity measures. Thankfully, federal and state agencies across the country have taken notice, and, through efforts like those outlined in AWIA, measures are now being taken to secure these systems and educate the owners and operators on good cyber hygiene.

Ultimately, all control systems are different and will have different needs. The risks and recommendations mentioned here may not make sense for all control systems or even all water and wastewater systems, but cybersecurity is an effort of continuous improvement. Hopefully some of the risks and recommendations in this article will provide insight into your own system and will motivate you to take the initial steps of improving the cybersecurity of your system.

If you are looking to have a cybersecurity assessment performed on your control system, reach out to us or checkout our blog on Preparing for an Assessment of your Industrial Control System.

Learn more about DMC's Industrial Networking and Cybersecurity expertise and contact us for your next project.

The post A Cybersecurity Assessment for Water and Wastewater Systems appeared first on DMC, Inc..

A Cybersecurity Assessment is a process of uncovering the cybersecurity vulnerabilities of a system and rating the risks that are related to those vulnerabilities. Assessments will vary widely depending on the system under consideration, the assessment team, and the needs of the managing organization. Yet there are some commonalities you will likely find with any assessment. Below we will lay out what you can generally expect from a Cybersecurity Assessment from the preparation and the onsite work, to the deliverables and the next steps.

Preparation

Before anyone arrives on location for the onsite portions of the assessment, there are some general preparation steps that can be expected for most assessments:

• Establishing the Team – Identify who will be on the assessment team, considering the following roles:
  • Assessment Team Lead
  • Assessment Engineer(s)
  • Plant/Site Manager
  • Controls/I&E Lead
  • Safety Lead
• Preparing Documents and Drawings – Locate any technical documents and drawings or diagrams that will help the assessment team better understand the system under consideration.
• Questionnaire – Answer a survey of questions to gain information about the system and policies and procedures of the organization.

Onsite

The onsite portion of the assessment, like the preparation work, is essentially an effort in information gathering. The technical work will vary widely depending on the scope of the system, but you will often find the following phases:

• Kick Off – The entire team will meet to review the objectives and schedule for the assessment.
• Site Tour – Site staff will show the assessment team the plant, control rooms, and networking rooms.
• Visual Inspection – The assessment team will perform a visual inspection of the networking and controls equipment.
• Asset Inventory – The details of all critical equipment including manufacturer, firmware, IP Address, and more will be documented.
• Network Scanning – The network will be passively scanned to discover devices, configurations, and potential vulnerabilities.
• Risk Assessment and Rating – Together with the entire team, the vulnerabilities and threats to the system will be identified, categorized, and rated.
• Report and Review – A report will be created and presented detailing all of the findings including vulnerabilities, risk assessment, asset inventory, and network diagrams.

Deliverables

When considering what you need to be getting out of your assessment, consider your needs and why you are conducting an assessment in the first place. You may have a regulatory requirement to produce a cybersecurity assessment report detailing the cybersecurity vulnerabilities of your system, or you may have a mandate to reduce certain risks over the next year. Consider these common deliverables you may want from an assessment:

• Vulnerability Report – A report detailing all discovered vulnerabilities of the system.
• Risk Assessment Report – A report detailing the risks (i.e., the likelihood and severity of potential incidents as they relate to system vulnerabilities).
• Mitigation Recommendations – A list of potential controls or strategies to mitigate the cyber risks to the system.
• Network Diagrams – Diagrams should detail your networked equipment and their corresponding security zones.
• Asset Inventory – A list detailing all critical equipment and their identifiable information.

Next Steps and the Cybersecurity Lifecycle

After an assessment is complete, you will want to make full use of the newfound information about your system. The assessment report will be the guide through the next phases of the cybersecurity lifecycle. The exact form of the cybersecurity lifecycle will vary between different sources (we recommend looking at the ISA/IEC 62443 standard and the NIST Framework for Cybersecurity), but they all generally present the same work. We choose to break the lifecycle into the following 4 phases:

• Assessment – The system will be holistically evaluated to uncover the cyber vulnerabilities and cyber risks to the system.
• Design – Using the risk analysis and proposed mitigations from an assessment, appropriate engineering controls and strategies will be chosen in order to reduce the cyber risk of the system.
• Implementation – The entire team will work together to install and configure the selected controls and strategies, minimizing disturbance to operations.
• Maintenance & Response – Even after solutions have been implemented the system will need to be monitored, maintained. Infrastructure, both controls and policies & procedures, will also need to be in place to better help the organization better respond to potential cybersecurity incidents.

Learn more about how DMC can help you manage the cybersecurity of your system and contact us today!

The post Preparing for a Cybersecurity Assessment of Your Industrial Control System appeared first on DMC, Inc..