Manage ICS Changes before they Manage You!

Anshul Agarwal

Effective Industrial Control Systems (ICS) change management is pivotal in heavy process industries. These systems continually change due to factors such as renovations, capacity additions, process changes, compliance norms and operational & maintenance cycles. Keeping track of changes is a challenging task, so automating these changes becomes essential. Despite some help from DCS Engineering Workstation Software (EWS), manual methods are prone to errors.

This blog post will explore the current challenges and repercussions, and suggests an automated solution – configuration management. It provides insights into managing ICS changes efficiently and a future-oriented perspective on the subject.

Renovation & Modernization (R&M)

Process plants undergo R&M activities for increased efficiency or to replace equipment that has reached its end of life. Additionally, it is common for R&M activities to have a phased approach. This often requires that the plant remains operational at least at partial capacity. Under this scenario, control system logic and configuration changes happen regularly to account for the two activities occurring in tandem – commissioning activities for the new equipment and changing existing plant design. Managing changes at such a fast pace, in a running plant, and keeping a marker of each change to an authorized/responsible person is quite challenging.

Capacity Addition

Owner operators frequently undertake projects to increase system capacity, such as adding new equipment. This equipment often utilizes many existing common plant systems and capabilities, like sourcing cooling water from an existing pump house or increasing the existing control system’s capability to operate the newly added assets. Whether you utilize the available spare capacity in the plant’s control system or add new control system hardware, there will be new logic written to the system along with the needed modifications to the existing logic.

Change in Process

Owner operators may change the output parameters or change the output itself, either permanently or periodically, which would require a new configuration in the control system. Change in process may also require changes to field equipment or their operating parameters. This requires changes to operating logic in the control system, which may be accompanied by changes to control system hardware.

Change in Norms of Statutory Compliances

I would clarify this point using two examples. One is a power plant where the government may ask generating companies to use a Free Governor Mode of Operation (FGMO) or Restricted Governor Mode of Operation (RGMO) as the Primary Frequency Response (PFR). Another example would be a refinery where governments, for environmental reasons, tighten fuel quality standards over time or increase ethanol blending.

Operations & Maintenance (O&M) Cycle

Every process plant undergoes these stages as part of the O&M’s cycle: normal operation, breakdown maintenance and overhauling. Each of these stages requires a different logic to be applied, even for the same set of field equipment.

Throughout the lifespan of any industrial process plant—and thus, ICS engineer—breakdown maintenance and overhauling are activities that occur repeatedly. These tasks necessitate ICS engineers to disable certain aspects of the logic, adjust/force set points within the logic, possibly add new logic/configuration or a combination thereof, as required. This procedure is unavoidable as it ensures safety and allows for the necessary tasks to be performed. Post completion of maintenance activities and overhauling the system is to be normalized for standard operation, which involves re-enabling the bypassed logic and returning the set points to their previous values, or adjusting them as necessary.

Current Scenario

ICS engineers are required to maintain a record of control system configuration changes manually. Many times, they have help from the DCS Engineering Workstation Software (EWS), which keeps an audit trail for the changes made.

Challenges with the Current Scenario

ICS engineers may miss out on properly and sequentially maintaining the manual record for the changes and they will be prone to human errors.

ICS engineers at the plant normally work multiple shifts and are also rotated across shifts. So, an engineer who joins the next shift may not be aware of the changes that were made to the control system configuration in the earlier shifts and may not understand the context of the changes made. Further, loss of change data may occur over time and over multiple system changes if the records are not properly detailed, maintained or indexed.

The audit trail facility available from control system OEMs will generally have the following challenges of its own:

- They can only track changes for OEMs' own make of control systems. Control systems of various makes may control different plant equipment, and logic changes are often required across multiple control systems. This situation makes it difficult for ICS engineers to track changes across different control systems.
- Audit trail software can generally track changes across only one DCS. Again, a plant has multiple DCS’s, and tracking changes across multiple, even if each system is running an instance of the audit trail software in its EWS, becomes challenging.
- Taking reports for review by managers or automation team leadership can be challenging because engineering workstations are normally located in server rooms or control panel rooms near to control rooms in the field. These machines cannot be accessed remotely through the IT network, making it difficult to get on-demand reports when required.
- The audit trail software does not have an authorization mechanism to keep track of who authorized a particular change in the system or to record details as to why a change was made or reversed. This becomes important in scenarios where many changes are made only for trial and reverted later.
- Audit trail software does not alert users to system vulnerabilities from the cybersecurity perspective, many of which are related to how ICS engineers manage their systems. Even if that data becomes available to ICS engineers through other means, for example through cyber security audits, but knowing the criticality or priority among multiple vulnerabilities and a mechanism to manage/track those vulnerabilities in the system is equally important which stands missing.
- ICS engineers need to understand the impact of any change in system configuration on the overall system. EWS application software generally gives out those details as the list of logic sheets/areas where the data point being changed is used., which makes it difficult for an ICS engineer to understand the impact and make quick decisions.
- Backups - EWS application software can only create backups for its own system and not for other control systems that are part of the process plant. Even if different systems have their own backup solution, synchronizing these backups, especially during system restoration, would also become challenging in the absence of a single source of backup and change records.

Unwanted and Unseen Consequences of Current Scenario

ICS engineers are usually present during the final step on any maintenance activity because only after the maintenance on field equipment is complete (by electrical and mechanical maintenance teams) can ICS engineers normalize the control systems for operation. So being the ones closing things up in the last leg of the maintenance, they are almost always under pressure to reduce downtime. This can lead to the following:

(A)    The current scenario reduces their efficiency and pushes them to overwork, increasing the possibility of further mistakes in maintaining/ normalizing control system configuration and thus potential failures or safety incidents.

(B)    The cybersecurity posture of the control system takes a back seat, and many times even the established security barriers are bypassed. Such bypasses are rarely tracked and not alerted later because of a lack of such a mechanism available in the system.

(C) Tracking changes manually can be is only prone to errors but is also slow and subjective to a certain extent. This can be detrimental to the overall performance of the plant or organization as ICS engineers are sometimes bogged with the task to find missed/erroneous changes post failures.

(D)  Increased Mean Time to Repair (MTTR) and system failures/shutdowns or reduced efficiency because of a small miss in the control system configuration will directly lead to loss of revenue.

The Way Out

The way out for automation engineers (ICS engineers) is to automate this activity.

This can be done by deploying a configuration management solution to address the above challenges. If I were to put together my wish-list that I would look for in any such solution it would include the features below:

Cover and track changes across almost all major makes of DCS and PLCs.

Enable an authorization mechanism workflow with the facility to add remarks and a process to review, revisit and revert a change done to the system.

Provides a list of control system vulnerabilities, their priority and the recommended solutions to mitigate those vulnerabilities there improving the plant's cybersecurity posture.

Graphical representation of interconnections and risk analysis will help ICS engineers to make better and faster decisions.

Backup and restoration across different makes and types of control systems is possible.

The solution that does not reside or run on any of the DCS machines and therefore does not impact the control system's performance and thereby has no impact on my control system guarantees.

Deep coverage from Level 0 to Level 3.5, including IT & HMI Hardware – basically working on and tracking system changes on all levels of automation system

A solution that can be deployed during Factory Acceptance Tests (FAT) and/or commissioning activities to create system reference from them

A solution that can be aligned to your plans for expansion (addition of new DCS – different or same OEMs), upgrades and R&M - with backward as well as forward compatibility.

Support for statutory compliances with needed checks and reporting.

Can provide reporting and tracking facilities on an organizational level with enterprise-level implementation.

A solution with a well-established track record – good references & use cases from an organization with a vision for the future.

In conclusion, managing changes in Industrial Control Systems (ICS) is vital in a rapidly evolving technological landscape. Whether it is due to renovation & modernization projects, capacity additions, variations in process, compliance norms, or regular operations & maintenance cycles, ICS engineers are regularly challenged with tracking, authorizing and implementing adjustments to ensure system security and efficiency.

Current manual methods have inherent risks of errors and inefficiency. As such, the ideal solution would be to automate this activity with a configuration management solution. This comprehensive tool should be capable of tracking changes across various DCS and PLCs, providing control system vulnerabilities, offering graphical views of interconnections, managing backups across different systems, and most importantly, ensuring improved cybersecurity posture and compliance regulation adherence. An effective solution would not only enhance system efficiency but would also streamline workflow, decisively offering a significant step in proactive ICS management.

Feel free to reach out to me for further discussion.

OT/ICS Cybersecurity