Ordering Information:
ORDER NUMBER:  91226
DATE AVAILABLE: Fall 2008

PRINCIPAL INVESTIGATORS:
Roger O’Halloran, Richard Jarrett, Geoff Robinson, and Peter Toscas

OBJECTIVES:
The overall objective of this project was to evaluate and test data processing methods for detecting anomalies and abnormal events in online distribution system monitoring data. An important goal was to distinguish patterns related to rapid changes in water quality from the normal variability and trends that occur in distribution systems.

BACKGROUND:
Online monitoring systems have been increasingly adopted to monitor water quality in distribution systems. They can potentially provide rapid warning of system events or contamination, which is required to meet stringent water quality standards as well as for security issues. AwwaRF has funded a number of activities in this area that have identified several issues that need to be addressed to enable accurate data analysis and timely identification of anomalous events.

HIGHLIGHTS:

1. Systematic collection of relevant metadata is essential to allow interpretation of online water quality monitoring data.
2. Examining differences in the data first provides a simple way to find events of interest.
3. Control charts of data differences using either 5-sigma or dynamic limits can be used to identify anomalies.
4. Forward prediction using Kalman filters can be used to identify anomalies on a variety of time scales and to assess whether such anomalies are worthy of attention.
5. Tracking water flow between sensor locations using innate fingerprints will allow confirmation and tracing of anomalies in real time.

APPROACH:
The project was conducted in the following stages:

1. Literature Review and Methodology Selection
2. Sourcing and Validating Online Data Sets
3. Detailed Data Analysis
4. Development of Recommendations
5. Validation and Testing of Proposed Methodologies

Data processing methods were assessed against their ability to:

• develop baseline levels, including adjustments for diurnal, weekly, and seasonal trends, and the usual levels of variation;
• discern significant trends and recognize large single point disruptions;
• examine a series of data from a single site, including the possibility of simultaneously measuring and analyzing several variables; and
• undertake a coordinated analysis of data collected at a number of different sites, and detect trends and patterns across a number of sites.

RESULTS/FINDINGS:

1. Data collection, evaluation, and quality control

• Information collected must be time-stamped using a coordinated timekeeping system, and artifacts such as daylight saving must be addressed.
• A regular schedule of sensor calibration and maintenance is essential.
• A log of relevant system metadata is vital to allow proper assignment of events.
• A pre-processing step is required to identify faulty data.

2. Anomaly detection

• Control charts of first differences with 5-sigma or dynamic control limits, particularly for data recorded every 10–15 minutes, were found useful for identifying anomalies.
• Forward prediction using Kalman filters was the best technique for identifying anomalies on a variety of time scales and determining whether a change is worthy of attention.

3. Event confirmation and identification

• Simultaneous assessment using several univariate Kalman filters was seen as preferable to a more complex multivariate approach and allowed visual assessment of multiple alarm codes for different variables.
• Tracking water flow between sensor locations using innate fingerprints was demonstrated using real distribution system monitoring data.
• The water tracking methodology being developed promises to allow confirmation of events and tracking the propagation of contamination episodes through the distribution system in real time.

IMPACT:
Online monitoring systems require standard operating procedures to provide accurate data and reliable detection of events. An overall map and description of the distribution system is essential. Individual water utilities should determine alarm levels using the event template methodology reported. Confusion on what to do when the alarm goes off means that development of a confirmation and validation step is essential. The water parcel tracking technique should enable confirmation and tracking of contamination events in real time. Linking this method to calibration of hydraulic and water chemistry models of distribution systems should also be investigated.

RESEARCH PARTNER:

• CSIRO, Australia

PARTICIPANTS:

• Philadelphia Water Department
• Oklahoma City Water Department
• City West Water Ltd., Sunshine, Victoria, Australia
• Hunter Water, Newcastle, NSW, Australia

ISBN 978-1-60573-028-9