Quality Services to the Energy Industry For Over 30 Years
Energy around New York... Photos by Joe Lopes

Validation, Editing and Estimation (VEE) under AMI and Deregulation

I presented this paper back in 2000 at the AEIC Load Research Conference. The introduction is shown here, with a link to the complete paper below. (There’s a link to the 2000 AEIC Load Research Conference presentation as well.)

INTRODUCTION

Load Research has had a long and varied history, having been carried on the coattails of a variety of applications since the early days of Load Research in the 1960’s and 1970’s. With the collection of load data associated with load research came the need for quality assessment, decisions on how (and whether) to fix problem data and, once data was sufficiently clean, expanding results to the population, whether the load data represented one customer or part of a group representing tens of thousands.

Whoever coined the phrase “garbage in, garbage out” was certainly thinking of situations encountered in load research. With the high cost of load survey equipment and operations, small carefully crafted samples meant that quality from each site was paramount. Consider that one sample site might represent the pattern of tens of thousands of customers and you can appreciate the implications of including problem data in an expansion of results to model population load characteristics.

Problem data and data loss are caused by a combination of factors, including equipment failure, data communications losses, human error, neglect, weather, computer failures and bad luck. Equipment and procedures have improved over time, but data problems have not disappeared. Those of us who have been in the industry for many years can look back on the many hours spent fixing data problems, only to have technology eliminate them, or so we think – and create new ones. 

In the past few years, Automatic Meter Reading (AMR) and Automated Metering Infrastructure (AMI) technology and systems have become more prevalent, and many utilities have adopted such systems, or at least initiated pilot programs to test the technology and logistics, including customer response.  These new systems enable more complex pricing for small customers, two-way communications, load control options and linking to Home Area Networks (HAN) for residential, as well as Energy Management Systems (EMS) in businesses.  The potential for collecting interval  data is almost limitless, and so is the need for validation, editing and estimation (VEE) to ensure that the data is valid!

Over the years, the reasons for collecting data have changed, from PURPA compliance, to conservation studies, load management, rate design, cost-of-service studies, demand-side management, technology assessment, billing, profitability, competitive threats, load profiling to enable reconciliation of sales from multiple suppliers within a service area and, most recently, dynamic pricing under AMI.  The issues of data quality remain, with only slight differences, and the turnover of experienced staff has meant that some of the experience and techniques for validating, processing and analyzing load data has been lost.

This paper was written to refresh and reminisce for the old timers, teach the new generation, and highlight the similarities and differences associated with data validation, editing and estimation (VEE) and data expansion under the new deregulated utility environment as it will be affected by the new technologies involved in Advanced Metering Infrastructure (AMI) and Meter Data Management (MDM).   As we march through the 2000’s, we need to take a fresh look at validation and editing techniques and their implications on the new applications for the data.

Published Paper: Lopes-Valeditaeic2k

2000 Presentation: valedit

INTRODUCTION
Load Research has had a long and varied history, having been carried on the coattails of a variety of
applications since the early days of the 1960’s and 1970’s. With the collection of load data associated
with load research came the need for quality assessment, decisions on how (and whether) to fix problem
data and, once data was sufficiently clean, expanding results to the population, whether the load data
represented one customer or part of a group representing tens of thousands.
Whoever coined the phrase “garbage in, garbage out” was certainly thinking of situations encountered in
load research. With the high cost of load survey equipment and operations, small carefully crafted
samples meant that quality from each site was paramount. Consider that one sample site might represent
the pattern of tens of thousands of customers and you can appreciate the implications of including
problem data in an expansion of results to model population load characteristics.
Problem data and data loss are caused by a combination of factors, including equipment failure, data
communications losses, human error, neglect, weather, computer failures and bad luck. Equipment and
procedures have improved over time, but data problems have not disappeared. Those of us who have
been in the industry for many years can look back on the many hours spent fixing data problems, only to
have technology eliminate them, or so we think – and create new ones.
In the early days of load research, between approximately 7,000 to 10,000 BC (10,000 days Before
Competition, or between 1968 and 1975), magnetic tape recorders with bulky cartridges, and later
cassettes, were used for load studies. This was a major improvement over strip charts, which were hard
to read and generally unreliable, somewhat like stone tablets. Mag tapes did not solve all the problems.
The primary data problems were caused by tapes jamming or outages, which caused missing or too few
intervals of data, and bad splices, which caused mismatches between the end of one tape and the
beginning of the next. When the customer lost power, the recorder stopped, too. In about 7,000 BC
(1975), man discovered batteries! This enabled a time pulse to be continuous on recorders, so if the
customer lost power, the recorder did not. This eliminated the uncertainty about when the interruption
occurred, particularly when there were several. Then, man discovered electronic recorders about 3,000
BC (mid 1980’s), and this virtually eliminated the mechanical failures associated with mag tape (Cro-
Mag) recorders. Further advancements in electronics and means to “download” data and install
equipment were also made in the past 15 years, such as an optical probes, telephone modems, wireless
communications and meters “under the glass”.
Over those years, the reasons for collecting data changed, from PURPA1 compliance, to conservation
studies, load management, rate design, cost-of-service studies, demand-side management, technology
assessment, billing, profitability, competitive threats and, most recently, load profiling to enable
reconciliation of sales from multiple suppliers within a service area. The issues of data quality have
1Public Utility Regulatory Policy Act of 1978, which required all utilities to collect load research data on major rate
classes comprising more than 10% of retail sales.
remained, with only slight differences, and the turnover of experienced staff has meant that some of the
experience and techniques for processing and analyzing load data has been lost.
Validation, Editing & Expansion in a Deregulated Environment
2000 AEIC Load Research Conference 2 J. Lopes, Applied Energy Group, Inc.
As a result, this paper is being written to refresh and reminisce for the old timers, teach the new
generation, and highlight the similarities and differences associated with data validation and editing
under the new deregulated utility environment. As we approach 2000 AD (that’s about 2,000 days After
Deregulation, which began about 5 years ago), we need to take a fresh look at validation and editing
techniques and their implications on the new applications for the data.

1 comment

1 Saint Usifo { 05.09.21 at 4:13 pm }

More information

Leave a Comment