Battling data silos

Data collection schemes abound, but operational silos limit effectiveness

Dan Headrick
21 June 2017

5 min read

While big data analytics, the Industrial Internet of Things and artificial intelligence are creeping into our homes, cities and industries, most utility companies have yet to see the full value of these technologies. Although leading utilities are experimenting on many fronts, experts caution that only a fully integrated approach that eliminates traditional data silos will provide the insights needed for effective management.

Tucked between Rancho Cucamonga and San Bernardino in Southern California, 20 suburban homeowners in the city of Fontana have volunteered for a visionary pilot project: to test whether smart homes can function as operational grid nodes, managing demand response and load shifting while generating electricity from rooftop solar panels.

“The thermostat may be the hub that everything integrates around,” said retired utility executive Randy Brecheisen, who serves on boards of member-owned utility organizations in the southeastern United States. “Appliances, garage doors, everything can all integrate into the controller of that thermostat. We’re trying to capture and integrate all of the resources to further leverage upstream. Solar, demand side management (DSM) resources, electric vehicle potential, distribution and generation, batteries...if we can integrate all that into a resource and have control of all that as a resource, then we’re in a position where we can impact the generation upstream and transmission supply.”



But smart meters represent just one slice of rapidly proliferating information technologies competing for utility dollars. From generation, transmission, distribution and load management to market trading, retail pricing, consumption, billing and storage, data has become the true energy behind the industry. Utilities are scrambling to adapt, but few have a vision for achieving a comprehensive overview or management capability.

“It’s a very, very confused marketplace out there,” said David Socha, a practice partner with the Singapore-based energy consulting firm Teradata International. “Everyone’s suddenly a data and analytics company: ERP vendors, smart meter vendors, startups, even companies that used to sell you books online. It’s no wonder that utilities might be hesitant to jump in.”


The combination of artificial intelligence (AI), sensor-enabled data collection via the Industrial Internet of Things (IIoT) and data analytics offer dizzying potential. But which technologies will become industry standards? What are the best applications? How can they be integrated with other digital initiatives throughout the power generation and distribution industries?

According to Navigant Research, a global market research and consulting firm that analyzes clean technologies, many utilities are starting with the lowest level of data collectors: smart meters. Worldwide, Navigant reports, smart meters, particularly with advanced metering infrastructure (AMI) communication capabilities were expected to represent about 30% of all smart meters by the end of 2016, and are forecast to rise to 53% by the end of 2025.

Swiss electricity producer and distributor Alpiq, for example, recently launched a smart meter and AI algorithms for buildings and facilities management. Alpiq continuously measures electricity consumption and loads on the grid, factors in weather forecasts and tracks electricity prices. Based on data patterns over time, it also learns the behavior of users for each load-controlled device installed in homes and commercial buildings.

In 2017, the German research organization Fraunhofer-Gesellschaft introduced an AI-managed smart meter that goes a step further, not only measuring but also controlling the electricity usage of as many as 20 appliances through a single node.

Enedis, the French networks business of EDF Group, which manages 95% of the power grid across France for 35 million customers, has invested heavily in a data and analytics platform with aims to manage usage regionally. Its “Linky” smart meters send and receive data and instructions through computer hubs at transformer substations, enabling Enedis to manage loads automatically and gain greater visibility into the grid.


While smart meters can help utilities shed loads at times of peak demand or give consumers insights into managing their energy use, they do little to keep the system running. In New South Wales, Australia, state-owned electricity infrastructure company Ausgrid is tracking that issue by outfitting 22,000 workers with digital field data collection technology.


According to Navigant Research, smart meters with AMI communications capabilities are expected to account for 53% of all smart meters by the end of 2025.

The technology guides workers in maintaining more than 250 power stations, 500,000 power poles, 30,000 small distribution substations and nearly 50,000 kilometers (31,000 miles) of above- and below-ground electrical cables. As a result, Ausgrid has improved end-to-end work cycle productivity by an average of 72%. The company expects to save US$60 billion by 2025, according to a January 2016 World Economic Forum white paper entitled “Digital Transformation of Industries” in collaboration with Accenture.

Canadian utility Manitoba Hydro International, meanwhile, is testing audiovisual headsets for its workers in the field. Miles away, managers use real-time simulation and location positioning information transmitted by the headsets to identify the affected equipment and access its maintenance history. That information is fed back to workers on site, along with information on how to make the repair. Depending on what they learn from the pilots, company officials hope to expand the capability to other operations.

“We’re all walking around with the world’s information in our pockets,” said Ken Hepburn, vice president of marketing of Silicon Valley-based company RealWear, which is testing its field service headgear in Canada. “AI will merge with this portability to push contextually relevant information. That’s inevitable. It’s just a matter of when that will happen, but it will probably be pretty fast.”



Spanish utility Iberdrola has taken the collection of information systemwide. From a single, automated control center in Toledo, Spain, workers operate in real time with machine learning, analytics and robotics to perform condition monitoring, predictive forecasting and reliability maintenance. These functions are particularly critical to Iberdrola’s varied clean energy mix: 7,000 megawatts (MW) of installed power from 220 wind farms, 70 mini-hydropower plants and more than 6,000 wind turbines across nine countries.

Information from sensors monitoring about 2 million operational signals, for example, help managers improve insights into fault detection, turbine and control system malfunctions. Preventive measures can be taken remotely, reducing operational and maintenance expenses. Iberdrola expects to save US$387 billion over the next decade and cut 2.4 billion metric tons of carbon emissions due to reduced trips into the field, according to a 2016 World Economic Forum white paper.


While these utilities have begun to digitalize operations at every level, the industry as a whole has been slow to adapt.

“There are many reasons why most utilities have yet to really see the value in big data – or any kind of wide-ranging, integrated data and analytics,” Teradata’s Socha said. Chief among them is a tendency to keep operations siloed, which hinders information sharing. Each operation functions relatively independent of the others.

To fully benefit from all of the smart options available to them, Socha said, utilities need to break down the data silos that separate their operations and create a holistic management system.

“Holistic” can be a challenge in decentralized decision-making environments, however. Utilities already have invested in a host of systems optimized for each function, and replacing those with a one-vendor solution is not an attractive option. Instead, utilities should seek a platform that can federate all of their best-of-breed applications while eliminating silos, enabling real-time digital continuity and pushing the right information to users rather than forcing them to hunt it down.

“No matter where they invest first, that investment needs to be in capabilities, tools and a platform that can underpin all their analytics opportunities,” Socha said. “Strategically, the biggest opportunity is to start down the path toward becoming data-driven.”

For information on how to combine connectivity, AI and IIoT into a fully integrated system, visit:

Related resources