Recent technological advancements have led to a deluge of data originating from several domains, for instance, smart cities, the internet, scientific sensors, and social networks. Big data is a term that was incepted to cope with the ever-rising volume, velocity and variety of data. Big data are becoming the focus of many engineering and scientific domains. Big data systems include a number of tools and methods to acquire, store, and process data while leveraging the parallel processing power to undertake complex transformations as well as analysis.
The design and implementation of big data framework systems for a particular application is not a clear-cut mission. This is due to the fact that data is in multiple, autonomous and heterogeneous sources with complex and evolving relationships, and steadily grows. Above all, the rise of big data applications in cases where data collection has grown is beyond the capacity of the existing software and hardware platforms to manage, store, and process within an acceptable time.
Many utilities are adopting smart grid technology to form the basis of long range planning in a bid to improve power supply reliability, incorporate distributed generation resources, use power farms effectively, and enable users to engage in controlling how they use energy. To satisfy this, smart meters are being employed in many utilities as an initial step. This incorporation of smart meters results in significant increase in data which can be overwhelming if not managed appropriately. If this data is managed effectively, it can lead to a better understanding of customer behavior and help in promoting the stability of the smart grid.
Amr Munshi and Yasser Mohamed at The University of Alberta presented a new framework that could be used as a start for innovative research and take smart grids to the next level. They presented an implementation of the framework on a secure cloud-based platform. The framework was also applied on two scenarios in a bid to visualize the energy, for a single-house and a smart grid containing 6000 smart meters. Their research work is published in Electric Power Systems Research.
The authors implemented the framework on a secure IaaS cloud-based platform and presented relevant configurations as well as source codes. In addition, the authors presented interfaces between a number of component combinations that were able to communicate together. The framework was hosted on an IaaS Google cloud platform that presented an improved accessibility, and scalable infrastructure. They authors established a secure link between the framework’s cluster nodes through the Secure Shell protocol.
The implementation of the framework was applied to two scenarios: single-house with micro generators and a real smart metering electricity behavior data set for 6000 houses and businesses. However, the authors realized that the practical implementation of the framework together with the required configuration and coding, and application, could be helpful in the development of big data frameworks for other disciplines that could be profitable to businesses, promote the development of science and technology and enable sound decision making for government sectors.
Amr A. Munshi and Yasser A.-R. I. Mohamed. Big data framework for analytics in smart grids. Electric Power Systems Research, volume 151 (2017), pages 369–380.Go To Electric Power Systems Research