Efficient Storage of Big Weather Data for Renewable Power Production Prediction
Many countries have set aggressive renewable integration targets. Achieving these targets requires fundamental changes to the management of the electric grid. One of the main challenges of renewable integration is the capability of forecasting the power production from intermittent renewables, in particular wind and solar. The more accurate these predictions become, the more efficiently can dispatchable resources be dispatched and the more efficiently can renewable power generation companies take part in wholesale electricity markets.
Both wind and solar power, the two most rapidly expanding types of renewable power, are driven by local weather conditions, in particular wind speed, temperature, and solar irradiance. The ability to forecast these weather data, together with accurate generator models, thus enables wind and solar power production forecasts. Statistical forecasting methods can contribute to short-term forecasting, which plays a particularly important role for preventing negative impacts of large-scale renewables integration in power systems.
The goal of this thesis is to develop efficient storage systems for big weather data using the Hadoop stack , in particular HBase , and a fully distributed cluster implementation. The database will be populated with actual weather data from publicly available sources, in particular using a web service offered by the Norwegian Meteorological Institute . The functionality of the big data storage should be demonstrated by a number of standard queries, such as relevant aggregations and visualizations. If time admits, the data can be leveraged to test innovative machine learning based forecasting techniques.
Description of the tasks
- Set up distributed Hadoop and HBase on cluster
- Design database
- Populate database with publicly available data
- Specify test queries and measure performance
- Interest in big data management with relecant applications
- Solid Java coding skills
This project will be carried out collaboratively with Technical University of Munich (TUM), Germany with professor Hans-Arno Jacobsen from TUM as co-supervisor. The master student will be given the opportunity to visit and work closely with researchers and students from professor Jacobsen´s group on Energy Informatics. A visit can last for up to one semester and will be funded by a new project on Energy Informatics at IFI. The teaching and working language at TUM will be english. No proficiency in german is required.
- Apache Hadoop Video Tutorial. http://www.youtube.com/channel/UCOKFIXDVK-ntuaFMb8IPqcw
- Lars George. HBase – The Definite Guide. O’Reilly. 2011.
- api.met.no. http://api.met.no/#english
- E. Lorenz, T. Scheidsteger, J. Hurka, D. Heinemann, C. Kurz. Regional PV Power Prediction for Improved Grid Integration. Progress in Photovoltaics: Research and Applications (19), pp. 757-771. 2011.
- J. Huang et al. Wind Energy Forecasting: A Review of State-of-the-Art and Recommendations for Better Forecasts. California Renewable Energy Forecasting, Resource Data and Mapping. http://uc-ciee.org/downloads/appendixB.pdf