An Integrated Lakehouse Architecture with Data Quality Management Framework for Smart Grid Sensor Data

Faculty of Information Technology Ho Chi Minh City University of Technology and Engineering, Ho Chi Minh City, Vietnam This email address is being protected from spambots. You need JavaScript enabled to view it.
Faculty of Information Technology Ho Chi Minh City University of Technology and Engineering, Ho Chi Minh City, Vietnam This email address is being protected from spambots. You need JavaScript enabled to view it.
Faculty of Information Technology Ho Chi Minh City University of Technology and Engineering, Ho Chi Minh City, Vietnam
Faculty of Information Technology Posts and Telecommunication Institute of Technology Ho Chi Minh City, Vietnam This email address is being protected from spambots. You need JavaScript enabled to view it.

^* Corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Reliable smart-grid operation depends on high-quality sensor data for energy monitoring, anomaly detection, and short-term load forecasting. Missing values, invalid measurements, and inconsistent schemas can propagate through energy data pipelines and reduce the reliability of downstream analytics. This study develops an open-source Lakehouse-based pipeline with an embedded metadata-driven Data Quality Management framework for smart-meter data. The proposed Synchronous Quality Gate validates records during the Bronze-to-Silver transition and separates valid readings from invalid records through an auditable quarantine mechanism. Experiments using a public high-frequency household electricity consumption dataset show that the framework improves data consistency while supporting accurate one-step-ahead load forecasting. Random Forest achieved the best forecasting performance (R² = 0.9781 kW, MAE = 0.1508 kW, RMSE = 0.2811 kW), while the DQM-enabled pipeline improved LSTM forecasting accuracy compared with the raw-data baseline. The mean latency overhead of synchronous validation was 13.85%, indicating that data-quality control can be embedded into smart-grid pipelines without excessive processing cost. The findings demonstrate that quality-aware energy data engineering can strengthen the trustworthiness of smart-grid analytics and improve the operational value of high-frequency smart-meter data.