Demonstrating Challenges in Predicting Occupant Temperature Setpoints: A Comparative Evaluation of Traditional and Transformer-Based Machine Learning Models

Department of the Built Environment, College of Design and Engineering, National University of Singapore, Singapore

Abstract

Heating, Ventilation, and Air Conditioning system stands as an essential component of modern building infrastructure, which is widely used to regulate indoor thermal conditions. Among the control parameters, the temperature setpoint stands out as one of the key variables in its operation, determining when and how heating or cooling is delivered. Accurately predicting occupant preferred temperature setpoints has the potential to improve HVAC energy efficiency and achieve comfortable indoor conditions by ensuring that heating and cooling are delivered precisely when and where they are needed. This can be achieved through the use of predictive machine learning (ML) algorithms, which learn the relationship between occupant preferences and environmental or physiological parameters. However, the application of ML algorithms in modeling thermal preferences remains limited due to consistently low prediction accuracy. This study identifies three key challenges underlying these limitations: (1) Difficulty predicting under unseen environmental conditions, (2) Lack of adaptability to changing preferences over time, and (3) Poor performance with limited training data. This paper aims to systematically demonstrate these key challenges and guide the selection or development of more effective thermal preference models for real-world HVAC operation. In this study, ten widely-used ML algorithms are selected for thermal preference modeling, and their predictive performance is compared against a novel transformer-based architecture, which we previously developed to address these three challenges. We utilize the data derived from ECOBEE Donate Your database for performance evaluation, which comprises over 100,000 thermostat users in North America. Results show that all models suffer a 20-30% drop in accuracy when predicting for previously unseen data. However, the transformer-based architecture, which leverages pre-trained models to capture diverse thermal preference patterns, outperforms the best widely used algorithm (R² of 0.63 vs. 0.49). Additionally, we demonstrate that the performance of ML algorithms often stagnates or declines with the increase of the training set, and how employing a transformer mechanism helps to continuously improve the predictive performance. Furthermore, with sparse training data, ML architecture leveraging pre-trained models achieves an R² of 0.67 by the 25th data point, outperforming the best widely used machine learning algorithm, which reaches an R² of 0.62.