Fusing visual and sensor data with multimodal transformers for indoor environmental quality assessment in educational settings

¹ Department of Civil, Architectural and Environmental Engineering, College of Engineering, Drexel University, 3141 Chestnut St., Philadelphia, PA 19104, USA
² Department of Electrical & Computer Engineering, Klesse College of Engineering and Integrated Design, The University of Texas at San Antonio UTSA, 501 W. César E. Chávez Blvd, San Antonio, TX 78207, USA
³ Department of Architecture, Design & Urbanism, Antoinette Westphal College of Media Arts and Design, Drexel University, 3501 Market St., Philadelphia, PA 19104, USA

^* Corresponding author: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

Indoor environmental quality (IEQ) plays a central role in the health, comfort, and cognitive performance of building occupants, particularly in educational environments where students spend extended periods of time. Prior work has shown that elevated carbon dioxide (CO₂) concentrations, increased total volatile organic compounds (TVOCs), particulate matter (PM), and suboptimal hygrothermal conditions negatively affect attention, memory, and decision-making among students. As educational buildings move toward more occupant-centric and energy-efficient operation, artificial intelligence (AI) tools offer new opportunities for real-time IEQ assessment. However, existing approaches often rely solely on sensor data and lack the contextual awareness needed to interpret environmental shifts that are influenced by occupancy, activity patterns, or space configuration. This study presents a multimodal transformer (MulT) model that fuses synchronized environmental sensor data with time-lapse RGB images to assess seven IEQ indicators: dry-bulb temperature (T), relative humidity (RH), CO₂ concentrations, TVOCs, and PM (PM₁, PM_2.5, PM₁₀). Data were collected at 5-minute intervals from four representative educational spaces, namely, a conference room, a hybrid laboratory, and two classrooms, over 7 days each from March to May 2025. The model combines a convolutional neural network (CNN) for local visual features, a Vision Transformer (ViT) for global spatial reasoning, and a cross-modal fusion transformer to jointly estimate all IEQ variables. Results show significantly strong convergence and robust generalization. On a dedicated test set containing unseen paired samples from all environments, the model achieved a test mean squared error (MSE) of 85.05 and a mean absolute error (MAE) of 3.64, which demonstrates that the learned multimodal embeddings transfer well to new images and sensor combinations, preserving accuracy outside the training distribution. The MAE of ~3.6 across seven heterogeneous IEQ variables further highlights the model's strength, given that the targets span different physical scales. The approach effectively captures both environmental dynamics and visual cues such as occupancy level, equipment use, and lighting conditions, improving assessment stability in diverse indoor scenarios. The findings highlight the potential of MulT-based architectures for real-time IEQ monitoring in educational settings and point toward broader applications in automated building diagnostics, early-warning systems, and adaptive HVAC control strategies.