AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Building Simulation Article
Article Link
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Research Article

A reinforcement learning approach for thermostat setpoint preference learning

Hussein Elehwany1( ), Mohamed Ouf2, Burak Gunay1, Nunzio Cotrufo3, Jean-Simon Venne3
Department of Environmental Engineering, Carleton University, Ottawa, Canada
Department of Building, Civil, and Environmental Engineering, Concordia University, Montreal, Canada
BrainBox AI Inc., Montreal, Canada
Show Author Information

Abstract

Occupant-centric controls (OCC) is an indoor climate control approach whereby occupant feedback is used in the sequence of operation of building energy systems. While OCC has been used in a wide range of building applications, an OCC category that has received considerable research interest is learning occupants’ thermal preferences through their thermostat interactions and adapting temperature setpoints accordingly. Many recent studies used reinforcement learning (RL) as an agent for OCC to optimize energy use and occupant comfort. These studies depended on predicted mean vote (PMV) models or constant comfort ranges to represent comfort, while only few of them used thermostat interactions. This paper addresses this gap by introducing a new off-policy reinforcement learning (RL) algorithm that imitates the occupant behaviour by utilizing unsolicited occupant thermostat overrides. The algorithm is tested with a number of synthetically generated occupant behaviour models implemented via the Python API of EnergyPlus. The simulation results indicate that the RL algorithm could rapidly learn preferences for all tested occupant behaviour scenarios with minimal exploration events. While substantial energy savings were observed with most occupant scenarios, the impact on the energy savings varied depending on occupants’ preferences and thermostat use behaviour stochasticity.

Keywords

reinforcement learningpreference learningoccupant-centric controlssmart thermostatsoff-policy learning

References

 
ASHRAE (2020). Standard 55-2020—Thermal Environmental Conditions for Human Occupancy. Atlanta, GA, USA: American Society of Heating, Refrigeration, and Air-Conditioning Engineers.
 
Awalgaonkar N, Bilionis I, Liu X, et al. (2019). Learning personalized thermal preferences via Bayesian active learning with unimodality constraints. arXiv: 1903.09094.
 
Barrett E, Linder S (2015). Autonomous HVAC control, A reinforcement learning approach. In: Albert et al. (eds), Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science, Vol 9286. Cham: Springer.
Crossref
 

Borgeson S, Brager G (2011). Comfort standards and variations in exceedance for mixed-mode buildings. Building Research and Information, 39: 118–133.
Crossref Google Scholar
 

Cheng Z, Zhao Q, Wang F, et al. (2016). Satisfaction based Q-learning for integrated lighting and blind control. Energy and Buildings, 127: 43–55.
Crossref Google Scholar
 

Daum D, Haldi F, Morel N (2011). A personalized measure of thermal comfort for building controls. Building and Environment, 46: 3–11.
Crossref Google Scholar
 
DOE (2011). Buildings Energy Data Book. US Department of Energy.
 

Fanger PO (1970). Thermal Comfort: Analysis and Applications in Environmental Engineering. Copenhagen: Danish Technical Press.
 
Feldmeier M, Paradiso JA (2010). Personalized HVAC control system. In: Proceedings of 2010 Internet of Things (IOT), Tokyo, Japan.
Crossref
 
Graesser L, Keng WL (2019). Foundations of Deep Reinforcement Learning: Theory and Practice in Python. Boston, MA, USA: Addison-Wesley Professional.
 

Guenther J, Sawodny O (2019). Feature selection and Gaussian Process regression for personalized thermal comfort prediction. Building and Environment, 148: 448–458.
Crossref Google Scholar
 

Gunay HB, O’Brien W, Beausoleil-Morrison I (2013). A critical review of observation studies, modeling, and simulation of adaptive occupant behaviors in offices. Building and Environment, 70: 31–47.
Crossref Google Scholar
 

Gunay HB, O’Brien W, Beausoleil-Morrison I, et al. (2016). Implementation of an adaptive occupancy and building learning temperature setback algorithm. ASHRAE Transactions, 122(1): 179–192.
Google Scholar
 

Gunay HB, O’Brien W, Beausoleil-Morrison I, et al. (2018). Development and implementation of a thermostat learning algorithm. Science and Technology for the Built Environment, 24: 43–56.
Crossref Google Scholar
 
Haarnoja T, Tang H, Abbeel P, et al. (2017). Reinforcement learning with deep energy-based policies. arXiv: 1702.08165.
 

Haldi F, Robinson D (2008). On the behaviour and adaptation of office occupants. Building and Environment, 43: 2163–2177.
Crossref Google Scholar
 

Han M, May R, Zhang X, et al. (2019). A review of reinforcement learning methodologies for controlling occupant comfort in buildings. Sustainable Cities and Society, 51: 101748.
Crossref Google Scholar
 

Han M, May R, Zhang X, et al. (2020). A novel reinforcement learning method for improving occupant comfort via window opening and closing. Sustainable Cities and Society, 61: 102247.
Crossref Google Scholar
 

Heidari A, Maréchal F, Khovalyg D (2022a). An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach. Applied Energy, 312: 118833.
Crossref Google Scholar
 

Heidari A, Maréchal F, Khovalyg D (2022b). Reinforcement learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use. Applied Energy, 318: 119206.
Crossref Google Scholar
 

Hoes P, Hensen JLM, Loomans MGLC, et al. (2009). User behavior in whole building simulation. Energy and Buildings, 41: 295–302.
Crossref Google Scholar
 

Huchuk B, Sanner S, O’Brien W (2021). Development and evaluation of data-driven controls for residential smart thermostats. Energy and Buildings, 249: 111201.
Crossref Google Scholar
 
IEA (2012). World Energy Balances for 2012. International Energy Agency
 

Jayathissa P, Quintana M, Abdelrahman M, et al. (2020). Humans-as-a-sensor for buildings—Intensive longitudinal indoor comfort models. Buildings, 10: 174.
Crossref Google Scholar
 

Jung W, Jazizadeh F (2019). Comparative assessment of HVAC control strategies using personal thermal comfort and sensitivity models. Building and Environment, 158: 104–119.
Crossref Google Scholar
 

Kazmi H, Mehmood F, Lodeweyckx S, et al. (2018). Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems. Energy, 144: 159–168.
Crossref Google Scholar
 

Konis K, Annavaram M (2017). The Occupant Mobile Gateway: A participatory sensing and machine-learning approach for occupant-aware energy management. Building and Environment, 118: 1–13.
Crossref Google Scholar
 

Lai D, Chen C (2019). Comparison of the linear regression, multinomial logit, and ordered probability models for predicting the distribution of thermal sensation. Energy and Buildings, 188–189: 269–277.
Crossref Google Scholar
 

Lee S, Karava P, Tzempelikos A, et al. (2019). Inference of thermal preference profiles for personalized thermal environments with actual building occupants. Building and Environment, 148: 714–729.
Crossref Google Scholar
 

Li D, Menassa CC, Kamat VR (2017). Personalized human comfort in indoor building environments under diverse conditioning modes. Building and Environment, 126: 304–317.
Crossref Google Scholar
 

Li W, Zhang J, Zhao T (2019). Indoor thermal environment optimal control for thermal comfort and energy saving based on online monitoring of thermal sensation. Energy and Buildings, 197: 57–67.
Crossref Google Scholar
 

Lork C, Li W, Qin Y, et al. (2020). An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management. Applied Energy, 276: 115426.
Crossref Google Scholar
 

Mason K, Grijalva S (2019). A review of reinforcement learning for autonomous building energy management. Computers & Electrical Engineering, 78: 300–312.
Crossref Google Scholar
 
McKee E, Du Y, Li F, et al. (2020). Deep reinforcement learning for residential HVAC control with consideration of human occupancy. In: Proceedings of 2020 IEEE Power & Energy Society General Meeting (PESGM), Montreal, Canada.
Crossref
 
National Research Council of Canada (2017). National Energy Code of Canada for Buildings 2017.
 

O’Brien W, Wagner A, Schweiker M, et al. (2020). Introducing IEA EBC annex 79: Key challenges and opportunities in the field of occupant-centric building design and operation. Building and Environment, 178: 106738.
Crossref Google Scholar
 

Odonkor P, Lewis K (2019). Automated design of energy efficient control strategies for building clusters using reinforcement learning. Journal of Mechanical Design, 141: 021704.
Crossref Google Scholar
 

Ouf MM, Park JY, Gunay HB (2021). A simulation-based method to investigate occupant-centric controls. Building Simulation, 14: 1017–1030.
Crossref Google Scholar
 

Park JY, Dougherty T, Fritz H, et al. (2019a). LightLearn: An adaptive and occupant centered controller for lighting based on reinforcement learning. Building and Environment, 147: 397–414.
Crossref Google Scholar
 

Park JY, Ouf MM, Gunay B, et al. (2019b). A critical review of field implementations of occupant-centric building controls. Building and Environment, 165: 106351.
Crossref Google Scholar
 
Park JY, Nagy Z (2020). HVACLearn: A reinforcement learning based occupant-centric control for thermostat set-points. In: Proceedings of the 11th ACM International Conference on Future Energy Systems.
Crossref
 

Pazhoohesh M, Zhang C (2018). A satisfaction-range approach for achieving thermal comfort level in a shared office. Building and Environment, 142: 312–326.
Crossref Google Scholar
 

Peng Y, Nagy Z, Schlüter A (2019). Temperature-preference learning with neural networks for occupant-centric building indoor climate controls. Building and Environment, 154: 296–308.
Crossref Google Scholar
 
Sato K, Samejima M, Akiyoshi M, et al. (2012). A scheduling method of air conditioner operation using workers daily action plan towards energy saving and comfort at office. In: Proceedings of 2012 IEEE 17th International Conference on Emerging Technologies & Factory Automation (ETFA 2012).
Crossref
 
Shi Z (2018). Building operation specialist: A probabilistic distributed fault detection, diagnostics and evaluation framework for building systems. PhD Thesis, Carleton University, Canada.
 

Sutton RS, Barto AG (2018). Reinforcement Learning: An Introduction. Cambridge, MA, USA: MIT press.
 
Tartarini F, Frei M, Schiavon S, et al. (2022). Cozie Apple: An iOS mobile and smartwatch application for environmental quality satisfaction and physiological data collection. arXiv: 2210.13977.
Crossref
 

Valladares W, Galindo M, Gutiérrez J, et al. (2019). Energy optimization associated with thermal comfort and indoor air control via a deep reinforcement learning algorithm. Building and Environment, 155: 105–117.
Crossref Google Scholar
 

Vázquez-Canteli J, Kämpf J, Nagy Z (2017). Balancing comfort and energy consumption of a heat pump using batch reinforcement learning with fitted Q-iteration. Energy Procedia, 122: 415–420.
Crossref Google Scholar
 

Vázquez-Canteli JR, Nagy Z (2019). Reinforcement learning for demand response: A review of algorithms and modeling techniques. Applied Energy, 235: 1072–1089.
Crossref Google Scholar
 

Vázquez-Canteli JR, Ulyanin S, Kämpf J, et al. (2019). Fusing TensorFlow with building energy simulation for intelligent energy management in smart cities. Sustainable Cities and Society, 45: 243–257.
Crossref Google Scholar
 

Wang Z, Hong T (2020). Reinforcement learning for building controls: The opportunities and challenges. Applied Energy, 269: 115036.
Crossref Google Scholar
 

Yoshino H, Hong T, Nord N (2017). IEA EBC annex 53: Total energy use in buildings—Analysis and evaluation methods. Energy and Buildings, 152: 124–136.
Crossref Google Scholar
 
Zhang Z, Chong A, Pan Y, et al. (2018). A deep reinforcement learning approach to using whole building energy model for HVAC optimal control. In: Proceedings of 2018 Building Performance Modeling Conference and SimBuild co-organized by ASHRAE and IBPSA-USA.
 

Zhang H, Tzempelikos A (2021). Thermal preference-based control studies: Review and detailed classification. Science and Technology for the Built Environment, 27: 1031–1039.
Crossref Google Scholar
 

Zhang H, Tzempelikos A, Liu X, et al. (2023). The impact of personal preference-based thermal control on energy use and thermal comfort: Field implementation. Energy and Buildings, 284: 112848.
Crossref Google Scholar
 

Zhou S, Hu Z, Gu W, et al. (2019). Artificial intelligence based smart energy community management: A reinforcement learning approach. CSEE Journal of Power and Energy Systems, 5: 1–10.
Crossref Google Scholar
Building Simulation
Volume 17 Issue 1,
January 2024
Pages 131-146
DOI: 10.1007/s12273-023-1056-7
Cite this article:
Elehwany H, Ouf M, Gunay B, et al. A reinforcement learning approach for thermostat setpoint preference learning. Building Simulation, 2024, 17(1): 131-146. https://doi.org/10.1007/s12273-023-1056-7

298

Views

2

Crossref

6

Web of Science

6

Scopus

0

CSCD

Altmetrics
Received: 22 February 2023
Revised: 12 June 2023
Accepted: 22 June 2023
Published: 01 August 2023
© Tsinghua University Press 2023
Return