User-oriented verification of automation stations

. Building automation systems are used to meet the increasing requirements for energy efficiency and sustainability of buildings. In such systems, BACnet has established itself as the standard communication protocol for communication between devices from different manufacturers. In order to ensure conformity with the BACnet standard, the devices are certified for conformity. In practice, however, the implementation depth described in the standard profiles is not sufficient for operating a system. As a result, many users demand an extended implementation depth of the BACnet protocol. In most cases, an AMEV certificate is sought, which extends the BACnet objects and their properties in comparison to the standard certification. Despite the BACnet standard certification and the fulfilment of the AMEV test certificate, problems often occur during the operation of building automation systems. These problems can be attributed to the fact that neither the functionality nor the performance of automation stations are determined. In addition, special user requirements are not checked before commissioning a building automation system. For this reason, a model was developed with the help of which a pre-commissioning of the automation station is possible in order to automatically check it for errors and special user requirements before commissioning. Initial tests of automation stations revealed that some user requirements were not met.


Introduction
Energy conservation and energy efficiency play an increasingly important role in today's world. The demand for more energy-efficient systems is also reflected in the EU energy targets. These, for example, call for energy consumption to be reduced by 20% by 2020 compared with 2005 [1]. Building automation systems are increasingly being used to meet the increasing requirements for energy efficiency and sustainability of buildings. These should make the operation of technical systems, such as heating systems or ventilation systems, as energy-efficient as possible.

State of the art
BACnet is the first communication protocol for communication in building automation systems in building automation to be standardized worldwide by ISO. It enables manufacturer-neutral communication between different building automation components and creates an essential prerequisite for transparent and costsaving planning and project planning procedures in building automation [2]. DIN EN ISO 16484-6 defines standard methods for verifying compliance of the implementation of the BACnet protocol with the BACnet standard [3]. In BACnet test laboratories, manufacturers of BACnet products can have their devices verified for conformity to the BACnet standard in accordance with the current DIN EN ISO 16484-6 test standard. BACnet Test Framework (BTF) software is used to perform these tests.
In practice, however, the implementation depth described in the standard profiles is usually not sufficient for operating a plant, because minimum requirements are not defined for object types, properties or read and write permissions. As a result, many users require an extended implementation depth of the BACnet protocol. In most cases, the fulfilment of a BACnet profile defined by the AMEV is required, which extends the BACnet objects and their properties in comparison to the standard certification. The AMEV certificate was introduced as proof of compliance with the aforementioned BACnet profiles. It certifies that the device of a manufacturer supports the BACnet profiles mentioned in the AMEV recommendation [2].
Despite the BACnet standard certification and the fulfilment of the AMEV certificate, building automation systems often encounter a wide variety of problems. For example, it may happen that required optimization programs were not implemented correctly in the automation station or that not enough objects can be created in the station for the automation task to be performed. The reason for such problems is that neither the BACnet standard certification nor the AMEV test certificate makes statements about the functionality and performance of automation stations. Since errors usually  [4] only become apparent during commissioning or during operation, eliminating them requires additional time and costs.
In order to avoid the problems described above in the future, the TH Köln -University of Applied Sciences in Cologne is carrying out a research project in cooperation with the building management of the city of Cologne. This project is developing a procedure to automatically examine automation stations for special user requirements before the stations are put into operation. In the following sections, this procedure is explained and first results are presented.

Methods
In order to extend the BACnet standard certification and the AMEV certificate, a test procedure was developed that can be divided into four clusters. These clusters include  Figure 1. In the following sections, the necessity, implementation and initial results of the four clusters of application-oriented verification of automation stations are discussed in more detail. The focus is on cluster 1 (verification of standard applications by simulated plant engineering). Only the basic functions of the other clusters will be explained.

Cluster 1: Verification of standard applications through simulated plant technology
The operation of technical building equipment systems shows that these systems are often not as automated as required by the user. This is usually due to errors in the programming of optimization programs in the automation station. For this reason, cluster 1 carries out a pre-commissioning of a real automation station in order to check the optimization programs and time programs stored in it for correct function before the station is commissioned. For this purpose, a MATLAB/Simulink simulation model was developed, which is shown in Figure 2.
This model can be divided into two submodels, the standard plant model (blue) and the standard room model (orange). Real heating or ventilation systems can be simulated with the help of the system model. The model can be adapted to the real system to be simulated. System-relevant variables such as mass flows or fluid temperatures as well as heat losses via pipelines are simulated. The room model is used to simulate the behavior of a real room. Similar to the plant model, it can be adapted to the room to be simulated. For example, the room size or properties of the room enclosing components can be adapted to the real situation. When simulating room behavior, variables such as room temperatures, CO2 concentrations and heat losses via room-enclosing components are taken into account. MATLAB/Simulink simulation model [5] Dynamic weather data, which are available as data of a test reference year, are integrated into the simulation model in order to take heat losses to the environment or solar inputs into account.
The automation and control of the simulated system is carried out by the automation station to be tested. For this purpose, the station objects required for the control task are automatically connected to the corresponding variables of the simulation model with the aid of a Python script. Values from the station can be written to the simulation and values from the simulation can be read from the station. A variable to be written is, for example, the manipulated variable of a mixing valve; a variable to be read is the simulated room temperature.
Since the simulation does not take place at real speeds but at increased speeds, the time between the model and the automation station must be synchronized. After each simulated hour, the time in the automation station is overwritten by another Python script.
Once the automation station and simulation model have been successfully connected, the optimization programs required by the user can be tested for correct functionality with the aid of test scenarios. The test scenario shown in Figure 3 is used, for example, to test the functionality of a required outdoor temperature controlled control system. By transferring a weekly changing outdoor temperature to the automation station, it can be checked whether it reacts as required by the user. In the case of test scenarios defined in Figure 3, the set flow temperature calculated by the station should decrease as the outdoor temperature rises.
After running the simulation in MATLAB/Simulink, the hourly values of all simulated and relevant variables are exported to an Excel file. This file is used to evaluate, on the basis of previously defined criteria, whether the program to be tested has been implemented according to user requirements and where errors occurred.  Table 1 contains the results of a check of the optimization program of an outdoor temperature controlled control. For the evaluation of the simulation results, the energy guidelines of the city of Cologne [7] as well as their appendix Building Automation [6] were used as representatives. They contain a description of how an outside temperature controlled control for a heating system in the buildings of the city of Cologne can be implemented in the automation station used. By comparing the user requirements and the simulation results, a statement can be made about which functions have been implemented correctly and where there are problems in the implementation of the optimization program. It can be seen from the table that the correct target flow temperature is determined during daytime operation, but is not reduced by the required 20K during nighttime operation.

Outddo temperature [°C]
Week test scenario for an outdoor temperatureregulated control system To obtain more detailed information about which functions are not correctly implemented, the automatically generated diagram in Figure 4 can also be used. It shows a daily course of the simulated variables.
The red area shows that at least one of the required functions was not performed during night operation. The green area shows that all required functions were implemented correctly for daytime operation.

Cluster 2: Advanced Interoperability Testing
The BACnet standard certification is carried out with the aid of DIN EN ISO 16484, which specifies, for example, which properties of a converted object must be readable or writable. A certification according to DIN EN ISO 16484-5 includes the control of the correct implementation of each function. However, no special user requirements are checked here. One requirement, for example, could be that the user requires a minimum character length for specific properties, as shown in table 2. The reason for this is that, for example, the system identification key contains 24 characters and a character length of 16 characters for the property object name of the object Analog Input would not be sufficient for this case.
For this reason, cluster 2 is used in application-oriented verification to perform an extended check of the requirements for BACnet objects. For this purpose, test profiles are first defined as test criteria according to the user requirements. Subsequently, the previously defined criteria are checked with the help of the BACnet test framework. The test results can be taken from an Excel spreadsheet. A section of such a table can be found in Table 2. This table shows that the properties of the Analog Input object were checked as to whether the required access rights and character lengths were observed. It is apparent that not all requirements required by the user were met and that the tested station is therefore not suitable for use in accordance with the requirements.

Cluster 3: Communication features
Cluster 3 handles the investigation of the communication features of the automation station to be tested. The necessity arises from the fact that in practice it often turns out that components are physically present, but cannot be reached via the network. Therefore, in cluster 3 of the application-oriented verification, a check is made whether the MS/TP networks were set up correctly, whether all components are physically present and whether the network is accessible. Furthermore, the behavior of the station in an overloaded network is analyzed in this cluster. The network load is caused by faulty BACnet packets or a broadcast storm, among other things. The station is checked to see how it behaves in an overloaded network and whether it is still reachable at all in such a case, whether it delivers an error message or whether it completely stops communication.

Cluster 4: Limits of the technical equipment
A test of the performance of the automation station is also not included in the BACnet standard certification. For this reason, cluster 4 checks the limits of the technical equipment of the automation station to be tested. The test is carried out using the BACnet test framework. Here, for example, it is automatically checked how many objects can be instantiated in an automation station (e.g., trend objects). In addition, a check is carried out to see how many values a trend object can contain. With the help of this check, users can quickly see whether the performance of the automation station is sufficient for their application.

Summary and conclusion
In the previous sections, a procedure was presented that extends the BACnet standard certification. The necessity of such a procedure arises from the fact that only general and not user-specific requirements are tested in standard certifications. As a result, problems or errors often arise during the operation of technical building automation systems. These are often due to errors in the programming of building automation systems. Subsequent troubleshooting usually results in additional effort and costs.
With the help of application-oriented verification of automation systems, additional effort and costs can be avoided. This is because the real automation station to be used in the plant automation is subjected to a technical test in the real building before it is put into operation. This test can be adapted to different user requirements. Errors that would otherwise only become apparent during operation can thus be detected and eliminated ahead of time. As a result, the automation station functions as desired by the customer from the time of commissioning and errors and problems during plant operation can be reduced.