Evaluating the Practicability of the new Urban Climate Model PALM-4U using a Living-Lab Approach

. Numerical urban climate models have the potential to be commonly used tools in urban development processes in practice. With integrated modules for building and envelope simulation these complex models allow the assessment of measures in the building sector on the urban climate, e.g. for mitigation of urban heat island effects or preparing for the effects of climate change. However, the currently existing models do not fulfil the requirements that arise in the field of urban planning, as they lack for example functionality, user-friendliness, are hard to integrate in the municipalities’ technical equipment or are not freely available. The German research and development project Urban Climate under Change (2016-2019) developed and validated a new innovative urban climate model called PALM-4U. Aim of the research was to create a model that meets the requirements of users in science as well as practitioners in engineering offices and urban administrations. Therefore, technical features and operational functionalities which the model has to meet to support users in their daily work have been assessed in a first project phase. In total more than 200 requirements were collected which are summed up in the so called “User and Requirements Catalogue”. They served as the basis for testing and evaluating the model’s real-world applicability. To ensure that these complex requirements are met, the whole project follows a transdisciplinary approach integrating science (model development and data assimilation) and practice (user requirements, testing and evaluation) applying a living lab approach: Stakeholders from participating cities and companies took part in on-site workshops, introducing the model with practical use-cases. Afterwards, participants were given tasks covering different features of the model’s applications, which they tested in personal use. The model fulfils the majority of the tested requirements and the users appreciated the model’s concept and functionality. But further development is necessary to provide the practitioners a tool that is applicable in their daily work: Preparation of input data, a user-friendly graphical user-interface, enhanced interfaces to other software and planning tools, use cases that were prepared from experts as well as guidelines and tools for result assessment and interpretation were main suggestions for improvement..


Introduction
Co-development has become a buzz word over the past years -stakeholders should be involved in everything. But how can successful stakeholder engagement be implemented? We will provide insights from the large German research and development project Urban Climate under Change (short: [UC] 2 ), which was funded by the German Ministry of Education and Research (BMBF) since 2016 for a first period of three years.
[UC]² aims to develop, evaluate and apply an innovative, highly efficient and practice-oriented urban climate model for entire cities. The new model is called PALM-4U (Parallelized Large-Eddy Simulation Model for Urban Applications, read: PALM for you). Chapter 2 gives an overview on the model PALM-4U, its capabilities and typical fields of application.
As shown in Fig. 1, the research project [UC]² is organized into three modules, which take over specific tasks and collaborate to validate the new model and ensure its practicability:  Module A -MOSAIK project: Development of PALM-4U [5],  Module B -3DO project: Acquisition of observational data for model evaluation [6],  Module C -UseUClim and KliMoPrax projects: Review of PALM-4U´s practicability and userfriendliness [7]. This paper focusses on evaluating the practicability, which was the main task in module C, which itself was made up of two consortia: KliMoPrax and UseUClim. Both focus on different target groups and follow different procedures and processes. Together with the user and requirements catalogue and the final evaluation report, they jointly develop central Module C products. These are described in chapter 3. To ensure PALM-4U´s practicability and userfriendliness, partners from urban planning practice are involved into the model development as well as into its testing and evaluation. For this purpose, UseUClim's practice partners consist of employees from municipal environmental and urban planning offices (Cities of Chemnitz, Dresden and Leipzig, all located in Germany) and an architecture and engineering consulting company active in urban planning (Sweco GmbH). To structure this collaboration, UseUClim applies the living lab concept and, thus, provides a platform that brings together all relevant stakeholders to allow a systematic user-science interaction, as described in chapter 3.

Urban Climate Model PALM-4U
The new urban climate model PALM-4U is based on the large-eddy simulation (LES) model PALM [1] [2]. Thus, it can directly resolve turbulence both time-wise as well as spatially. This allows a more accurate representation of mixing processes in the urban canopy as well as shortterm phenomena as for example gusts, which is in particular important for transport processes and the assessment of peak loads.
Within the [UC²]-project the original PALM-model was extended with additional components that are relevant for urban climatology: A complex 3Dgeometry of buildings, vegetation and terrain, representations of urban surfaces and vegetation, detailed radiative transfer in the urban canopy layer, a chemistry module for transport and conversion of reactive species and tools for biometeorological analysis [12].
Due to these capabilities PALM-4U is applicable for a wide range of scientific and real-world problems related to urban development and planning, air pollution control, human comfort and for adaptation to the regional consequences of global climate change [3]. Typical fields of application are:  PALM-4U is using the LES approach for simulating the airflow. As already mentioned this technique is able to resolve small-scale effects and turbulence and thus is able to represent gusts and peak wind loads. The urban built environment provides many obstacles and canalization effects. Thus, locally uncomfortable and possibly dangerous wind speeds can occur, which can be identified and quantified with the help of such detailed simulations. The effects of wind on buildings and their facades, on pedestrians and on other elements in urban areas, e.g. trees, can be assessed,  PALM-4U includes the physical models of relevant contributors to the urban heat island effect. Thus it can be used to assess the microclimate in the urban canopy layer. The effectiveness of measures to reduce the urban heat island, e.g. the integration of green areas or the designation of cold air pathways, can thus be quantified,  Included analysis tools for biometeorological indices [13] such as the physiological equivalent temperature (PET) or the universal thermal climate index (UTCI) are a useful metric to assess heat stress for the population in cities, as these indices consider the effect of moisture and wind as well as long-and shortwave radiation in addition to air temperature,  PALM-4U comes with an aerosol module and an interface to the kinetic-preprocessor (KPP) for chemical processes in the atmosphere [12]. Multiple schemes and reactions are available. Also parametrizations for typical emissions, e.g. from traffic, are included. Due to the LES-approach the mixing as well as vertical and horizontal distribution of emissions due to the airflow can be represented. This allows a more accurate assessment of trace gas concentrations in a simulation domain and thus a more accurate evaluation of potential risks due to these gases. The model PALM-4U is highly scalable and allows building-resolving simulations from small scales like single buildings up to city districts and whole cities like Berlin [4].
PALM-4U is typically controlled with a script-based approach. In addition, a web-based graphical user interface (short: GUI) for PALM-4U was developed in [UC]². The GUI was intended to be a simplified approach to create model setups, execute simulations and view the results. Thus, not the full functionality of PALM-4U was accessible with the GUI. Only a scriptbased approach for input data preparation is currently offered.

Living lab approach
For the evaluation and improvement of the pracicability of PALM-4U, UseUClim followed a Living-Lab approach in which the project's practice partners are directly involved in the development process and apply the model themselves. Therefore, the project was structured in three consecutive phases [8]: 1) Exploration: Identification and assessment of relevant stakeholders and their requirements for a practical urban climate model.
2) Experimentation: Testing the model prototype together with practice partners in in-house trainings to evaluate user demands and identify potentials for future development.
3) Evaluation: Based on the feedback of the practice partners as well as the experiences of the network partners the practical suitability of PALM-4U was assessed.
The project structure is summarized in Fig. 2.

Exploration
The first project phase developed a common understanding of which requirements a practical and user-friendly urban climate model should fulfill and who is going to use it or its results. Several methods were applied to identify potential users and their requirements [8]:  Desk research on relevant literature, standards and experiences of previous projects were used to identify common requirements and typical stakeholders of urban climate models. This also ensured to include scientific requirements,  Several workshops with German municipalities and UseUClim's practice partners gave further insight in their workflow and helped to identify further requirements that are relevant for users from practice,  A stakeholder-analysis identified potential usergroups or organizations that are currently using urban climate models or their results. In addition, potential future users were identified. All stakeholders were grouped into categories that share common interest or use cases. These are: Municipalities, economy, science and civil society. Relevant actors of these user groups were identified and invited to share their experiences in an onlinesurvey,  To analyze the requirements UseUClim developed a standardized online-survey and shared it with its practice partners and other relevant stakeholders.
The results of this survey were also used to rate the relevance of requirements. Altogether, 108 participants gave around 11.100 answers and comments [8],  A newly developed simulation software should be compatible with existing IT-and data-infrastructure to facilitate its introduction in practice. Therefore, UseUClim analyzed and identified software interfaces to identify practical requirements on handling of data input and output. Typical open data interfaces and relevant software tools were identified, like for example GIS, CAD or 3D visualization.
In addition, specialized simulation tools that can be used for continuative analyses were identified, like for example traffic or building simulations.
The results of these various sources were collected and translated into a list of precise requirements. Each requirement comes with an acceptance criterion that describes how this requirement can be fulfilled. This facilitated the review-process, described in chapter 3.3, significantly.
All requirements were systematically synthesized in a two-part user and requirements catalogue (short: URC) together with the partner project KliMoPrax. Thus, the URC is an essential element to review PALM-4U´s practicability and user-friendliness. It not only provides a basis for subsequent project steps, but is also seen as a communication tool to translate demands that arise from urban development and planning practice to the model development [7]. In consultation with the modules A and B, its structure as well as each requirement´s final wording was determined. Thanks to this approach, it explicitly reflects the perspective from practice and, simultaneously, provides optimal input for the model developers and the collection of observational data. Modules A and B also rated if the requirements can be implemented into PALM-4U. Due to the designated project plan and technical limitations, not all requirements from practice could be implemented in this first project phase of [UC]².
The collection and discussion of requirements in this interdisciplinary project lasted a significant time, but was essential to develop a common understanding of what a "practical model" must include and helped to foster the exchange between science and practice. The URC consists of two parts, namely a compilation of requirements in a tabular format, and an explanatory document with further information on the requirements and stakeholder analysis. In total 240 user requirements were documented and structured into five categories, see Fig. 3: 1) technical infrastructure and system prerequisites, 2) functionalities and scientific requirements, 3) input data, 4) output data, and 5) graphical user interface.
A preliminary version of the UCR was placed at the disposal for all project and practice partners. A final version was published at the end of the project, which includes requirements that were identified during the experimentation and evaluation phases [9] [10].

Experimentation
During the experimentation phase, both the scientific and the practice partners from UseUClim applied PALM-4U to simulate typical planning scenarios. The experiences from both perspectives were used to identify potentials for further model development and to rate the model's practicability.
Since PALM-4U and its graphical user interface (short: GUI) were developed in parallel, only a prototype of the model could be used in the experimentation phase.
The practice partners in UseUClim were trained in the use of the model in two training phases. They were able to work independently with PALM-4U for several months after each training phase. On demand of the practice partners, the model was operated exclusively via the GUI. Thus, the evaluation of the practical suitability of PALM-4U in UseUClim focused mainly on the GUI.

Internal Test Applications
Internal test applications were initially carried out to prepare and develop the user training courses. Both the application of the model via the GUI and the scriptbased application were tested and suggestions for improvement were communicated to the model and GUI developers. After completion of the second training phase, additional internal test applications were carried out to evaluate selected requirements that could be answered without direct user feedback or whose testing required more in-depth tests that could not be carried out in the training courses. These primarily include requirements for special model capabilities and functionalities.

User application: Phase I
In training phase I, the focus was on testing and improving the existing prototypical model basis, especially the first GUI-prototype. Each practice partner was introduced to PALM-4U with on-site teachings. There, handling the model was taught using ready-made practical examples and application scenarios. These used a real-world example of a district of 1 km² around the Ernst-Reuter-Platz in Berlin and ficitional urban planning scenarios in this area. Fig. 4 shows simulation results of the near-surface wind field from an exemplary simulation that was part of the first training course. All input data was prepared before the training course, as no practical tools for creating the input data were available at this project phase. As the GUI didn't provide functionality to view the simulation results, an additional software for data visualization was introduced to the practice partners: Paraview [11], which was identified in the interface analysis during the "Exploration"-phase. After the training the practice partners were able to use PALM-4U for several months in self application. This phase was concluded with a joint workshop and each practice partner submitted a report on the experiences. Suggestions for improvements of the model were regularly communicated to the developers.

User application: Phase II
In training phase II the focus was on the evaluation of the practical suitability of the current version of PALM-4U and the GUI. The capabilities of the GUI were extended after the first phase and included revised setups for typical applications of urban climate models (thermal comfort and wind comfort) and functionality to view the simulation results on maps. Thus, no external software was necessary to view the results in this second phase. As there were no tools to create the input data for PALM-4U, UseUClim developed an appropriate software for the training courses. Also a user manual was developed by UseUClim and tested in the trainings.
All practice partners were enabled to the extent that they could prepare their own basic municipal data and use it for simulations with the GUI of PALM-4U in the trainings. Following the training sessions, all practice partners were able to use PALM-4U incl. GUI in their own applications. The second training phase was concluded with the Module C final workshop. Each practice partner submitted a report on their experiences.

Collection of user feedback
The user feedback of the practice partners was recorded using various methods:  The direct user dialogue during the training and the self-application help to collect feedback on the operation and suggestions for further development of the model and GUI. This direct contact was particularly important for getting suggestions on improvement of the GUI's functionality and design.
An important condition for a vital user dialogue was to foster an open discussion during the trainings and to do bilateral teachings if problems occurred. Also the user manual was improved on user-feedback,  Based on the requirements of the URC to be evaluated, a questionnaire was created for each training and application phase and distributed to the practice partners in order to collect standardized feedback with regard to the requirements to be evaluated,  In addition, at the end of each user-application phase the practice partners prepared an individual feedback report documenting their experiences and assessments from the application of PALM-4U. These reports provided valuable additional input for evaluation as well as suggestions for improvement and further development and revealed new user requirements.
During the application of PALM-4U new insights were gained and thus new requirements for the URC were identified by the practitioners. These new requirements were collected and integrated in the final version of the URC [9]. As these requirements were collected in the last project phase, they were communicated to the model developers to enhance future developments and were not used for evaluating the practicability of PALM-4U.

Evaluation
The third project phase "Evaluation" combines the works of the two previous phases "Exploration" and "Experimentation" to evaluate the practicability of the new urban climate model PALM-4U and its GUI. The used methodology and the relevant findings are described afterwards.
For the evaluation, PALM-4U was used in version 6.0 and the GUI in a nearly finished development version.

Methodology
The URC worked as basis for the evaluation of PALM-4U's practicability. As explained in chapter 3.1, not all identified requirements could be implemented in the working plan of this first project phase. Thus, only the requirements that were rated as "implementable" were used for evaluating the practicability. In the end, 157 of the URC's 240 requirements were evaluated.
Every requirement comes with an acceptance criterion that describes how the requirement can be rated as "fulfilled". Besides "fulfilled" and "not fulfilled" a third category was introduced named "partly fulfilled". This was allocated, if a requirement contains a list of smaller requirements, where only a subset was fulfilled. Requirements were assigned to test scenarios. These were tested in the "Expermentation"-Phase by the practice partners and the scientific partners of UseUClim. Test scenarios were set up both for testing technical requirements (e.g. requirements for certain model capabilities) and for testing more subjective criteria (e.g. requirements for user guidance or for understanding the GUI).
Some requirements could not be evaluated, as they were finished too late in the project or the necessary ressources were missing.
As PALM-4U can be operated in two ways, the evaluation was also carried out twice: For the scriptbased-control and the GUI. Requirements that only affected the GUI were not evaluated for the script-basedcontrol method and vice-versa.  capabilities indicate potential for future applications. For a final rating of PALM-4U's functionalities, the validation with measurement data collected by Module B must be finished. Input data must always be converted into a PALM-4U compliant NetCDF data format. Currently there are no user-friendly tools available for this complex task, e.g. integration in GIS-software or a GUI. As this was not part of the scope of the [UC]²-project, it must be addressed in future works.

Results
For a practical application, the usage of the scriptbased approach is only suitable for model experts as it requires profound knowledge. Thus, a GUI is essential for the practicability and the dissemination of PALM-4U. The evaluation of the requirements for the output data shows that a GUI clearly contributes to userfriendliness and practicability. The accessibility of the result files is only achieved by displaying them in the GUI's map viewer and by exporting the PALM-4U NetCDF result files into common formats, like for example shape-files. But the handling of the map viewer was rated as not intuitive and important analysis functions are missing. The GUI only covers a selection of three predefined use cases. For some of them, necessary assessment tools are missing, like for example the evaluation of cold-air drainage flows.
Altogether, the GUI in its current version cannot be evaluated as suitable for the practical use. Basic functionalities are fulfilled in parts and in parts, the structure is logical and intuitive, at least for setting up simulations. Also, the basic concept of using reduced and simplified inputs with preassigned standard values was evaluated positively by the users. However, important functions for the practical work are only partially or not implemented, like missing functionalities for data preparation or the functionality to output time dependent data as animations or different language versions.

Conclusion
Integrating stakeholders into an interdisciplinary scientific project and into the complex task of developing an innovative simulation model is a challenging process. By applying a Living-Lab approach, UseuClim was able to do this. A significant amount of work and communication must be invested to develop a common understanding of each partner's expertise, demands and objectives. However, integrating the users from the beginning of the project made sure that the requirements of practitioners were recognized and respected by the developers. Vice versa, the insights into the scientific development process fostered the understanding and acceptance of the project's results and the simulation model itself.
The direct user integration during the experimentation phase, where the partners from practice applied the model in typical planning tasks, was essential for the engagement process. The contributions of the practice partners covered the whole spectrum of the model and GUI development process and ranged from hints on improving the design of the user interface to suggesting new features, interfaces and application cases of the model.
For testing the practicability of PALM-4U a development version of both the model and the GUI were tested with the users from practice. The model itself fulfills the majority of the tested requirements and the users appreciated the model's concept and functionality. But further development is necessary to provide the practitioners a tool that is applicable in their daily work: Preparation of input data, a user-friendly graphical user-interface, enhanced interfaces to other software and planning tools, use cases that were prepared from experts as well as guidelines and tools for result assessment and interpretation were main suggestions for improvement.

Outlook
The project "Urban Climate under Change" was extended for a second three-year project phase (2019 -2022) by the German Ministry of Education and Research (BMBF) where the suggested improvements of the practice partners will be implemented and structures for a successful operationalization of PALM-4U will be developed. The projects UseUClim and KliMoPrax then form a joint consortium under the name ProPolis (www.uc2-propolis.de) to achieve these tasks. Part of this new project will be the development of a new GUI for users from practice, where the described findings of the first project phase will help to improve the again pursued living-lab-approach.