Operational forecast uncertainty assessment for better information to stakeholders and crisis managers

Flood forecasting uncertainty is crucial information for decision makers. However, deterministic only forecasts have been communicated in France until now, like in many other countries. The French Flood Forecast Centres (FFCs) recently set up a new service which aims at publishing quantitative forecasts along with their associated uncertainty. Two surveys (one of the greater audience and identified end-users, another of FFCs worldwide) were conducted to design it. The forecasters' toolbox was then supplemented with two new tools. The first one provides automatic forecasting uncertainty estimations calibrated on past forecasting error series. The second one allows the forecasters to incorporate their own expertise to adjust the automatically calculated uncertainty estimation. The evaluation of the forecast uncertainty estimations issued in real time in 2014 suggests that even if these assessments are perfectible, they are already informative and useful for end-users. The first feedbacks from forecasters and crisis managers also show that if the assessment of probabilistic forecasts remains a technical challenge, their use is foremost a human challenge. This move is a paradigm change for both forecasters and decision makers. Therefore, they have to be accompanied in order to achieve this deep shift in their professional practice.


Operational forecasts uncertainty sources
Since many human populations settled close to rivers, violent flash floods affecting headwater catchments as well as slower floods of alluvial plains, are among the natural hazards which bring about most of casualties and damages worldwide [1].A single massive event may FDXVH VHYHUDO WHQV RI *¼ of damages (e.g., a 100-year flood in the JUHDWHU 3DULV DUHD ZRXOG FRVW XS WR *¼ [2]).
In order to efficiently manage these different types of natural hazards, Flood Forecasts Centres (FFCs) have been set up in many countries.They are government agencies or are funded by private firms such as hydroelectricity suppliers.Even if forecasting is a key resource for flood mitigation, it remains a challenging task.Indeed, operational flood forecasting has to take into account several uncertainty sources, which limit (sometimes drastically) the accuracy and the horizons of the issued forecasts.
Among the main operational uncertainty sources, the meteorological inputs uncertainty must be quoted: the uncertainty associated with rainfall and temperature forecasts limits the lead times for informative and useful flood forecasts.Several methodologies are actively studied to deal with this source of uncertainty, e. g., ensemble forecasts [3] or the analogous sorting approach [4].Hydrological and hydraulic modelling uncertainty (including parameter uncertainty) is far from being negligible either.Modelling improvement but also model updating and post-processing are still active research fields [e.g. @ 6RPH PRUH µRSHUDWLRQDO ¶ XQFHUWDLQW\ sources should not be forgotten, such as real-time observation uncertainty or rating curve uncertainty which can become very significant when it comes to large floods [8].

Forecasting uncertainties: information needed by the decision makers
Thus, current research intends to increase understanding of hydrological processes, to improve forecasting tools and to reduce the predictive uncertainty.Nevertheless, decision makers and especially crisis managers, have to cope with the current forecasting tools and their actual limits and associated uncertainties even if some improvements may be achieved in the future.Uncertainty is an essential part of the decision making process and should be considered.As an illustration, if the limit discharge before the overtopping of a levee is about 1000 m 3 /s and the evacuation of the area protected by this levee should be decided, forecasts of a maximum discharge of 800 m 3 /s (deterministic forecast), or of a maximum discharge comprised between 600 and 1200 m 3 /s, or of a 30% chance to exceed the limit discharge of 1000 m 3 /s, clearly do not have the same information content and may lead to contrasted decisions.The deterministic forecast may in this example be misleading for the decision maker.Uncertainty is then an inherent part of the problem which has to be taken into account when a flood event occurs.It should not put a brake on decision making.
Forecasts are a piece of information among many others available to the crisis manager and the uncertainty level is a key component of this information that the decision maker needs in order to take the right decision.

Explicit assessment and communication
Beyond the technical challenges of uncertainty assessment, its communication to decision makers or to the greater audience is not a simple task, especially in a context of crisis management.This implies a clear framework based on mutual trust where the roles of the forecaster (the often so-called expert) and of the decision maker are well separated.It must ensure that the given information is properly transmitted, correctly understood and fairly used.Therefore building a flood forecasting service which includes the communication of quantitative uncertainty assessments, requires dealing with organizational issues regarding the whole crisis management system [9,10].

Scope
On the basis of the conclusions drawn by two workgroups created in 2012 by the French Ministry of Ecology, Sustainable Development and Energy to prepare the evolution of the governmental flood forecasting system, SCHAPI (French national flood forecasting service) and local FFCs are currently implementing a new service.It aims at publishing quantitative discharge and water levels forecasts along with uncertainty assessment, usable by decision makers.These workgroups gathered hydrologists and social scientists from the research community, and forecasters working in operational services.Their suggestions concern the technical options adapted to the operational context to properly assess flood forecasting uncertainty.They also discussed the organization needed in order to design and efficiently run this new service.This article aims at describing this system and at discussing some challenges FFCs are confronted with: (a) How can a trained forecaster deal with these new explicited uncertainties?(b) How to efficiently communicate them to end users and crisis managers in order to help them to ³DVVHVV WKH ULVN, pick DFWLRQV RU GR QRWKLQJ´ [11].
The next section of the article briefly presents the results of an international survey of the current practice of FFCs, prior to the operational implementation of the new methodology.Then, the French operational system is described.Section 4 discusses the lessons to learn from the evaluation of the first forecasting uncertainty assessments (issued and published in 2014) and from the feedbacks of forecasters and decision makers, regarding the operational challenges in building a coherent set of good practices.

A survey of worldwide practices
A survey of flood forecasting uncertainty assessment and publications was realized in 2013 in order to choose the technical options best suited to the French operational context [12].It gathered answers from 18 operational FFCs, most of them being government agencies (15); the other 3 FFCs issue forecasts for hydroelectricity suppliers.They are located in Europe, North America and Australia; their territories are under diverse hydroclimates (from Sweden to Australia).They also operate within different operational cultures and frameworks [13].
A large majority (89%) computes quantitative forecast uncertainty estimations.Different methodologies are in use (Fig. 1).In many cases, uncertainty assessment is based on the combination of at least two approaches.Preferred methods are based on meteorological ensemble forecasts (61%) to take into account what is often considered as the major source of uncertainty in flood forecasting, and post-processing based on a statistical a posteriori forecasting error analysis (63%).In 15 cases (83%), uncertainty assessments are published along with the forecasts.Again, various displays are used (Fig. 2).The most popular one (73%) is to plot a small set of quantiles of the predictive uncertainty (Fig. 3).Besides the results on technical options, the survey revealed that the operational implementation is not immediate because uncertainties broadcasting modifies WKH IRUHFDVWHU ¶V MRE LQ GHSWK &DUHIXO JXLGDQFH WR WKH forecasters and support to the main forecast users were highlighted as keys to success by many respondents.
3 The French operational system to provide uncertainty assessment along with hydrological forecasts SCHAPI and the FFCs are currently upgrading their operational forecasting system to explicit their predictive uncertainty for forecasts issued at some 580 stream gauges.The deterministic hydrologic and hydraulic models used by FFCs are supplemented by two new tools: x OTAMIN, which computes quantitative assessments of the predictive uncertainty, based on the analysis of past forecasting errors of the GHWHUPLQLVWLF PRGHOV ,W SURYLGHV D µILUVW JXHVV ¶ to the forecasters; x EXPRESSO: an interactive tool which helps the forecasters to express their own (subjective) MXGJHPHQW DQG WR SURGXFH WKH µILQDO ¶ XQFHUWDLQW\ assessment.

OTAMIN
A strong research effort is dedicated to the quantitative assessment of forecast uncertainty.Several approaches have been studied and tested.They differ in the uncertainty source(s) they take into account, their theoretical grounds and their usability for operational real-time purposes.Thus, methods like the multi-model approach [14,15], the GLUE method [16] or ensemble forecasts [17,18] only consider a few uncertainty sources (respectively, the model uncertainty, the parameter uncertainty, and for the latter the meteorological input uncertainty most often).GLUE is built on a (very) simplified theoretical framework whereas Bayesian approaches [19,20,21] intend to be more rigorous in expliciting assumptions (model error embedded in the likelihood function).These methods commonly use a sampling scheme (e. g., the MCMC algorithm) which can be time-costly and may not be appropriate for today ¶V real-time applications.Another popular approach is deterministic forecasts post-processing which aims at assessing the total predictive uncertainty, independently from the different uncertainty sources.Among others, major examples are the hydrological uncertainty processor [22], the meta-gaussian approach [23] and the quantile regression [24].
OTAMIN post-processes the forecasts issued by the deterministic models.Its output is a set of predictive quantiles, in order to give a brief description of the predictive uncertainty (Fig. 4).Its calibration is based on a posteriori error series analysis.Among the numerous available algorithms, two methods were selected to be included in OTAMIN: a parametric one and a non parametric one: the popular quantile regression [24] and QUOIQUE [25,26].The former method assesses conditional predictive quantiles as a linear function of the predictand (the deterministic forecast) in the normal quantile transformation (NQT) domain.The latter computes a set of error quantiles from a series of a posteriori forecasting errors for different ranges of the predictand (also the deterministic forecast) in order to describe the observed uncertainty (Fig. 5).This description is then added to the deterministic forecasts in real-time.
Indeed, this requires to define the error series.OTAMIN offers two simple error models: an additive error and a multiplicative error, in order to being able to deal with water level and discharge forecasts.At this point, time correlation is not taken into account (e. g., no auto-regressive scheme is implemented so far).
OTAMIN has a number of limitations, many of them in common with most post-processors.It heavily relies on the data, since the underlying assumptions are limited: there is no strong controlled error model.Therefore, a lack of data may be highly detrimental (which is often the case with hydraulic models).Its performance is often questionable in extrapolation (for predictand values higher than in the calibration data set).Last but not least, for operational matters, it can only be calibrated based on hindcast data in many cases, especially for new models or if real-time forecasts have not been properly archived.In such a case, it may not consider the full predictive uncertainty and therefore provides an optimistic

EXPRESSO
EXPRESSO is a graphical workspace where forecasts are displayed in a simplified way: three curves are plotted.They correspond to three predicted quantiles as functions of lead-time.These quantiles usually are q 0.1 , q 0.5 (median) and q 0.9 .The forecaster can modify the shape of these curves in order to increase/reduce the magnitude of the hydrograph (or limnigraph), or speed-up / slow-down the rising limb or the recession one (and then move the time of the peak), etc.Thus, he/she can increase or reduce the associated uncertainty.This is done by selecting some parts of the curves, and moving them XVLQJ µGUDJ DQG GURS ¶ IXQFWLRQDOLWLHV This tool has been conceived to allow the forecaster to quickly express her/his own expertise, on the basis of an DXWRPDWLF SUHGLFWLYH XQFHUWDLQW\ DVVHVVPHQW 7KH µILUVW JXHVV ¶ LV EDVHG RQ WKH UDZ GHWHUPLQLVWLF IRUHFDVW WKDW WKH forecaster selects among the available outputs of different models (possibly run with several different input scenarios) after OTAMIN post-processing.
They are communicated to public decision makers (email) and published through a governmental website for the greater audience (www.vigicrues.gouv.fr,Fig. 6).It is therefore strictly necessary to keep the displayed information simple, even if more information can be brought to governmental crisis managers and decision makers by phone.These new tools and this new process are not part of WKH IRUHFDVWHU ¶V URXWLQH \HW +RZHYHU ILUVW IHHGEDFNV both from forecasters and decision makers, reveal that explicitly assessing and communicating forecasting uncertainties is not an easy step and may lead to some significant shifts in their professional culture and practice.

First feedbacks from forecasters
When first confronted to this new assignment, some forecasters may be somewhat puzzled, because they reckon that it significantly modifies their practice.It is the classic sharpness / reliability dilemma.Many forecasters feel that they are mostly praised for providing accurate information.Indeed, decision making is a much easier task in a perfectly known future: providing very sharp forecasts is then often seen as a condition to be UHFRJQLVHG DV DQ µH[SHUW ¶ &RPPXQLFDWLQJ XQFHUWDLQWLHV LV then quite disturbing [27].In particular, forecasters worry about the reactions of the crisis managers they are in contact with: would they understand?Would they be able to use this information, especially when provided with a large ± yet justified ± uncertainty range?What would they think of him/her if he/she does not provide a forecast which is perfect-looking (that is, a deterministic forecast)?Some forecasters expressed concerns about the way their work would be evaluated.The expectations had then to be clearly stated here: the forecaster management must define what a good forecast is.The objective of reliability has to be emphasized, and the sharpness of uncertainty UDQJHV LVQ ¶W WKH XOWLPDWH JRDO.This clarification is a necessary ingredient to set-up a climate of trust for this new objective.If their confidence is not gained, the forecasters are not in a position to produce reliable forecasts: some may believe that producing unrealistically sharp forecasts put them in a safer position.
Not all forecasters are experts in hydrology or hydraulics.Thus, another strong concern of some forecasters regards their ability to have and to express their own uncertainty assessments.In particular, many of them asked how they could do better than the calibrated post-processor (OTAMIN).Consequently, some forecasters expressed reluctance to add their own H[SHUWLVH E\ PRGLI\LQJ WKH µILUVW JXHVV ¶ ZKLFK LV WKH OTAMIN output).It had to be made clear that OTAMIN does not assess tKH µWRWDO ¶ SUHGLFWLYH XQFHUWDLQW\ e.g., it does not take into account the meteorological uncertainty but only the rainfall-runoff model uncertainty in many cases).Furthermore, OTAMIN is only able to provide an µDYHUDJH ¶ HVWLPDWLRQ EDVHG RQ LWV FDOLEUDWLRQ RQ D OLPLWHG data amount, whereas the human forecaster has the ability to adapt to specific situations, exactly in the same way he RU VKH LV XVHG WR µFRUUHFW ¶ GHWHUPLQLVWLF PRGHO IRUHFDVWV WR take into account some specificities of the current flood that the model cannot detect or manage.

First evaluation of predictive uncertainty estimations
Besides some initial training course, this practical shift is managed by a systematic training for every forecaster (for instance, every time he/she starts an oncall week).Moreover, regularly-scheduled evaluations of past forecasts are needed to help forecasters to detect over-confidence (too narrow confidence intervals) which is a frequent drawback when starting [27] (more rarely, under-confidence).This allows each forecaster to SURJUHVVLYHO\ µFDOLEUDWH ¶ KLVKHU VXEMHFWLYH DVVHVVPHQW Such an exercise was conducted within the FFC of the Loire, Cher and Indre rivers in 2015.The first predictive uncertainty estimations have been evaluated after one year of training and practice at 49 forecasting locations (Fig. 7) [28].Some of these rivers are subject to fast flooding (even flash floods) while others are alluvial plains rivers.The forecasts at the former sites are the results of the rainfall-runoff model (GRP, [29]) while the forecasts at the latter ones are computed by hydraulic models (HYDRA, which is a common professional hydraulic model, or some simplified propagation models) run with observed and rainfall-runoff forecasts inputs.
1,318 forecasts were analysed (discharge and water level).They had been issued for three distinct flood events.It is noteworthy that the evaluated forecasts with their associated uncertainty estimations have been published: the evaluation is based on real-world case study and not on a somewhat disconnected and more theoretical exercise.However, the 2014 floods were moderate events on these rivers and the forecasters had not been put under harsh pressure (no major asset under threat).
Mainly the reliability and the accuracy of the uncertainty assessments were investigated.Forecasts were first inspected visually.Then the frequency of non exceedance for different lead times was plotted for the WKUHH SXEOLVKHG µWUHQGV ¶ DQG YLVXDOO\ FRPSDUHG WR WKHLU expected values at each location (Fig. 8).These plots were shown to the forecasters during their annual meeting in order to initiate the discussion.
A significant variability in the forecasts reliability was observed among the different locations.The discussion among the forecasters team focused on the results for lead WLPHV XS WR WKH µWDUJHW OHDG-WLPH ¶ RI WKH ))& 7KLV particular lead-time corresponds to the commitment of the FFC to the decision makers.For catchments where forecasts are issued thanks to rainfall-runoff models, the target lead time is close to the time of concentration of the catchment, whereas it is chosen close to the travel time between stream gauges for the forecasts based on propagation models computation (floodplain floods).Target lead times range from 3 to 60h.
The shared evaluation showed that the a posteriori observation exceeds the upper trend in about 1 out of 10 cases (as expected) for many sites, if not most of them, whereas the median and lower trend are less well assessed.During the meeting, several forecasters explained that they felt uncomfortable with any modification of the median trend.They only moved it in very specific cases, when they detHFWHG DQ µREYLRXVO\ ¶ wrong forecast.They also acknowledged the fact that they most often focused on the upper trend (as an LQGLFDWLRQ RI WKH µZRUVW ¶ VFHQDULR ZKLFK LV D TXHVWLRQ often asked by crisis managers) and do not spend much time on the lower trend, seen as not so useful.
The forecasters discovered that uncertainty was often highly underestimated for the very first time steps.This may be explained by the methodology used and its interactions with the model updating procedure.
Moreover, Fig. 9 shows a pseudo PIT diagram for only three probabilities (0.1, 0.5 and 0.9) for all the forecasting locations where more than 20 forecasts were issued in 2014.For each station, the predictive 0.1, 0.5 and 0.9 quantiles are computed for its target lead time (circles: 0 to 6h; crosses: 6 to 24h; squares: longer than 24h).Some forecasts were issued in a deterministic mode, therefore there are more than 9 stations for the 0.5 quantile.It is interesting to note that the reliability level is not correlated with the target lead time and consequently not to the forecasting modelling type.
Indeed, many forecasters first worried that it would be much more difficult to assess predictive uncertainty for locations where rainfall-runoff modelling is needed to issue forecasts, because of the meteorological uncertainty.
After one year of practice, the discussion with the forecasters suggests that the predictive uncertainty is rather correctly assessed on catchments or channels which are well known and where forecasters are used to the forecasting models.The estimation is much more difficult and sometimes even random-wise for sites which are less studied (by the forecasters) or for models which have recently been set up.
This first evaluation was driven for the whole forecasters ¶ team.Overall, it was seen in a positive light: the reliability was judged acceptable for operational purposes on many catchments for a start, even if some progresses have clearly to be achieved.This is particularly true for the most important stations for operational issues, where forecasters took time to build their own analysis and did modify the first guess issued by the statistical model.It reassured some forecasters who first doubted that they were able to give reliable predictive uncertainty assessment.It is then encouraging WR SXUVXH WKLV SURFHVV DQG WR DFKLHYH RQH ¶V SHUVRQDO HYDOXDWLRQ LQ RUGHU WR LPSURYH RQH ¶V RZQ DVVHVVPHQW skills.

Design of this new service
The SCHAPI and FFCs are currently working on the evolution of the actual www.vigicrues.gouv.frwebsite which is accessible to both crisis managers and the greater audience.The new website would include several new features.To help to define needs and priorities, a large survey has been performed mid-2014 towards crisis managers, local stakeholders and the greater audience in five selected recently-flooded urban areas.
The display of forecasts with uncertainty assessment was one of the central issues of this survey, and it appears that the great majority of the surveyed crisis managers are willing to deal with uncertainty assessments (85% of them stating that adding uncertainties to forecasts brings useful operational information).Such a result was not expected.Indeed, the existence of uncertainty is acknowledged and if no estimation is given, decision PDNHUV RIWHQ µDGG ¶ WKHLU own subjective uncertainty estimation [30] but feel uncomfortable with this practice.That is why uncertainty assessment and broadcast appeared as a long-awaited information.Surprisingly, crisis managers see no barrier to provide this kind of information to the greater audience.Yet, some remain still a little reluctant, wondering if operational decisions could really accommodate with uncertainties.As a possible link, ¼ of the study sample considers not being familiar-enough with uncertainties, which enlightens the need for training and support.

Reaction of crisis managers
The new service was presented to local governmental crisis managers at several occasions in 2014 and 2015.Some general presentations about the quantitative forecasts to be communicated and broadcasted, and the predictive uncertainty estimations were shown during the annual meetings of the local governmental crisis managers and the forecasters.Crisis managers confronted for the first time with predictive uncertainty estimations gave their opinion during these meetings.
A parallel can be drawn between some reactions of the forecasters and some of the decision makers.First, many felt reluctant to use this quantitative information because it was difficXOW WR µWUDQVODWH ¶ LQWR WKHLU RZQ language.The deciphering of the information appeared as a key issue in the meetings between FFC and crisis managers.Games based on scenarios is an interesting first possibility to help them to decipher the information.First games were preferably designed for assets which are disconnected from their jobs, in order to focus on the decision making process.Then, drills (crisis exercises) based on synthetic flood scenarios can help in dealing with the use of the uncertainty information in real cases.Some decision makers were disturbed by the fact that lower and upper trends are not the minimum and maximum forecasts.They felt rather uncomfortable with this kind of information and wondered how the greater audience or their bosses would react.
Even when the information was fully understood, many crisis managers declared that it seemed more difficult to decide what to do when this quantitative LQIRUPDWLRQ ZDV PDGH DYDLODEOH ³+ow do I decide with µ\RXU ¶ ORZHU DQG XSSHU WUHQGV"´ is a sentence heard several times.They had no obvious way to process this new information.As underlined by Nobert et al. [10], there is a real need of support for the implementation of this new practice.Uncertainty information would be useful to decision makers if and only if they would be able to properly deal with it: if the proper tools (e.g., decision rules) are available to process the enriched information in order to make the right choices.

Crisis management planning
Therefore, this new tool for the decision makers was presented and discussed by and with crisis management planners on the occasion of the revision of the flood management plan for the western civil security zone (France counts 7 civil security zones).In this document (still in progress), the planners have to explicitly describe how to process the information transmitted by the FFCs and how to take it into account into the crisis management planning.They have to compare the relative costs of different actions they can take (including non action) and to build the crisis a priori management rules which will help the decision makers to use the information available during the crisis, at best.Such rules can be based on the probabilities of exceedance of a few thresholds: non action and actions costs are weighted according to false alarms and missed events probabilities.Accompanying planners in the integration of this new information is also an absolute need.
,Q SDUDOOHO WKH GHFLVLRQ PDNHUV ZHUH LQYLWHG WR µSOD\ ¶ with this information in the same way they would have to use it for real operations: the FFC provided synthetic quantitative forecasts and predictive uncertainty estimations to decision makers during crisis exercises which are organized on a regular basis.

First results and learnings
Actually, decision makers GRQ ¶W seem to be reluctant to forecasting uncertainty, but to crisis management uncertainty.
After a couple of years of repeated presentations and drills, we now observe a larger acceptance of uncertainty assessment usefulness and of the way this information would be provided and would have to be used.However, even if the way to deal with this information is clearer, this is far from being a completely achieved change.We expect that several years will still be necessary to make WKLV D µKDELW ¶ ZKLFK GRHV QRW UDLVH DQ\ IXUWKHU TXHVWLRQV This type of reactions is not specific to a single country.The learnings of the 2015 January storm on the North-Western states of the USA [11] (in a country where forecasters have been used to produce and publish probabilistic forecasts for years) show that it is a common pattern.

The greater audience
The mid-2014 survey also considered the greater audience through the organisation of a round table, gathering local stakeholders and citizens from the city of Lourdes (affected the year before by a dramatic flood event) and a phone survey targeted on the five urban areas having recently been flooded (150 answers collected).Flood forecasts displayed with uncertainty assessment is now seen as a need and an important way to progress and gathered 89% of positive opinions among the phone survey sample.
We have no feedback from the greater audience about the way they would receive and deal with the broadcasts.However, similar experience in the meteorological field shows that deciphering the quantitative forecasting uncertainty assessment is not easy or intuitive for the greater audience, neither it is to pick the appropriate behaviour.It seems important to adapt to the diversity of the public.Therefore SCHAPI and the FFCs are currently designing different reading levels of this information.These reading levels should range from very simple messages up to more quantitative information, usable by trained people.

Conclusions
The French Flood Forecasting centres recently designed a new real-time publication of their forecasts with associated uncertainty estimations.Two surveys over 18 international FFCs and over the greater audience and crisis managers have been conducted to select technical options and an organization suited for the French operational context.Two new tools have then been implemented to help the forecasters to provide such assessments based on an automatic method and on his/her own experience.Forecasts are now provided in the form of three trends.The central one corresponds to the predictive median, while the lower and the upper ones to the 0.1 and 0.9 predictive quantiles.
This article intends to present this move ³from beginning to end´ IURP WKH FRQFOXVLRQV RI WKH preliminary studies and surveys to the teachings of the first evaluations) in order to discuss the entangled challenges of explicit uncertainty assessment, communication and use by forecasters and crisis managers.
First, the forecasting uncertainty estimations issued and broadcasted in real time for three moderate flood events have been evaluated.They proved to be rather acceptable for operational use.The upper trend (0.9 predictive quantile) is satisfactorily assessed at most locations, even if significant progress has still to be achieved for the lower trend.Indeed, uncertainty is much better assessed on catchments and for models that the forecasters know quite well.
Uncertainty apprehension differs from one culture to another.Even if uncertainty is seen unfavourably, surveys showed that a large majority of crisis managers and of the greater audience now considers that forecasters and they have to deal with this uncertainty.Indeed, meetings with crisis managers made us fully aware that decisions makers are reluctant to crisis management uncertainty but not necessarily to forecasting uncertainty.Communicating predictive uncertainty is not sufficient: beforehand, the crisis management plans have to think up the decision rules to follow to process this information.
It is difficult and maybe hazardous to draw any conclusion from an on-going process.Our first feedbacks show that it is a deep shift (paradigm change) for both forecasters and forecast users.A long-lasting process which is indeed a technical challenge but foremost a human challenge is ongoing: the evolution of the framework for both flood forecasters and decision makers.This evolution has to be driven simultaneously in the two communities: during a flood crisis, the questions and the expressed expectations of the decision makers are D VLJQLILFDQW SDUW RI WKH µVRFLDO SUHVVXUH ¶ XQGHU ZKLFK WKH forecasters work.A change of the state of mind is needed.If the forecasters are not convinced that decision makers ask for reliable forecasts, even if less accurate, to make the best decisions, then no deep change in the practice is possible.

Figure 1 .
Figure 1.Most popular forecasting uncertainty assessment methods among the 18 surveyed flood forecasting centres.

Figure 2 .
Figure 2. Most popular ways to publish quantitative forecasting uncertainty assessment among the 18 surveyed flood forecasting centres.

Figure 3 .
Figure 3. Discharge forecasts published on the Centre d'Expertise hydrique du Québec (Québec FFC, Canada).The forecast is achieved on March, the 30 th (red circle).The lower and upper forecast curves correspond to the 0.25 and 0.75 quantiles.

Figure 5 .
Figure 5. Calibrated predictive quantiles assessed by OTAMIN using the QUOIQUE algorithm on a posteriori errors of the HYDRA model at Gilly-sur-Loire.

Figure 6 .
Figure 6.Mock-up of the forecasts plotting for the French government website for flood vigilance and forecasts www.vigicrues.gouv.fr

Figure 9 .
Figure 9.'Simplified' PIT diagram for the 8 stations under study where more than 20 forecasts have been issued during the flood events of 2014.For each station, the predictive 0.1, 0.5 and 0.9 quantiles are computed for its target lead time (circles: 0 to 6h; crosses: 6 to 24h; squares: longer than 24h).Some forecasts were issued in a deterministic mode, therefore there are more than 9 stations for the 0.5 quantile.

Figure 7 .
Figure 7. Locations of forecasts issued by the FFC of Loire, Cher and Indre Rivers during the October and November 2004 flood events.Sites where forecasts are based on GRP rainfall-runoff model outputs are shown as green circles.Square represent sites where forecasts are based on simplified hydraulic modelling.HYDRA hydraulic model is used on sites shown by triangles.

Figure 8 .
Figure 8. Frequency of non exceedance for predictive 0.1, 0.5 and 0.9 quantiles computed on the 22 forecasts issued at Gilly-sur-Loire during the October and November events.The target lead time is 18 h (green dashed line).Horizontal dashed lines show the perfect values.