scenario

ART22.pdf

Scenario planning in public policy: Understanding use, impacts and the role of institutional context factors Axel Volkery, Teresa Ribeiro European Environment Agency, Strategic Futures Group, Kongens Nytorv 6, DK 1050

Received 24 november 2008 Received in revised form 15 july 2009 Accepted 27 july 2009 Scenario planning has formed a growing area of interest on the interface of academia and public and private sector policy-making.

less attention has been paid to studying the use, impacts and effectiveness of scenario planning in public policy-making. This article combines preliminary findings from a review of evaluative scenario literature with workshop discussions among scenario practitioners, using environmental relevant policies as a case study. Subject to the nascent evaluative scenario literature

our preliminary findings highlight that scenario planning still is executed often in a rather ad hoc and isolated manner

and is geared mostly towards indirect decision support such as agenda-setting and issue-framing. The slim evidence base aggravates the assessment,

but the potential of scenario planning to prepare public policy-making for the uncertainties and surprises of future developments and better manage complex decisions involving conflicting societal interests is clearly not fully utilized.

Scenario planning Decision support Political and institutional context Evaluation 1. Introduction Policy-makers and business leaders often face strategic decisions with uncertain future outcomes.

Scenario planning has been developed as a method to represent and deal with such deep uncertainties. Over the recent decades, it has formed a growing area of interest on the interface of academia and public and private sector policy-making.

& Social Change Methodological approaches to scenario planning are covered well in the academic literature. Their use and impacts in the realm of public policy is treated,

While a lot of scenario work has been conducted so far a systematic analysis of its use, impacts and effectiveness in environmental policy-making is still missing.

what we call the evaluative scenario literature, introducing a pool of 52 literature pieces that deal with one

or several aspects of using scenario planning 2. This article merges the findings from this review with the results from a workshop with environmental scenario practitioners and policy-makers.

Our analysis is informed predominantly by a focus on scenario planning in environmental policy and other environmental relevant policy fields.

evaluation and learning amongst environmental scenario practitioners and policy-makers about new forms of long-term strategic policy-making and their enabling conditions.

We understand scenario planning broadly as the aim to identify a range of new threats and opportunities that arise across a set of plausible alternative scenarios,

describe a range of possible consequences for candidate policies, help discover policy options demonstrably robust to long-term uncertainties and surface some of the blind-spots of an organisations'policy,

and the institutional embedding of scenario planning with greater care than it has been done so far. Making better decisions under conditions of deep uncertainty requires not only rigorous analysis. Even well-constructed,

thoroughly analysed scenarios can be of little use and relevance, if the organisational capacity to absorb them is poor,

The distinction between scenarios as products and scenarios as processes is relevant in this context. Process is as important as the product 7. The remainder of this article is organised as follows:

and leading questions for the analysis. Section 3 synthesizes the findings from the review of evaluative scenario literature.

and sketches a first analysis of success factors and barriers to scenario planning in public policy. Section 5 discusses the main findings

and Section 6 synthesizes implications for the future development of the evaluative scenario literature. 2. Functions of scenario planning in the policy-making process The scenario planning literature highlights a wide range of decision support functions 8,

9. Ideally, scenario planning helps policy-makers making better sense of changes in their external environment, spotting early warning signals and refining perceptions of existing or emerging problems and corresponding problem-solving strategies 10.

Moreover, scenario planning helps surfacing and managing conflicts between diverging societal interests and values and helps finding common ground for future action,

Participatory scenario planning also helps mobilizing action by different public and private actors. In the longer run, it can trigger cultural change in the way institutions

Scenario planning needs to be rooted in a sufficient understanding of the milieu in which political decisions are made.

BLOSSOM is an acronym for Bridging LONG-TERM Scenario and Strategy analysis Organisation and Methods, where organisation refers to the relevance of the organisational

and institutional context and methods refer to the need of developing methods to better align long-term scenario

This diversity requires sound governance of scenario planning, as different formats might be necessary to conform to diverging information needs and context conditions.

The different functions of scenario planning can be distinguished into indirect and direct forms of scenario-based decision support (17,

Moreover, scenario planning can offer a policy risk-free space to visualize, rehearse and test the acceptability of different strategies without being implicated by the actual constraints of day-to-day policy-making.

These direct forms of scenario planning require delivery of more targeted information and insights on the candidate policies or strategies in questions, eliminating less favorable alternative options and focusing on the preferable ones.

1. How is used scenario planning: is geared it more towards indirect forms of decision support or more geared towards direct forms of decision support,

How can those in charge of setting up a scenario planning exercise in public policy effectively proceed? Which institutional arrangements might be beneficial to the purposes of scenario planning and

which capacities need organisations in the public sector develop? These questions guided our review of the available literature 2

we limited ourselves mainly to literature on scenario planning, because scenarios provide a potentially very attractive approach to addressing uncertainty and complexity.

Furthermore, the focus of our review was on literature that explicitly attempts to evaluate scenario planning approaches,

including comparative studies of the strengths and weaknesses of several different exercises and assessments of the impacts scenario planning approaches can have on decision-making processes Fig. 1. Forms of scenario-based decision support. 1200 A. Volkery, T. Ribeiro/Technological forecasting

& Social Change 76 (2009) 1198 1207 and organisational performance. To identify relevant literature a number of experts were asked for suggestions.

A narrow focus on evaluative scenario literature was kept, but the aim was to be inclusive with regard to what was included under this heading.

As part of this analysis we organised a two-day international workshop gathering thirty environmental scenario practitioners from governments

when using scenario planning. 3. Synthesizing the available evaluative scenario literature 3. 1. Types of evaluative scenario studies The literature attempting to assess the impacts of scenarios employs different types of evaluative methods.

Some studies describe the theoretical benefit of scenarios, some drawing on general psychological understanding of human decision processes and biases

Some studies provide detailed comparisons of several scenario exercises in order to assess the factors affecting the success of such exercises 17,22 25.

A particular type of work is ethnographic studies examining how scenarios are used within organisations 26.

In addition, survey instruments in workshop settings are useful tools to measure how different types of scenarios can affect decision makers'understanding of the challenges they face and preferences for response options 27.

There are also data available on the types of businesses that use scenarios most often large firms in capital-intensive industries with long (greater than 10 years) planning horizons.

However only a handful of studies report empirical results on the central issue how scenario use correlates with an organisation's performance 28.

or are difficult to single out from the complexity of influence factors. 3. 2. Impacts of scenario planning The few studies that attempt to evaluate the effectiveness of scenarios find them useful 29,30.

We only identified one empirical study that concretely assesses the effects of scenario use on organisational performance:

and find that scenario use correlates with increased profit and return on capital 28. A comparison of four scenario exercises related to global change applications suggests climate scenarios are used mostly to support further modeling and analysis,

though they can also help frame public debates 17. A limited relevance for decision-making in policy processes is recognized by a review of practice in the broader area of foresight:

Using scenarios can increase participants'perceptions of their strategic communication and conversation skills. This has been confirmed by studies that gather data on individual participants in a scenario planning project 20.

One example is the study of Groves et al. who presented decision aids incorporating scenario concepts in a series of workshops to managers

and stakeholders of a California water agency and measured participants'increasing understanding of the challenges posed by climate change

using scenarios to address the challenges facing firms can easily fail for another reason: managers can reject the scenarios

because no viable options existed that would enable the business to address the threats the scenarios conveyed 32.

Several studies highlight potential shortcomings of scenario exercises or emphasize ways in which they deviate from the practice described in the case study literature.

For instance, deductive approaches to scenario development are used commonly. They usually rank driving forces, select the two most important driving forces

(i e. those driving forces which are highly uncertain in terms of their future development and could have a decisive impact on the topic at hand)

and plot them on two axes to structure the subsequent development of four diverging scenario stories.

Much of this literature highlights the need for consensus on the scenario axes in order to foster a common basis of understanding.

A close examination of a qualitative scenario project conducted by The netherlands Institute for Spatial Research identified three different and contradictory interpretations of the scenario axes employed 26.

The main impacts of scenarios often result more from the process of developing them rather than from any published product describing the scenarios that were created.

The value in close collaboration between scenario 1201 A. Volkery, T. Ribeiro/Technological forecasting & Social Change 76 (2009) 1198 1207 developers and users, particularly at the beginning and ending stages of a scenario exercise,

can also be shown for scenario exercises related to global change applications 17. Process is an important factor,

which people trust scenarios and thus use them. Trust can based on trust in sources, that is,

in the people who develop the scenarios or trust in content, that is, the reliability of the information known to be used in constructing the scenarios.

Additionally trust is gained from methodological credibility, i e. confidence in the method used to generate the scenarios,

or trust in narrative, i e. the extent to which the scenarios build on existing metaphors and beliefs.

Finally, trust in dissemination is important, that is the stature of the people who present the scenarios 11.

A number of studies propose ways that government agencies might be organised to make better use of information contained in scenarios.

One study for the U s. government suggests several steps for using long-range forecasts more effectively,

and act on scenarios 34.3.3. Treatment of surprise and discontinuities A reason often quoted for using scenario planning as a decision support tool is its ability to reduce overconfidence about the future 35.

However, a number of studies argue that it is difficult for scenarios to accommodate or anticipate surprises or discontinuities.

The addendum to the Hart Rudman Commission which warned of the dangers of a 9-11-scale terrorist attack,

examined 20 scenario studies of U s. national security and found that all tended to focus on extrapolations of current concerns

A comparison of the treatment of surprises in 22 scenario studies highlights that seven of the eleven scenarios including discontinuities were exploratory,

None of the model-based scenario exercise included surprises 35. In this context another study suggests that standard scenario approaches tend to systematically exclude surprising or paradoxical developments as inconsistent or logical impossible.

Building scenarios from trends rather than key driving forces might reduce the combinations that normally become excluded in the scenario axis approach, because of potential inconsistencies.

Inductive approaches more easily allow focusing on wild cards, the authors argue, and extend the scenario building process to introduce seemingly paradoxical elements to force a sensitivity analysis on the assumptions underlying claims of inconsistency 36.

Inductive approaches, in opposition to deductive approaches, are guided not by a structuring Framework scenarios rather evolve by brainstorming events and spinning larger stories around these events.

However, one might argue that the higher costs of inductive approaches in the beginning might pay off later in the process given the fact that unmet expectations about the ability of scenarios to deliver novel insights about future developments oftentimes lead to frustration or rejection of the approach by policy-makers.

but our review did not provide the space. 3. 4. Differences between public and private sector applications Much of the literature on scenario theory

thus examine potential differences between scenario exercises in the public and private sectors. A comparison of five public sector scenario exercises addressing regional development concludes that there is little difference in the actual day-to-day work of developing scenarios between public and private sector applications 23.

However, scenario users in the public sector can face more difficult challenges in establishing the client

framing the purpose of the engagement, and gaining the participation of the all the relevant parties.

Methods that work well in developing scenarios for small groups, well known to the scenario developers, may not work well in developing scenarios that can be used by large organisations or in broad political debates 37.

The previously mentioned ethnographic study of scenario axis practice in The netherlands concluded that the diffuse and heterogeneous nature of objectives

and interests faced by public agencies may make impossible any consensus about the meaning of scenario axes 26.

Difficulties in applying traditional scenario methods among the heterogeneous stakeholders public sector scenario exercises must engage are reported also elsewhere in the literature 17.

The traditional scenario axis approach argues against including probabilistic information with scenarios. However, in some public sector applications such probabilities may prove useful.

and against including probabilities with climate scenarios a key study concludes that probabilities may be useful in some situations,

in particular when the key variables distinguishing the scenarios are few and quantitative and the potential scenario users are numerous and diverse.

Probabilities may be less useful when the scenarios are rich, complex narratives; their purpose is heuristic exploration;

and the users are few, similar, and known 39.1202 A. Volkery, T. Ribeiro/Technological forecasting & Social Change 76 (2009) 1198 1207 4. Reviewing practitioners'perspectives 4. 1. Relevance

On the basis of their personal experience, workshop participants were asked to rate the direct and indirect forms of scenario-based decision support both on a scale from 1 (very low) to 5 (very high) with regard to the specific relevance (importance)

Overall, participants identified a higher relevance of more indirect forms of scenario-based decision support. Stimulating wider debates about possible futures and clarifying an issues importance were ranked as the objectives of highest relevance.

and our own literature review 2. Scenario exercises are used often as ameans of collecting and synthesizing complex information

In particular, scenario planning was considered less relevant and less successful in appraising the robustness of options for future action.

Scenario planning can easily clash with established routines of political decision-making which are informed by policy path-dependencies.

The level of political difficulty is higher for these forms of scenario-based decision support, as they touch upon a number of vested interests around policies,

scenario planning follows a rather linear, rational logic of identifying a problem structure, shaping problem solutions and determining the solution that works best across a range of alternative futures.

when evaluated against the framework of scenarios. Another argument concerned a potential mismatch of scenario formats

and information demands by policy-makers. The literature discusses the notion of scenarios often being hollow diamonds that sparkle alluringly

but fail to contain real value to the decision-making process 35. Our process came to similar conclusions.

Workshop participants also acknowledged the general difficulties of evaluating impacts of scenario planning. However, there was a common agreement among workshop participants that the potential of scenario planning is currently underutilized in dayttoday policy-making.

Apparently, a rather ad hoc use is still prevalent. Scenario planning and futures thinking in more general terms,

was described to often be isolated a rather activity. 4. 2. Factors determining success Institutional capacity-building was deemed an important requirement for making overall progress.

and the skills and experience of those using the scenario outputs were regarded to be of critical importance.

and experiences of those carrying out the scenario exercise is the selection or negotiation of the scenario methodology.

Discussions pointed to recent approaches to strengthen capacities for scenario planning and foresight within governments. Examples include:

often in form of dedicated units. 5 Examples of using scenarios to frame and support decision-making include the 5 The Future Analyst Network (FAN-Club) has been established as a permanent network of people dealing with future-related issues in different departments, agencies and the private sector. 1203

In Sweden, the Swedish Environmental Objectives Council which is responsible for monitoring Sweden's set of long-term environmental quality objectives used a set of four alternative scenarios to generate a range of new,

On the other hand, the strategy unit in the Department for Agriculture in The netherlands is an example for an actor in charge of long-term sectoral analysis. A number of other procedural options can be utilized to strengthen the role of scenario planning.

ex-post or midterm policy evaluation could make long-term scenario planning become a more integrated component of the policy cycle.

long-term scenarios could help to broaden the focus of assessments, which several evaluation studies found as one weakness of efforts so far 41.

where scenario planning has been part of developing Scotland's 2nd National Planning Framework (NPF), setting out the range of strategic alternatives that were considered as part of the early development of the NPF,

Discussion of findings The evaluative scenario literature remains nascent, making it difficult to come to an elaborated discussion.

As a consequence, those in charge of scenario planning in public policy, but also those trying to evaluate ongoing practice,

Scenario planning is geared more towards indirect forms of decision support in the early phases of the policy cycle,

A lot of progress needs to be Fig. 2. The key success factors of scenarios and other futures methodologies. 1204 A. Volkery

however, towards getting scenario planning more fully incorporated into processes of policy design, choice and implementation. While many studies report on individual learning effects, there is a larger gap

when it comes to understanding the impacts of scenario planning on overall organisational performance. The literature does not report too many differences between public and private sector applications.

But scenario practitioners often confirm that the heterogeneous nature of objectives and interests faced by governments makes it difficult to establish a key client,

which is indeed a key aspiration of scenario planning. In addition, policy-makers and strategists often have not only different time horizons than scenario planners,

but also very different attention foci. While this is long known, a mutual confusion about information needs, expectations and capacities to conform to these aspirations continues to exist.

Contradictions between product and process functions of scenario planning might aggravate this confusion. Does this assessment mean that future efforts should concentrate rather on indirect forms of scenario-based decision support,

since they are easier to implement? Our discussions with scenario practitioners pointed into the different direction.

Having an impact on the design and choice of policies remains a litmus test for the relevance of scenario planning.

Current conditions in many governments and public administrations constrain the ability to effectively pursue direct forms of scenario-based decision support.

Many scenario exercises are commissioned in an ad hoc manner, and usually are executed by external contributors. Follow-up is often lacking,

and the role and purpose within the decision-making process is not always clear. It is not remarkable to diagnose a limited impact on political decision-making

when the governance conditions for scenario planning are developed not rightfully. The struggle of balancing requests for open-minded

Scenario planning needs to conform to a variety of information demands, conflict constellations and diverging concerns for scientific credibility, legitimacy and relevance, both within and outside government and public administration.

It requires flexible and adaptive scenario formats and processes that vary between open and closed formats of interaction with many or few actors 12.

Notwithstanding the context dependency of scenario planning, more and more scholars argue that effective scenario planning therefore needs procedural stability in addition to flexibility.

Creating the right incentives for policy-makers to engage in scenario planning gain trust into the process

to undertake scenario planning regularly and report on it, for example through submitting yearly or multi-annual foresight reports.

and weaknesses of different institutional arrangements. 6. Conclusions There is no uniform experience of using scenario planning,

Many practitioners consider scenario building to be a craft or art of challenging mind-sets in strategic conversations about long-term futures.

on the other hand, scenarios are seen as part of a scientific inquiry, as in the case of global environmental assessments such as the Millennium Ecosystem Assessment or the IPCC, work on the systematization of methods

Notwithstanding the context dependency of scenario planning two key conclusions can be drawn from our analysis for further developing the evaluative scenario literature.

and to which effect scenario planning is used and how it can deliver to its promises. The second conclusion concerns the study of institutional arrangements that are put in place to make scenario planning work.

In this paper we reported findings from a scoping phase of a project that in its following phases will hopefully contribute to filling this gap.

There are important claims in the literature about scenario's ability to affect the cognition of individuals and groups by reducing overconfidence,

Numerous case studies suggest that scenarios can improve the performance of organisations, but few studies have tested these claims.

the importance of the context and process of scenario creation is emphasized. But there is little evidence to connect these insights to resulting performance of the participating organisations.

The evaluative scenario literature should improve alongside these lines. Many public sector applications may require a more systematic connection between scenarios

and recommended decisions than do private sector applications. Forthcoming efforts should include conducting empirical studies of the impacts of scenarios on organisational performance and their key drivers;

and develop a deeper understanding of how the context and process of scenario creation affects their impacts.

If and how probabilistic information should be used with scenario planning in public policy would be another concern.

While probabilities may prove a hindrance in many business sector applications, others argue that the large,

diverse audiences for public policy scenarios often require some type of information about relative risks. In addition, the question of a potential correlation between scenario approaches and impacts and effectiveness of use require further evaluative efforts.

The postulate of some studies that inductive approaches seem to perform relatively less well in comparison to more inductive approaches

While the ability of a scenario planning exercise to deal with discontinuities and surprise factors is influenced by methodological design aspects,

(and either might be the case) scenario practitioners need to pay more attention to windows of opportunity, the areas of maximum uncertainty and doubt,

There are sufficient examples to suggest that scenario planning is an appropriate tool in the right circumstances

where the scenario builders and developers are sufficiently skilful and representative, and where the intended target audience of the work is engaged, understood and sufficiently responsive.

however, modesty is required about the claims made on behalf of scenario planning. References 1 Wavfrify, The role of futures thinking in government:

A Review Of Evaluative Scenario literature, Technical Report 3/2009, Copenhagen, EEA, 2009.3 R. J. Lempert, S. Hoorens, M. Hallsworth, T. Ling, Looking

A Review Of Evaluative Scenario literature, RAND WR-564-EEA, 2008.4 D. Mercer, Robust strategies in a day, Manage.

Decis. 35 (1997) 219 223.5 P. Wack, Scenarios: shooting the rapids, Harvard Bus. Rev. 63 (6 nov./

Scenario development and Analysis for Forward-looking Ecosystem Assessments. In: MA Millennium Ecosystem Assessment (in prep. Ecosystems and Human Well-being:

A Manual for Assessment Practitioners, forthcoming. 7 B o'Neil, S. Pulver, S. Vandeveer, Y. Garb, Where next with global environmental scenarios?

Lett. 3 (2008) 045012.8 P. Bishop, A. Hines, T. Collins, The current states of scenario development:

and evolution of scenario techniques in long range business planning, Futures 37 (8)( 2005) 795 812.10 R. J. Lempert, S w. Popper, S. C

New methods for Quantitative, Longer-Term Policy analysis, 2003, RAND MR-1626-RPC. 11 C. Selin, Trust and illusive force of scenarios, Futures 38 (1

Lessons from developing participatory land use scenarios on a European scale, Syst. Pract. Action Res. 21 (6)( 2008) 459 477.14 M. Howlett, M. Ramesh, Studying public policy, Policy Cycles and Policy Subsystems, Oxford university Press, Oxford

, K. Fisher-Vanden, D. W. Keith, L. O. Mearns, H. M. Pitcher, C. E. Rosenzweig, M d. Webster, Global-change scenarios:

and Conflict, Wiley, Chichester, UK, 1989.19 P. J. H. Schoemaker, Multiple scenario development: its conceptual and behavioral foundation, Strateg.

Exploring the relationship between scenario planning and perceptions of learning organization characteristics, Futures 38 (7)( 2006) 767 777.21 Y. Garb, S. Pulver, S. Vandeveer, Scenarios in society

, society in scenarios: towards a social scientific analysis of storyline-driven environmental modeling, Environ. Res.

Information and Influence, MIT Press, Cambridge, 2006.23 J. Oglivy, E. Smith, Mapping public and private scenario planning:

M. B. A. van Asselt, Practicing the scenario axis technique, Futures 38 (1)( 2006) 15 30.27 D. Groves, D. Knopman, R. Lempert, S. Berry,

C. Phelps, S. C. Kapsalis, Does scenario planning affect performance? Two exploratory studies, J. Bus. Res. 51 (2001) 223 232.29 J. C. Glenn, T. J. Gordon, Factors Required for Successful Implementation of Futures research

on discontinuity and scenario development, Futures 35 (5)( 2003) 423 443.36 Theo J. B. M. Postma, Franz Liebl, How to improve scenario analysis as a strategic management tool?

Change 72 (2005) 161 173.37 R. Lempert, Can scenarios help policymakers be both bold and careful?

G. Ringland (Ed.),Scenarios In public Policy, John Wiley, Chichester, 2002.39 D. G. Groves, R. J. Lempert, A new analytic method for finding policy-relevant

scenarios, Glob. Environ. Change 17 (2007) 73 85.40 J. A. Dewar, Assumption-based Planning: A Tool for Reducing Avoidable Surprises, Cambridge university Press, Cambridge, 2002,

Axel Volkery, Phd, is a project manager for policy and scenario analysis at the European Environment Agency in Copenhagen, Denmark.

His research interests are integrated environmental assessments, long-term policy analysis and institutional arrangements for effective scenario planning. He has authored

She has authored co several scenario and foresight publications at the European and international level. Teresa has worked in environmental policy issues for over 20 years

< Back - Next >