Evaluation

Evaluation (307)
Evaluation criteria (12)

Synopsis: Evaluation: Evaluation:

This session was devoted to the issues of evaluation of FTA and its techniques and processes.

The main high-level issues for evaluation were how to reconcile the routinization and standardisation that evaluation encourages with the creativity and dwild cardt nature of many of the ideas implicit in good FTA.

Devising acceptable evaluation methods to deal with visionary work in processes is not easy and raises a challenge for the FTA COMMUNITY.

The general consensus in the session was that FTA is a driver and an instrument for social change and as such will require high quality evaluation and at the same time within its own constructs,

visions and techniques will challenge current evaluation processes and ideas. Evaluation also serves to highlight the role of FTA as learning processes for stakeholders

and thereby encouraging widespread innovation in organisational responses to the challenges of the future. 6. Importing ideas As might be expected of a session dealing with new ideas on FTA there was a wide diversity of suggestions and issues presented.

They ranged over linking evolutionary theory with foresight to provide F. Scapolo/Technological forecasting & Social Change 72 (2005) 1059 1063 1061 new ways of framing studies, applying the concepts underlying marketing tools based on human behaviour to foresight design,

and issues only rather than aiming at solutions. 6 See in particular the experiences made in the UK especially the most recent evaluation of the UK Foresight 12,

In other words, AF should be interpreted as part of a broader continuous learning process that comprises the implementation and evaluation of specific policy measures as well as a monitoring of relevant developments in policy at large.

, evaluation and monitoring; what is still missing is the integration of these methods in a continuous and long-term strategy development process.

I. Miles, O. Saritas, Evaluation of the United kingdom foresight programme, Final Report, DTI, London, 2006.13 K. Cuhls, L. Georghiou, Evaluating a participative foresight process:

Eval. 13 (3)( 2004) 143 153.14 PREST, Evaluation of the Hungarian technology foresight programme (TEP), Report of an International Panel, University of Manchester, Manchester, 2004.15 R. Slaughter, A new framework

Furthermore, they have different management practices as concerns the launching, monitoring and evaluation of RTD projects;

iii) identification of practical networking and opening mechanisms for future cooperation, iv) implementation of joint evaluation and foresight activities,

average evaluation ratings for each criterion, corresponding standard deviations. Within each subarrea issues that tended to receive most support with regard to the three evaluation criteria (i e.

For example, the average value of evaluations with regard to suitability for WW-Net and proposed distribution of fundingwere calculated.

, by taking into account all evaluations) and at the country level (i e.,, results from the preceding three workshops.

and evaluation activities, including the joint foresight project Finnsight 2015 of the Academy of Finland and the National Funding Agency for Technology and Innovation (Tekes),

10.1016/j. techfore. 2008.02.004 public activities in research and development, regulatory impact assessment is a policy evaluation mechanism

and not yet widely spread trend is the move toward ex post evaluation of regulations, which is part of the progressive development of regulatory policies,

complementing ex ante evaluations 4. Ex post impact assessments are able to evaluate the efficacy and the efficiency of regulatory instruments by measuring

it has to be noted that the evaluation of standardisation processes or standards themselves is a rather rare and only recent phenomenon in the United states 7,

where standard impact assessment is part of RTD evaluation, because of the assumption that standards are part of the technological infrastructure

In the final chapter, we conclude with a brief critical evaluation of the methods based on the experiences collected in the project,

we start with an evaluation matrix (Table 1) presenting methodologies to assess the impacts of different policy instruments.

and are put in the context of possible evaluation and assessment technologies. Table 1 Evaluation matrix:

matching policy instruments and methodologies Innovation surveys Econometric models Control group approaches Cost benefit analysis Expert panels/peer review Field/case studies Network analysis Foresight/Technology assessment

Ex Post evaluation of Regulatory policies Proceedings from the OECD Expert Meeting September 2003, OECD (ed.),Paris:

19 G. Fahrenkrog, W. Polt, J. Rojo, A. Tübke, K. Zinöcker, RTD Evaluation Toolbox Assessing the Socioeconomic Impact of RTD-Policies, Seville:

which can be produced in evaluation, foresight, or TA projects and comparative studies of national and regional innovation systems etc.

This topic is of specific interest due to the fact that no common evaluation and assessment approach for foresight exercises has been developed to date.

Thus evaluations typically focus on assessments of whether or not these goals have been attained. The evolution of a common evaluation and assessment framework for foresight exercises

however, could facilitate the identification of good practices irrespective of their specific objectives and levels of implementattion aswell as facilitating the benchmarking and coordination of policies for socioeconomic development.

However, a common evaluation and assessment approach is not possible if evaluations are based on the specific objectives

and needs that foresight exercises typically try to address. Another level of reference is needed, i e. foresight exercises need to be assessed in terms of their contribution to the attainment of higher level, generic goals.

and implementation of foresight programmes by adding design and implementation principles to the existing evaluation issues of efficiency,

effectiveness, appropriateness, sustainability and additionality. 6. Understanding the dynamics of foresight systems in three countries A first attempt to understand the dynamics of foresight systems can be based on the evaluation of three foresight exercises;

The evaluation of the latest UK Foresight programme 5 sheds light on several direct and indirect impacts and highlighted several factors that affected the overall success of the UK foresight exercise.

The evaluation found that the initiative had succeeded in its attempt to provide the neutral, interdepartmental and interdisciplinary space needed for forward thinking on science-based issues.

the evaluation revealed that institutional settings and attitudes within public institutions strongly affected expectations and the uptake of foresight results.

the evaluation confirmed the stance that foresight exercises should have a specific‘client, 'and thus a specific focus, rather than multiple and diverse clients and objectives.

The evaluation noted too that public consultation at an earlier stage would have benefited particular topics

the evaluation stressed that aftercare was needed if the action networks formed were to become self-sustainable,

The evaluation of the second round of the Swedish Technology foresight initiative 6 found that indirect effects on foresight capabilities were marked more than the anticipated impacts of foresight results on policy-making.

the evaluation pointed out that the lack of a specific‘client'was unfortunate. Even though neither the second nor the first round were organised by government but by a consortium of agencies,

K. Guy/Technological forecasting & Social Change 75 (2008) 539 557 The evaluation was critical of the intervention logic,

The evaluation suggested that foresight might need to be done in parallel at different levels with different customers.

The evaluation also highlighted the fact that broader environmental factors can influence the success of a foresight exercise.

The evaluation of the eforesee Foresight exercise in Malta 7 highlights even more the value of the foresight process as an agent of cultural and behavioural change.

The evaluation also revealed the value of a one of the less-emphasised benefits of foresight exercises,

The evaluation noted, however, that sociocultural factors made the implementation of consensus-building quite challenging.

‘Futur the German research dialogue',Research Evaluation 13 (3)( 2004) 143 153.3 R. Barre, Synthesis of technology foresight, in:

I. Miles, O. Saritas, Evaluation of the United kingdom Foresight programme, Final Report, 2005.6 E. Arnold, S. Faugert, A. Eriksson, V. Charlet, From foresight to consensus?

An Evaluation of the Second Round of Swedish Technology foresight, Teknisk Framsyn, 2002 2004,2005. 7 J. Cassingena Harper, L. Georghiou, The targeted and unforeseen impacts of foresight on innovation policy:

the eforesee Malta case study, International Journal of Foresight and Innovation policy 2 (1)( 2005) 84 103.8 L. Georghiou, M. Keenan, Evaluation of national foresight activities:

tools to inform debates, dialogues and deliberations, Technikfolgenabschatzung, Theorie und Praxis 2 (14)( 2005) 74 79.14 R. Barre, M. Keenan, FTA Evaluation, Impact and Learning,

Organisational Alliances, Partnerships and Networks in Management Benchmarking Study, Washington Research Evaluation Network, www. wren. network. net/resources/benchmarking. 27 J. Airaksinen, A. Haveri, Networks and hierarchies in inter-municipal co-operation.

Research interests include the areas of research evaluation and impact assessment, foresight, national innovation systems and policies, scientific advice for policy-making and risk governance.

and led the EGIST (Evaluation of Government and Industry Strategies for Technology) group; and Technopolis Ltd, an innovation policy consultancy which he founded in 1989 and which,

and evaluation of the efficiency and impacts of their activities would be perceived as a burden.

measurement and evaluation beyond the traditional academic indicators Grants offered by universities are designed both to nurture talents and foster cohesion Inferior performance and a weakening position vis-à-vis the leading Triad

together with evaluation methods, understood as a useful tool, which would also enhance their visibility and social esteem. 34 5. Conclusions This article considered alternative futures for EU universities.

The evaluation of options is carried first out by the core team applying the different criteria in a balanced way for each scenario.

A comparison of the stakeholders'assessment with the sustainability evaluation of the core team offers insight into the sustainability deficits that a politically negotiated solution would entail (see Fig. 5)

Nevertheless, in the final evaluation of the process, they strongly welcomed the opportunity to conjointly develop the scenarios

the future of transport, Futures 40 (10)( 2008) 865 872.52 G. Munda, Social multi-criteria evaluation:

E. Störmer, Focus on actors initial experiences with system constellations in theory-based evaluations, Z. Eval. 7 (1)( 2008) 35 73.61 J. Mayers, Stakeholder power

the main emphasis being balanced in evaluation of the short-and long-term impacts of new technologies. All these future-oriented approaches try to illustrate

and evaluation in order to assess whether the foresight process has helped to achieve its objectives. The nature of foresight methodologies varies from creative to the evidence based and from expertise to interaction based working methods.

and a method for risk estimation and evaluation. The seasonal plan, for instance, aims at stimulating the risk identification process by listing the normal seasonal routines of the hydropower plant,

In the context of the Nordic hydropower production and distribution, for instance, the most threatening scenarios are selected for the risk estimation and evaluation process.

Hazard/risk estimation and evaluation phases, in turn, are identified phases where risks are estimated and evaluated by modelling them either quantitatively, semi-quantitatively or qualitatively.

/Technological forecasting & Social Change 76 (2009) 1163 1176 Fig. 1. However, the normal monitoring and evaluation actions are conducted

Time horizon 0 5 years 5 50 years Phases Scope definition, risk identification, risk estimation (probability, consequences), risk evaluation The pre foresight phase

Risk analysis of Technological Systems, Geneva, Switzerland, 1995.21 Center for Chemical process Safety (CCPS), Guidelines for Hazard Evaluation Procedures, 2nd Ed. AICHE, New york, USA, 1992

For the evaluation of the topics, a set of criteria was worked out. This was the basis for an online survey but also the pre-judgements of the selection phase.

In the first evaluation part the relevancy of the topic area was judged upon. For this, we used a five-step scale from very important to unimportant.

‘Futur the German Research Dialogue',Research Evaluation, vol. 13,2004, pp. 143 153,3. 15 M. J. Bardecki, Participant's Response to the Delphi method:

Scenario planning Decision support Political and institutional context Evaluation 1. Introduction Policy-makers and business leaders often face strategic decisions with uncertain future outcomes.

evaluation and learning amongst environmental scenario practitioners and policy-makers about new forms of long-term strategic policy-making and their enabling conditions.

ex-post or midterm policy evaluation could make long-term scenario planning become a more integrated component of the policy cycle.

different ex-ante evaluations procedures have been integrated into one regulatory policy appraisal approach to improve the quality of regulation,

which several evaluation studies found as one weakness of efforts so far 41. Another entry point could be Strategic Environmental Assessment.

A combination of skills development, capacity-building, undertaking evaluation more regularly, as well as making better use of existing knowledge

and evaluations of outcomes depend on prior judgements about the relevant system. The resulting framework that emerges from CSH is a set of boundary concepts usable as a checklist of critical boundary questions.

and evaluations of outcomes depend on judgements about the whole system to be looked at and how that will be done.

which covered locating the responsibility of risk evaluation in the value chain, the ethics of promising (by researchers and firms),

and lack of 13 REACH regulations Registration, Evaluation, Authorisation and Restriction of Chemical substances (EC 1907/2006) which entered into force on 1 june 2007.

6. Evaluation and discussion These co-evolutionary scenarios can prepare the ground for discussion of complex potential radical technologies via the combination of endogenous futures, the IC+framework and deep case research into actors

The evaluation of my workshops for Frontiers show that learning about other perspectives occurs, however it does not yet mean that the scenario method has proved practical in the long term (in the practices of the participants),

As is already clear from the evaluation of the workshop such scenarios support strategic anticipation.

Evaluation of national foresight activities: assessing rationale, process and impact. Technological forecasting and Social Change 73: 761 77.

including an evaluation of ongoing FNR programmes; the collection of statistical data and bibliometric data;

Setting Context/Identifying Priorities Data collection Bibliometrics Interviews International research trends Evaluation of FNR programmes Mapping of Lux.

where evaluations are supposed to feed back into the policy formation and implementation phases. Already in these cycle models, policy learning is seen as an essential ingredient of political governance.

and Weber (2006) and in particular the work on self-evaluation tools for foresight in the context of the Forsociety ERA NET (Forsociety 2007).

Self-evaluation tool for foresight project managers. http://www. eranet-forsociety. net (accessed 19 november 2007. Freeman, C. 1991.

Evaluation of the United kingdom foresight programme. Final report. Manchester: University of Manchester. Rhomberg, W c. Steindl, and M. Weber. 2006.

but signals a recognition that the consideraatio of counterfactual possibilities is intrinsic to reflexivity in the context of risk assessment and evaluation'.

Organising the present's futures towards an evaluation of foresight, knowledge flows and the coordination of innovation. http://www. iesam. csic. es/proyecto/formwp1. pdf (accessed September 2009).

Research Evaluation 13, no. 3: 143 53. Cunliffe, A. 2005. The need for reflexivity in public administration.

Axes of balance in Finnsight Although formal evaluation of Finnsight has been carried out, the panellists were asked

and the UK Research councils in that it allocates funding to high-quality basic research based on the evaluation of submitted research proposal.

Evaluation of national foresight activities: assessing rational, process and impact. Technological foresight and Social Change 73, no. 7: 761 77.

but crossed this with an interest in the evaluation of impacts and in the use of FTA in two domains, business and higher education 9, 10.

in part through improved monitoring and evaluation, and through preparedness to address global problems. It was against this history that the third conference from

and evaluation to tailor approaches to policy objectives (8, Forsociety). In order to improve Foresight impact on policy strategy building it has been proposed to complement collective Foresight processes with a strategic counselling phase where the outcomes are translated into strategic choices 9. To sum up,

Premature evaluation: The Scan process benefits from experienced participants, so the early meetings can seem ambiguous, unfocused, and unproductive.!

Project Outcomes Future perspectives Management Stakeholder engagement Nordic ICT Foresighta 17 Informative Evaluations of key ICT applications, Nordic scenario set in context of ICT development, scenario

and explicated multiple perspectives in the evaluation and analysis of these ideas. Drawing upon these lines of thinking,

There is a clear need for further research on evaluation of foresight impacts not only with the purpose of doing the evaluation of a project

but mainly to draw conclusions on how foresight can be improved as an instrument contributing to knowledge creation for policy and decision-making in more general.

The conceptual and empirical work on the evaluation of T. Ko nno la et al.//Futures 43 (2011) 252 264 263 foresight is ever more important to position foresight as one of the key supporting tools for policy-making to anticipate how major societal challenges ahead can be addressed such as those tackled in this paper, e g. security, sustainability and information society challenges.

377.5 L. Georghiou, M. Keenan, Evaluation of national foresight activities, assessing rationale, process and impact. technological forecasting and social change, Technological forecasting and Social Change 73 (7)( 2005) 761 777.6 O. Da Costa, P. Warnke, C. Cagnin, F. Scapolo

Similar conflicting forces will affect many smaller-scale FTA ACTIVITIES, in private organisations as well as in the policy sphere. 4 DEMATEL=decision making trial and evaluation laboratory, a structural modelling technique;

the (vi) assimilation and (vii) evaluation of this understanding to produce a commitment to action in a particular organisation.

and key technologies approaches coming to the fore, alongside other tools such as research evaluation and impact assessment.

This also raises the need for board evaluation measures against corporate and societal objectives as well as on the effect of stakeholders'communication and participation on management, transparency, performance (economic, environmental and social),

Based on experience of formal evaluation of foresight programs, it was claimed that‘‘lack of success had very little to do with the quality of the work that has been done and much more to do with initial and subsequent political positioning''.

'An important lesson for strengthening the impact of fta was drawn fromthe evaluation literature. In this field a distinction has been developed between summative evaluation,

focussed on measuring impact, and normative evaluation, which ismore concernedwith howto improve impact. While bettermetrics are obviously useful,

and should be pursued, there is a case that at this stage of the evolution of FTA,

sustainability and foresight research by applying monitoring and evaluation systems, Delphi survey, SWOT analysis and scenario development methods in various contexts over the last ten years.

Therefore, the evaluation of knowns surely can be considered as being the main purpose of MCA or MCDA;

''Subjective questioning of evaluation should be‘‘objectivised''.''This is difficult, for example, if social phenomena are analysed as, in general,

Parthasarati, P. and Levinson, D. 2010),‘Post-construction evaluation of traffic forecast studies'',Transport Policy, Vol. 17, pp. 428-43.

Ridgeway, J.,Zawojewski, J. S. and Hoover, M. N. 2000),‘Problematising evidence-based policy and practice'',Evaluation and Research in Education, Vol. 14 Nos

External and internal evaluations The Government Foresight report on Long-term Climate and Energy Policy was completed and approved in the government and by Parliament in October 2009.

A year after that, the government commissioned an evaluation on the effectiveness of the preparation process that would comment on the general progress of the process as well as the usability of this kind of foresight work in policy planning.

This internal evaluation was conducted by the Prime minister's office. One has to bear in mind that the evaluation concerned the whole foresight report,

of which the FFRC's scenario work represented only one part. The Prime minister's Office also wanted to publish the scenario process as such (Lauttama ki and Heinonen

All in all, three evaluations of the government futures report were conducted where the scenario process was discussed also:

one internal evaluation as mentioned above, and two external evaluations one being the official statement given by the Committee for the Future in Parliament (2011),

and another one by an expert (Wilenius, 2011). The Committee for the Future (2011) supported the government foresight report.

In the evaluation of government foresight report by Wilenius (2011), attention was drawn to the fact that the government foresight report could have dug deeper into the economic implications of climate and energy policy

the statements of evaluation are encouraging. In the evaluation, especially the scenario work is PAGE 312 jforesight jvol. 14 NO. 4 2012 acknowledged for bringing forward genuinely varied possibilities to achieve the emission targets.

Another mention concerning FFRC's input speaks of the role of expert workshops which in the evaluation are thought to be a very effective tool to find the right questions

and seek possible answers to these questions. Prime minister's Office organised a gathering on March 29,

Third, the ETC group is working to develop an International Convention for the Evaluation of New technologies (ICENT),

Public management should be able to internalise environmental costs through sophisticated environmental evaluation tools. Functional implications of Scenario B (2025.

While RIA evaluations usually employ indicators case studies and surveys as the most commonly used approaches (Blind, 2006), FTA would add to this methodological list other approaches that could render the evaluation of current laws,

as well as the process of replacing them with new ones, a more reliable and evidence-based procedure.

as well as the performance of ex post evaluation of regulations (which is part of the progress development of regulatory policies,

complementing ex ante evaluations (Organisation for Economic Co-operation and Development, 2003. Ex ante impact assessment is required to check all possible impact dimensions

drawing attention to the focal points of needed research in this field. 4. 2. 1 Evaluation, assessment and quality measurement.

ex post evaluation of regulatory policies'',Proceedings of the OECD Expert Meeting, September, Organisation for Economic Co-operation and Development, Paris. Porter, A l. and Cunningham, S w. 2005

because this would imply exclusion rules independent from a scientific evaluation of the fundamental working principles of a functional system described by the three criteria.

or distinctions and often using varying terminology the following basic elements (1) definition of task and system (2) analysis of technology, their applications and framework (3) impact assessment (4) evaluation and development

Scenarios are combined usually with the formulation and evaluation of strategic options, as they are seen as providing information inputs essential for assessing the profitability of curreen

and scope Planning Planning Planning D Run the business Inventory analysis Implementation and operation Accounting Delivery C Monitor the business Impact evaluation Verification and corrective action Auditing

Evaluation of Laboratory directed research and development investment areas at Sandia Kevin W. Boyack*,Nabeel Rahal Sandia National Laboratories, 1 P o box 5800, Albuquerque, NM 87185, United states

This analysis added value in that the IA leads obtained information that assisted them in the evaluation and redirection of their R&d activities.

Sci. 16 (4)( 1998) 322 346.8 K. W. Boyack, K. Bfrner, Indicator-assisted evaluation and funding of research:

First, impact evaluation was used to gain a systematic view of the past (see Halonen, Kallio, and Saari 2010.

This integrated process was called learning by foresight and evaluation (with the acronym LIFE, learning by foresight and evaluation;

and evaluation) process. for new services and service providers. In this knowledge space, the project fostered capacities for linking and sharing existing knowledge internally,

and impact evaluation. She has an MSC in Applied Geography from La Sapienza University of Rome.

The face-tofaac interview with him provided input to allow a first evaluation of our analyses.

4. 7. Calls to perform technology assessment (Step H) Much of the FIP process serves to promote the first type of technology assessment evaluation of competing technologies.

and CCS 08-C Lifestyle and environment 08-G Evaluation of and countermeasures to global warming 08-I Technology for urban waste minimization/material circulation for environmental conservation

Originality/value For the first time the paper presents an analysis of Russian foresight projects connected to the natural resources area and an evaluation of their influence on policy decision making.

B evaluation of the position of Russian R&d centres in the international S&t landscape; and B determination of the most relevant steps to be taken to support the development and commercialisation of science and technology.

2030 (FS2) The aim of this study was an evaluation of required resources and possible innovation projects for the most important S&t fields.

The FS2 criteria for the evaluation of technology groups were identified mostly on the basis of the results of FS1 (see Table I). National S&t Foresight:

and innovation policy it is necessary to have more detailed and precise information about resource requirements, the evaluation of emerging markets,

and Table III The interrelation of the projects Research focus Main results Questions for further investigation FS1 The evaluation of topics by the following criteria:

and increasing the budget for R&d funding and training The evaluation of required resources (human, financial and infrastructure) for the most important and well-developed topics and their market prospects FS2 The evaluation of technology groups (as combinations of the most important and well-developed topic on the basis of the results of FS1) by the following criteria:

contribution to innovative relevant projects, availability of human resources (researchers, engineers and technicians), required level of funding, fixed assets and others Estimation of resources and risks List

which the relative evaluation of this influence is given. 5. Conclusion The problems and major challenges that the natural resources sector is facing,

Table IV The influence of the foresight studies on policy decision-making Influence on policy-making Evaluation of influence on policy-making FS1 The foresight data were used as an information source for many political purposes:

The results of the FS2 project included the evaluation of technology groups and the identification of those that had the best prospects and conditions for further advancement;

of Science and Technology policy (NISTEP) Foresight and Strategy Planning Team, Korean Institute of S&t Evaluation and Planning (KISTEP), The Science

a re-evaluation of research and theory'',Technological forecasting and Social Change, Vol. 39, pp. 235-51.

Yet, evaluation on methods is important in order to identify‘good practices 'and to contribute to the robustness of FTA, in terms of process, content, and impacts.

time horizon15) and provide both methodological and evaluation guidelines. The experience of the European foresight Platform (www. foresight-platform. eu) could provide some relevant inputs towards this endeavour.

these methodological and evaluation guidelines should explicitly account for the wide range of context-dependent choices,

Quant. 28 (3)( 1994) 315 327.10 V. J. Caracelli, J. C. Greene, Crafting mixed-method evaluation designs, in:

Advances in Mixed-method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms, Jossey-Bass, San francisco, 1997.11 P. Goodwin, Why hindsight can damage foresight, Int. J. Appl.

Public policy 37 (1)( 2010) 31 40.16 L. Georghiou, M. Keenan, Evaluation and impact of foresight, in: L. Georghiou, J. Cassingena Harper, M. Keenan,

67 T. Wehnert, W. Jörß, Evaluation paper: foresight and decision-making, in: Deliverable 3. 1 of EFONET Energy Foresight Network, 2009, Available at:

< Back - Next >

Overtext Web Module V3.0 Alpha
Copyright Semantic-Knowledge, 1994-2011