of Science and Technology policy (NISTEP) Foresight and Strategy Planning Team, Korean Institute of S&t Evaluation and Planning (KISTEP), The Science
a re-evaluation of research and theory'',Technological forecasting and Social Change, Vol. 39, pp. 235-51.
Yet, evaluation on methods is important in order to identify‘good practices 'and to contribute to the robustness of FTA, in terms of process, content, and impacts.
time horizon15) and provide both methodological and evaluation guidelines. The experience of the European foresight Platform (www. foresight-platform. eu) could provide some relevant inputs towards this endeavour.
these methodological and evaluation guidelines should explicitly account for the wide range of context-dependent choices,
Quant. 28 (3)( 1994) 315 327.10 V. J. Caracelli, J. C. Greene, Crafting mixed-method evaluation designs, in:
Advances in Mixed-method Evaluation: The Challenges and Benefits of Integrating Diverse Paradigms, Jossey-Bass, San francisco, 1997.11 P. Goodwin, Why hindsight can damage foresight, Int. J. Appl.
Public policy 37 (1)( 2010) 31 40.16 L. Georghiou, M. Keenan, Evaluation and impact of foresight, in: L. Georghiou, J. Cassingena Harper, M. Keenan,
67 T. Wehnert, W. Jörß, Evaluation paper: foresight and decision-making, in: Deliverable 3. 1 of EFONET Energy Foresight Network, 2009, Available at:
, Study on indicator system for core patent documents evaluation, in: Proceedings of ISSI 2009-The 12th International Conference of the International Society for Scientometrics and Informetrics, Rio de janeiro, Brazil, 2009, pp. 154 164.17 C. Zhang, D. H
and outcome evaluations in terms of likelihood 16. Hence all alternative system models, plausible scenarios, and evaluations require consideration, without exception,
and none should be treated as the single best model representation, true scenario, or correct evaluation.
It is clear that there is a strong need for policy-making approaches that allow for dealing with deep uncertainty,
i e. with many different kinds of uncertainties, multiple models, a multiplicity of plausible scenarios and evaluations of these scenarios 17.
and evaluation criteria are contested (e g. noise and emissions versus economic benefits) 16. EMA offers a suitable technique to explore the potential implications of these uncertainties
the workshops allowed for a systematic ex-post evaluation of similarities and differences between the chosen scenario projects.
A running theme in the world of FTA is need the to address evaluation and impact.
But there are shared issues with the wider discipline of evaluation in dealing with time-lag between action
His research interests are demand-side innovation policy, foresight and evaluation. He is active in policy advice to governments and business and as Vice-president for Research and Innovation at the University of Manchester.
Empirically, the section is based on publicly available reports, in particular, the Research2015 project web page and evaluation report.
Denmark has established a well tradition of policy evaluation, which also includes science and innovation policies. The combination of the widespread use of policy evaluations and the DBT's parliamentary technology assessment might have constituted a platform for policy making that would make the need for foresight less urgent.
But this context has changed during the recent decade. Based on the Danish Globalisation Strategy from 2005, public expenditures on R&d have increased.
The evaluation found that 64%of all proposals came from the public research and education sector,
The evaluation of the process revealed that the material from the OECD horizon scan was used only to a very limited degree.
or 5 6%.An overview of the whole process is presented in Table 3. The evaluation of the Research2015 process may be considered a fifth phase.
The evaluation focused on both the research process and the result of the process 33.
recent experiences and future perspectives, Research Evaluation 19 june (2))(2010) 91 104.4 M. Keenan, R. Barre',C. Cagnin, Future-oriented technology analysis:
, agenda-setting, policy preparation, decision-making, implementation, and evaluation. In most evaluative studies on foresight, the added value is addressed from a‘supply-driven'point of view, by
In this paper, the evaluation is approached from a demand-driven perspective, in other words, from a user perspective i e.,
policy implementation and policy evaluation (see 8). Indeed, according to literature, scenario analysis methods can be used in the different phases of the policy cycle 1, 2, 4, 6, 9 11.
We used an independent evaluation that was conducted in 2010 by the University of Twente 21, based on interviews with 21 politicians and civil servants and a document analysis. Furthermore,
in the local studies, the evaluation of futures research methods was considered predominantly in view of how they are used for
whereas the evaluation in the national studies the evaluation was more about how users applied these methods
Wie geen vraag heeft zoekt geen antwoord,(Evaluation of Trendbureau Overijssel. Who has no Question Does Not Seek for an Answer
The conclusions provide a brief critical evaluation of the methods and an identification of requirements for future research.
and outlook. 2. FTA evaluation, impact and learning Another very important theme addressed by the FTA 2006 International Seminar relates to FTA evaluation, impact and learning.
The objective of tackling this issue was to address sponsor concerns for better accounts of demonstrable impacts.
The logic behind this theme was to recognise the work already done in closely related areas, such as programme evaluation, futures studies, planning,
The ideas were ranked then on the basis of these evaluations and estimations. 3. 1. 2. Results In total,
Although the presented combination of a large-scale survey with an expert evaluation as such is not new,
monitoring and evaluation of two new strategic indicators: community satisfaction and participation in decision making. The former increased from 46%into 72,
Also, such a system shall enable the development of associated collective and individual strategies with related action plans as well as monitoring and evaluation systems.
The subsequent case-specific discussions are followed by a cross-case evaluation. The article finishes with concluding remarks. 2. Toward networked foresight 2. 1. Analogies in the development of innovation management and futures research Liyanage 13, Niosi 14 and Ortt and van der Duin 15
In the case evaluations three different levels are used (visualized as gray-shading) for each component of the CIM to visualize their preference concerning openness and network orientation of futures research activities.
and their type is stated. 5. Evaluation of the cases In this section, we evaluate the three cases,
followed by a cross-case evaluation. 5. 1. Case 1: Rijkswaterstaat WINN The vision of the WINN program has developed into
evaluation, generation, planning, and deployment of business modeling concepts in yet underexplored business fields Project, singular activity 3. 6 Technology transfer program Increases the transfer activities from academia to business by detecting, stimulating
and Deltares (Table 6). Based on the CIM evaluation and the analyzed foresight activities the following conclusions can be drawn for Rijkswaterstaat:
Based on the CIM evaluation and the analyzed foresight activities the following conclusions can be drawn for EICT:
Based on the CIM evaluation and the futures research activity analysis the following conclusions can be drawn for the EIT ICT Labs:
and guide the future of the network. 5. 4. Cross-case evaluation 5. 4. 1. Toward networked foresight within the three cases In Fig. 5 the classification of the foresight activities in terms scope
and opportunities---4. Evaluation and weighing of problems and opportunities List P---List O---1. Literature study 2. Consultation of national international experts 6. Formation of clusters
Evaluation, Impact and Learning, Anchor paper Theme 2. Paper presented at Second joint horizon scanning Science and Public policy February 2010 18 International Seville Seminar on Future-oriented technology analysis:
one of the objectives of the research was to look at how the practitioners of foresight defined primary program success. After a comprehensive reviie of foresight evaluation
and evaluation in generral Barre and Keenan (2006) concluded that an Jonathan Calof is recognized as one of the leaders in intelliggenc and business insight.
Critical success factors for government-led foresight Science and Public policy February 2010 33 evaluation of future-oriented technology analysis (FTA) should be based upon an assessment of foresiigh quality in terms of the conjectures produced,
Conceptually, this would be multifaceted a complex evaluation based on the broad impacts of an FTA PROCESS. Similarly, Amanatidou and Guy (2006) have pointed out the direct and indirect benefits of FTA:
i e. more comprehensive policy content, more evaluation of policy through debate and higher perception of democratic legitimacy Decisions taken:
Evaluation, impact and learning. Paper presented at the Second International Seville Seminar on Future-oriented technology analysis, held 28 29 september 2006, Seville, Spain.
interdisciplinary research on quality of experience (Qoe) and quality of service (Qos) in mobile media environments, evaluation of user-and futureorieente innovation techniques in the ICT domain,
interactive applicatioons personalization, recommendation systems and evaluation of Qoe and quality of service (Qos) in mobile environments.
or only the final stage (e g. usability testing, evaluation etc.)(Haddon et al. 2005). ) However, the benefits of involving users continuously have already been investigated:
Instead, users should be involved from the idea stage right up to the postlauunc evaluation stage. Furthermore, as userdriive innovation deals with those user insights (needs, expectations etc.
Finally, the third stage (test market and pilotinng consisted of an evaluation of the results from the second stage on an individual application level
and evaluation Innovation development and production Test market and piloting Launch Adoption diffusion User diffusion,
evaluation Figure 1. Schematic overview of the three research phases User involvement in future technology analysis Science and Public policy February 2010 56 activities and finds it difficult to identify with the life
Concept evaluation Next, all the applications considered in the first phase were transformed into workable paper conceept
For this concept evaluation phase, we conducted a major survey among the i-City panel (n=420.
The advantage of such an evaluation is the panel's familiarity with mobile city concepts and experience with actual working applicatiions In total,
focusing on the evaluation of Qoe in a mobile living lab settiing In this context, we developed a five-step interdiscipplinar approach for linking Qoe to Qos parameters in living lab environments.
Drawiin on results selected from three distinct points of user involvement in the process of developing new products (i e. opportunity identification, concept evaluation and test market),
an evaluation based on a matching approach Spyros Arvanitis, Laurent Donzé and Nora Sydow This paper investigates the impact of the promotional activities of The swiss Commission of Technology
which were used first in labour market evaluations, have also been applied to evaluate the technology programmes of European countries (see Almus and Czarnitzki,
To the best of our knowledge, it is unique in Europe as a main promotional policy. 2 Methods of evaluation of measures of technology policy Evaluating the outcomes of subsidized projects is difficult,
evaluations of outcomes, i e. estimations of the impact of policy, proceed by means of an ex post assessment of the activities of the firms that have received subsidies.
There are several empirical strategies for mitigatiin selection bias in the ex post evaluations e g. regresssio with controls for unobserved effects;
in order to avoid ex ante selection bias would be to undertake an evaluation by awarding grants (subsidies) randooml within a pool of actors who are judged suitabbl for funding (Jaffe, 2002).
and Feller (2007) for recent reviews of the central issues related to the evaluation of the effectiveness of technology programmes.
See also Science and Public policy (34 (10), 679 752) dedicaate to‘New frontiers in evaluation'.'Finally, see OECD (2006a) for an analysis more from the point of view of the policy-maker;
2001) for the role of framework conditions for the evaluation of industry university collaboratioons and Polt and Streicher (2005) for the evaluation of large programmes such as the Framework programmes of the Europeea Union. 2. For overviews of Swiss
Evaluation methodologies, econometric models: microeconometric models. In RTD Evaluatiio Tool Box: Socioeconomic Evaluation of Public RTD Policies (EPUB), W Polt, J. Rojo, A Tübke, G Fahrenkrog and K Zinöcker (eds.
pp 101 118. Vienna: European commission. Arvanitis, S, H Hollenstein and S Lenz 2002. The effectiveness of government promotion of advanced manufacturing technologiie (AMT:
An evaluation of the effects of R&d subsidies. Economics of Innovation and New technology, 9 (2), 111 148.
Mapping the frontiers of evaluation of public-sector R&d programmes. Science and Public policy, 34 (10), 681 690.
Building programme evaluation into the design of public research-support programmes. Oxford Review of Econoomi Policy, 18 (1), 22 34.
Microeconometric evaluation studies. Research policy, 29 (4/5), 471 495. Lach, S 2002. Do R&d subsidies stimulate
Evaluation of government-sponsored R&d consortia in Japan. In Policy Evaluation in Innovation and Technology Towards Best Practices, pp 225 253.
OECD: Paris. Silverman, R 1986. Density Estimation. London: Chapman and Hall. Wallsten, S C 2000.
A comparative analysis is provided as well as a brief evaluation of meeting the needs of policy-makers to identify areas to intervene in by formulating appropriate policy.
strategic evaluation of large programs and projects; information and knowledge diffusion. It is considdere to be an interface organization in the ST&I environmment having responsibility for articulating the views of the government, private sector and academia,
and strategic evaluation studies, mobilizing more than 2000 experts per annum from an average of 300 institutions.
This paper will focus on strategic foresight, taking advantaag of CGEE's background in conducting strategic foresiigh exercises and strategic evaluation studies during the past decade, all of
Lee, S.-J. and Hua, J. 2004)‘ The economic evaluation of the food industry using biotechnology'(.
3. Evaluation of joint programmes; 4. Funding of cross-border research by national or regional authorities;
I.,Acheson, H.,Borra's, S.,Halle'n, A.,Maimets, T.,Mustonen, R.,Raffler, H.,Swings, J.-P. and Ylihonko, K. 2010)‘ Interim evaluation of the Seventh
A systems world needs systems evaluations',Research Evaluation, 13:3 17. Bach, L. and Matt, M. 2005)‘ From economic foundations to S&t policy tools:
exploring alternative futures and shaping the direction to follow, coupled with an evaluation of what has or has not been achieved from time to time to correct deviattion
The IPRM method combines roadmapping and the forward-looking evaluation of policy development paths. IPRM integrates the approach of technology roadmapping including such contents as enabling technologies, applications, products, markets and drivers with the perspectives of systemic policies and policy instruments.
Georghiou, L. and Keenan, M. 2006)‘ Evaluation of natioona foresight activities: Assessing rationale, process and impact',Technological forecasting and Social Change, 73: 761 77.
running and evaluation. For instance, in connection with joint programming (see Section 4. 3) six framework conditions have been defined:.
peer review procedures. forward-looking activities. evaluation of joint programmes. funding of cross-border research. optimum dissemination and use of research findings. protection,
and putting in place common schemes for monitoring and evaluation (67%).%co-ordination or clustering of ongoing nationally funded research projects (59%.
%generating multinational evaluation procedures (55%)(Matrix-Rambøll 2009. For the purpose of this paper we define five key functions in transnational research programming,
and evaluation of project performance Financial administration systems are designed not to cope with non-national contracts No shared points of contact/project ideas Differences in speed of implementation National differences in training for graduate students
references to OA in grant agreements) Problems of oversight related to research integrity Evaluation Evaluation of transnational programme refers to appropriateness, effectiveness and efficiency in execution of entire programme and its parts
programme level) of evaluation Timescales of national evaluation processes may vary considerably Differences in experience in monitoring,
evaluation and learning-based policy-making Lack of national support to fund longer-term collaboration in evaluation Embedding foresight in transnational research programming. 193 European experiences
evaluation and impact assessment are being discussed in Europe (European commission 2010a). At the level of innovation clusters the recently established first knowledge
and innovation communities (KICS) of the European Institute for Innovation and Technology (EIT) also face issues of governance, prioritisaatio and evaluation.
evaluation and overall legitimmac of the activity by way of allowing traceability of emerging jointly proposed themes or research topics.
and the evaluations of the projects, programmes and the instruments for transnational research programming should be conduccte by those with the most appropriate expertise.
External evaluation of the EIT (External Evaluation 2011. 15. Optimat et al. 2005) defines three types of dominant programme orientation strategies for national research systems in Europe:
External Evaluation. 2011) External Evaluation of the European Institute of Innovation and Technology (May 2011), Framework Contract on evaluation and related services (EAC 03/06), Final report on evaluation.<
<http://ec. europa. eu/dgs/education culture/evalreports/education/2011/eitreport en. pdf>accessed 15 march 2012. Gnamus, A. 2009) Comparative Report on S&t Cooperation of the ERA Countries with Brazil, India and Russia, JRC Scientific and Technical Reports, EUR 25022 EN.<
2009)‘ Evaluation and impact assessment of the ERA NET scheme and the related ERA NET actions under the 6th Framework programme Volume 1:
Evaluation for the European commission',<ftp://ftp. cordis. europa. eu/pub/fp7/docs/fp6-era net-evaluation-final-report-volume-1-q1-q5-d1
A comparative analysis is provided as well as a brief evaluation the needs of policy-makers if they are to identify areas in which policy needs to be formulated.
Building on the advantages and disadvantages of each tool and method, Section 4 evaluates the different approaches and tools based on evaluation criteria reflectiin the information needs of policy-makers about new and future opportunities and threats.
Evaluation of scanning approaches and methods The evaluation of the different approaches and methods faces several challenges.
Table 2). In this regard an evaluation across the different approaches and methods is difficult as their success is highly contextual.
Any useful evaluation of different methods based on the information needs of policy-makers should apply to methods
and methods presented in Fig. 2 certain combinations of methods can be created to provide a complete evaluation along the scanning process.
Table 3. Evaluation criteria for scanning approaches and methods Connections, clustering of weak signals and degree of relevance to a specific area Duration of weakness of signal, also associated with time at
conferences further outlook in literature Concepts and methods in horizon scanning. 217 Table 5. Analysis of three combined approaches across evaluation criteria (a) Semi-automated, participatory approach (b
Smith, J.,Cook, A. and Packer, C. 2010)‘ Evaluation criteria to assess the value of identification sources for horizon scanning',International Journal of Technology assessment in Health care, 26: 348 53.
The need to change the uses of essential natural resources by aligning all policy realms towards sustainabillity extending from policy design through implementtatio to evaluation..
Megatrends are used also as a part of the evaluation in market studies and analysis (Frost and Sullivan, 2010.
The evaluation panel consisted of senior Fraunhofer scientists and external experts. E5 million in funding for each challenge within the Future markets programme are to be spent (starting in 2011.
development and evaluation of new therapies for chronic skin diseases (b) Sterihealth: reducing the potential infection in hospitals, doctor's offices and care of the aged by new technologies for highly efficient on-site sterilisation.
< Back - Next >
Overtext Web Module V3.0 Alpha
Copyright Semantic-Knowledge, 1994-2011