|
<<
mock proposal | Barend van der Meulen
R&D evaluation training
Not mock proposal
But real stuff: Twente R&D evaluation
course
Background & developments
Content
Proposals?
Proposals
Courses in other regions
[ Us | CEEC | Latin America ]
More specific courses: specify issues,
differentiate audience
Follow up course
Background
Output of R&D program
evaluation network sponsored by European
Commission
Then focus on program evaluation and training
of evaluators
Yet: broadened to R&D evaluation more
general and also on users/commissioners
of evaluations
Framework
Three and halve day
International
Key teachers from PREST, ISI and Twente
Balance of teaching, experiences of users,
and exercises
Covering R&D program evaluation, Institute
evaluation, policy interventions
Contents
Policy contexts of evaluation
Design of evaluation: logic chart
Catalogue of methods: panels, bibliometrics
socio-economic impacts, networks, ....
From findings to recommendations
Policy uses of evaluation
Future of challenges
Participation
Staff of research organisations
/ ministries / consultants / EU evaluators
From many European countries (last time
10 countries)
Max. 30 participants
Costs for participation ~ Dfl 3,500,-
($1,400) excl. travel, incl. lodgings
March 4-7, 2001, University
of Twente
Next course
<< mock proposal
| group one
NETWORKS
§ How to maximize synergy?
§ How to improve management and effectiveness
of S&T?
§ How to create virtual distributed networks
that bring together assessment, foresight,
evaluation, and other perspectives and
other disciplines to provide improved
analysis and advice?
§ Vision: innovation systems to increase
societal benefits
Key challenges to S&T
§ Increased internationalization
§ Changing character of research (e.g.increase
interdisciplinarity, complexity)
§ New interfaces between S&T research,
innovation)
§ Increasing effectiveness of S&T research
funding § Integration of human capital
improvement into S&T research innovation
system § Increased number and types of
participants and stakeholders
Approach
§ Increase capability
in analysis, evaluation, information processing
§ Increase linkages domestically/internationally
between experts in S&T, policy analysis,
evaluation, foresight
§ Exploit diversity of perspectives, experience
and capabilities, both domestically and
internationally
§ Create real and virtual platforms for
exchange of knowledge, good practice,
people, ideas and facilitate collaboration/education
and interaction of policymakers and managers
§ provide incentives for experts to participate
Next steps
§ Assemble key experts
in relevant areas:
§ Domestic/international
§ Different organizations
§ Academia
§ Industry
§ Government
§ NGO
§ Different disciplines [not exclusive
but need to build synergy among experts
first]
Mechanisms
§ Periodic conferences/workshops
§ Use existing groups
§ Sessions at annual meetings of relevant
professional societies
§ Creation of nodes as centers of collaborative/interactive
activities
§ Website
§ Discussion groups
§ Access to data
§ Register of experts
§ New, fasten routes to publications
<< mock proposal
| group two
How to make data sets/methods
and evaluation (research) results widely
available and easily accessed?
Pooling of results
Building
a facility: inventory, storage, access roads
Action: conduct an experiment
for one year
Road map: advisory group, one responsible
actor, evaluation by community after one
year
Accessibility of existing
data sets
Knowledge about existing
data sets
Access to organized, maintained public
data sets
Storage and access to evaluation project
data set
Issues: privacy and quality, control of
future use, cost of accessibility issues
Action: conduct a workshop to develop
a strategy
Methods development
Place where one can get
an overview about problems and a set of
available methods
Where to find different methods (access,
presentation, quality process)
Forum
where to discuss the need for new methods
Immediate action: identification of problems
requiring new methods (tour de table)
<< mock proposal
| group three
[ pdf version : 7 kb ]
|
Chains
of knowledge
|
|
|
How
did the fusion of fields over
time lead to new disciplines
and new ways of working and
innovations |
|
Aim
to understand the 20 year relevance
of basic research for the economy
and society |
|
Basic
research in biology, physics,
mathematics |
|
Policy
issue – how much into basic
areas vs problem-centred approaches
– portfolio management bearing
in mind private sector investment |
|
|
|
|
Tools and methods
|
|
|
|
Gratefully
accept the unlimited budget
on offer – need this for development
of tools and methods |
|
Hindsight/Traces
but using advanced data mining
techniques |
|
Looking
forward from science |
|
Everything
of importance in small number
of primary journals |
|
Data
mining actual contents for emergence
of key concepts |
|
Also
use conventional methods -use
SCI to look at citations to
those journals with possible
use of patent cites |
|
Also
group of experts to look for
seminal work on theory cf DOE
Decades of Discovery |
|
Look
for intersection between above |
|
Pay
more attention to dead-ends
and contributions from science
to science |
|
Look
for spill-overs in other fields |
|
|
|
|
"Human
capital aspect"
|
|
|
|
Human
capital aspect |
|
Trace
diffusion of knowledge into
curriculum |
|
Careers,
international diffusion of ideas.
Tracing – where did the people
trained in these programs go? |
|
|
|
Look
backward from innovation |
|
Group
of experts to look back over
past 30 years to look for most
significant science-based innovations |
|
Separate
team to explore nature of contribution
and intersection with forward
looking methodology |
|
|
|
|
Requirements
|
|
|
|
Project
requires international, multidisciplinary
team of evaluators and social
scientists |
|
Budget
for development and cleaning
of databases |
|
Taking
existing tools and modifying
them for these purposes |
|
Develop
to be appropriate for this area |
|
Case-studies
to focus on societal impacts |
|
Meta-case
analysis on previous cases |
|
|
|
|
"Success
metric for this project"
|
|
|
|
Success
metric for this project |
|
Develop
new understanding of use of
evaluation tools and methods |
|
Convergence/consistency
between forward/backward |
|
Non-evaluation
community regards results as
casting light on problem |
|
|
|
<< mock proposal
| group four
[ pdf version : 8 kb ]
|
Evaluating
Societal Impacts of Research
|
|
Proposals
|
|
|
|
Evaluating
social impacts of organ transplantation
research |
|
OT
research focussed on patients
and on organ donor, not e.g.
on effects of donor family,
on the idea of being organ donor,
registration effects etc. |
|
|
|
|
Comparing
Social Impacts
|
|
|
|
Applying
current social impacts in different
sectors to understand the social |
|
Lots
of tools for measuring impact
tools |
|
|
|
|
Synthesis
|
|
|
|
Social
impacts tools for understanding
and evaluating social impacts
of organ transplantation research |
|
Compare
situations were technology is
well developed and were it is
really new |
|
Level
of social consensus. |
|
Nature
of impacts: direct - unknown |
|
Resource
concentration and distribution |
|
|
|
|
|
|
Case
selection
|
|
|
kidney
transplantations |
|
eye
transplantation |
|
xeno-transplantations |
|
stem
cell transplantation |
|
heart
transplantation |
|
|
|
|
Tools
comparison/ evaluation and development
|
|
|
Delphi
related techniques |
|
Simulations |
|
Nominal
groups |
|
Citizens
panels and Science court |
|
Contingent
valuation |
|
Cross
impact analyses |
|
Convergence
analysis |
|
|
Results
|
|
|
Possibility
to compare at least two different
techniques in different contexts
and decide what to do. |
|
|
"This
is a mock proposal..."
|
|
|
|
This
is a mock proposal because NSF
is against health, so please
find another area |
|
Violence
and crime |
|
Coal
burning |
|
|
|
|
|
<< mock proposal
| group five
[ pdf version : 8 kb ]
Innovation Capacity Building
in a Federal System
S&T Policy is National
yet S&T activity takes place locally
Importance of local context
A variety of resources are required
What differentiates sticky versus slippery
places?
Defining Sticky vs Slippery
Regions
Focus
Technology-intensive, small firms
Linked and networked
Industry specific
Biotech?
Instruments?
Are there synergies?
Leveraging?
How are these resources distributed among
firms in a state?
Judith’s Metric
Capacity Building
Federal-State Synergies
Federal Programs
Mission Agencies
Grants
SBIR
Procurement
Government Labs
CRDAs
State Programs
University Programs
Workforce Training
Public Venture Capital
Business Assistance Program
Tax expenditures
Methodological Concerns
Case Studies
Aggregate Data – no problem
Firm Level Data
Need to differentiate small and medium
sized firms – small, single establishment
firms; small, multiple establishment
Advisory Board
Industry
State Officials
An cross-, multi-, inter - disciplinary
Review
|
