Data-Driven R&D&I Management for Societal Impacts: Introduction and Application of AgroRadarEval
| Páginas | 89-111 |
| Fecha | 23 Diciembre 2024 |
| Fecha de publicación | 23 Diciembre 2024 |
| Autor | Daniela Maciel,Adriana Bin,Marie Ferré,James A Turner,Geraldo Stachetti Rodrigues,María Magdalena Costa,Miguel Sierra Pereiro,Juan Mechelk,Aurelle de Romemont,Kevin Heaune |
| Materia | Ciencias sociales |
J. Technol. Manag. Innov. 2024. Volume 19, Issue 4
ISSN: 0718-2724. (http://jotmi.org)
Journal of Technology Management & Innovation © Universidad Alberto Hurtado, Facultad de Economía y Negocios.
89
Data-Driven R&D&I Management for Societal Impacts:
Introduction and Application of AgroRadarEval
Daniela Maciel Pinto1* , Adriana Bin2 , Marie Ferré3 , James A Turner4 ,
Geraldo Stachetti Rodrigues5 , María Magdalena Costa6 , Miguel Sierra Pereiro6 , Juan Mechelk6 ,
Aurelle de Romemont7, Kevin Heanue8
Abstract: Recognizing evaluation results as a crucial source of information to support RD&I management, this article introduces ‘AgroRadarEval’,
an interactive tool aimed at fullling theoretical, conceptual, and practical gaps concerning the systematization of the use of evaluation results
in agricultural RD&I. Aligned with the principles of Responsible Research and Innovation (RRI) and Responsible Research Assessment (RRA),
AgroRadarEval aims to support leaders and managers of RD&I in reecting on the strengths and weaknesses of organizational capacities, culture,
collaborations, processes, and communications that underlie the use of evaluation results in agricultural RD&I. AgroRadarEval is built along three
support pillars: Evaluation Capacity Building, Impact-Oriented Evaluation Culture, and Reective Learning, and is operationalized through eight
interconnected dimensions: 1. participation and collaboration, 2. skills development, 3. promotion of an evaluation culture, 4. continuous feed-
back and adaptation, 5. integration with strategic planning, 6. monitoring, 7. inuences of the external environment, and 8. communication. is
study describes the development of the tool, its characteristics, illustrating its application in an agricultural RD&I organization. e study is target-
ed at leaders and managers of agricultural RD&I, evaluators, and researchers interested in research evaluation and enhancing the impact of RD&I.
Keywords: Use of evaluations results; Agricultural R&D management; Research Impact; Organizational management; Impact Assessment.
Submitted: October 21, 2024 / Approved: December 16, 2024
(1) Embrapa Territorial. Scientific and Technological Policy (PPG-PCT), University of Campinas, Brazil
(2) School of Applied Sciences, University of Campinas, Brazil
(3) French Agricultural Research Centre for International Development (Cirad), Montpellier, France.
(4) Ethical Agriculture, AgResearch Ltd, New Zealand
(5) Embrapa Meio Ambiente, SP, Brazil
(6) Instituto Nacional de Investigación Agropecuaria (INIA), Parque Tecnológico LATU, Montevideo, Uruguay
(7) French Agricultural Research Centre for International Development (Cirad), Paris, France
(8) Evaluation Unit, Teagasc Mellows Campus Athenry, Co. Galway, Ireland
*Corresponding author: daniela.maciel@embrapa.br
1. Introduction
Expectations for research, particularly that carried out by public in-
stitutions, have grown, especially regarding their capacity to generate
innovations and promote socially relevant changes (Reed et al., 2021,
2022; Spaapen, 2015). is reects a growing concern with social re-
sponsibility, ethics, and transparency in the use of public resources
for science and technology (S&T) and research, development, and in-
novation (RD&I) activities, marking a paradigm shi that encourages
the adoption of the principles of Responsible Research and Innova-
tion (RRI) and Responsible Research Assessment (RRA) (Curry et al.,
2022; Schuij & Dijkstra, 2020). e principles of RRI aim to ensure
that RD&I processes are conducted ethically, inclusively, and in a way
that benets society, encouraging reection on economic, social, and
environmental impacts (Felt, 2018). Complementarily, RRA focuses
on ensuring that research evaluations adopt appropriate methods and
diverse indicators, integrating the results into research agendas to
guide RD&I strategies. Together, RRI and RRA create a framework
where research processes and their evaluation work synergistically
to ensure accountability, societal alignment, and transformative im-
pacts.
In light of this, impact evaluation emerges as an indispensable tool that
embodies the principles of RRI and RRA by providing evidence of both
the actual and potential transformations brought about by research. It
measures and demonstrates whether expectations have been met or
exceeded, while promoting accountability, inclusivity, and alignment
with societal needs (Carley & Bustelo, 2019; Reinertsen et al., 2022).
Moreover, this evaluation provides a snapshot of the real eects of
interventions, functioning as an information source that allows for
strategic adjustments to ensure that research remains aligned with so-
cial demands (Lee et al., 2020). us, it can serve both to measure the
generated eects and to support organisational planning and learning,
indicating opportunities to optimise future results based on present ev-
idence (Mackay & Horton, 2003).
In the context of agricultural research, maximising the use of these
evaluation results is particularly relevant. In this eld, impact evalu-
ation has a long history, dating back to the 19th century (Evenson et
al., 1979) and being formalised in the 1950s (Colinet, 2021). Original-
ly focused on the economic impacts of research (Alston et al., 1995;
Evenson, 2001), these evaluations have come to include environmental,
social, cultural, and political impacts, thereby increasing their informa-
tional value for RD&I management (Horton & Mackay, 2003).
J. Technol. Manag. Innov. 2024. Volume 19, Issue 4
ISSN: 0718-2724. (http://jotmi.org)
Journal of Technology Management & Innovation © Universidad Alberto Hurtado, Facultad de Economía y Negocios.
90
Despite this evolution, the use of evaluation results by agricultural
RD&I institutions remains limited, more oen directed at accountabili-
ty than at organisational learning and strategic management (Joly et al.,
2016). is limitation highlights the absence of practical frameworks
that can operationalise RRI and RRA principles in agriculture, guid-
ing organisations in leveraging evaluation evidence to address societal
challenges. In part, one reason for this limitation is that the discussion
on the use of results in the eld of research evaluation is relatively re-
cent (Milzow et al., 2019; Van der Most, 2010). While the topic has
been addressed in programme and public policy evaluation since the
1960s, with theoretical, conceptual, and practical contributions (Alkin
& Christie, 2004, 2023), in research evaluation there is a clear lack of
systematised and structured approaches. In light of this, Lee (2010) em-
phasises the urgency of developing systematic frameworks to expand
the use of evaluation results in research, which is corroborated by Pinto
and Bin (2024).
With a focus on lling this gap, this study presents AgroRadarEval, a
data-driven tool that directly operationalises the principles of RRI and
RRA. Designed to support the management and leadership of agricul-
tural RD&I organisations, AgroRadarEval enables the systematic use
of evaluation results to align research with societal demands, fostering
ethical, inclusive, and impactful practices. Considering the principles
of RRI and RRA, the tool is grounded in theoretical and conceptual
pillars and operationalised through dimensions and questions to be
considered at the organisational level, with the aim of informing RD&I
strategies and promoting research that is more responsive to societal
demands, based on evaluation evidence. e question that guided the
development of AgroRadarEval was: “How can the use of impact eval-
uation results be systematised to maximise the social impact of agricul-
tural research?”
2. eoretical Background
State of the Art on the Use of Evaluation Results
is section highlights briey how discussions on the use of evaluation
results are organized in the elds of program and policy evaluation,
research evaluation, and agricultural research evaluation, and how they
are associated with current debates on impact-driven and socially re-
sponsive research, endorsed by the RRI and RRA movements.
Alkin and Christie (2004, 2023) organized the eld of program and
public policy evaluation into three main branches1, with the “Branch
of Use” focusing on the debate surrounding the use of evaluation re-
sults. is structure goes beyond theoretical discussions, encompassing
practical frameworks as well. For example, Stuebeam’s Context, In-
put Process Product (CIPP) model (Stuebeam, 1983, 2002) provides
a comprehensive approach to use-oriented evaluations, while Preskill
and Boyle (2008) propose Evaluation Capacity Building (ECB) with a
strong emphasis on use. ECB refers to the deliberate choice, by an or-
ganization, to develop, maintain, and enhance evaluation competencies
(Bourgeois & Cousins, 2013; Cousins et al., 2009).
In this branch, both theoretical and practical developments high-
light the need for guidelines that promote feasible evaluations, and
the capacity to assess an action (Rutman & Wholey, 1980) to support
decision-making by program managers and policymakers. Patton’s
“Utilization-Focused Evaluation (UFE)” expands the perspective on
use-oriented evaluations, highlighting that they are not solely intend-
ed for policymakers but also for any potential user, positioning the
evaluator as a strategic actor in the evaluation process. However, this
approach is questioned by Alkin (Alkin et al., 1979) and Weiss (1972,
1998) who argue that the evaluator should not have the role of judging
the results data.
On the other hand, Cousins (2014) advocates for a participatory ap-
proach at all levels of evaluation to ensure an eective connection be-
tween evaluation results and their users. At the same time, these authors
and others in the eld of program and policy evaluation explore evalu-
ation results in terms of organizational learning. Weiss and Chelimsky,
particularly, demonstrate the importance of evaluation results for the
continuous improvement of organizational practices and for strategic
decision-making (Chelimsky, 1977, 2015; Cousins & Leithwood, 1986;
Weiss, 1979).
e ‘Branch of Use’ is wide and encompasses aspects related to the
quality of the evaluation, including the choice of the methods and
the evaluator’s capacity to implement them eectively. It also includes
stakeholder involvement and the organizational capacity to conduct
impact evaluations, considering various levels of the organization that
may be involved in the process. is is particularly related to “ECB,
with models such as those by Preskill and Boyle (2008, 2009) and La-
bin et al. (2012), which seek to support organizations in strengthening
their evaluation capacity, ensuring that evaluation results are used to
maximize the positive impact on the evaluated programs and policies.
However, the ‘Branch of Use’ is fundamentally rooted in program and
policy evaluation. In the eld of research evaluation, as explored in the
following sections, there is a need to advance discussions on the use of
evaluation results by establishing elements that can foster this type of
context.
Use of Results from Research Impact Evaluation
In RD&I evaluation, an evolution in the use of results can be observed
in the last years (Pinto and Bin, 2024), shiing away from a focus on
accountability, to align with the principles of RRI and RRA. Beginning
in the 1990s with theoretical and empirical investigations focused on
the relationship between funding and accountability in RD&I (Hemlin,
1996; Horton, 1998; Luukkonen, 1995), the number of these types of
studies increased from the 2000s onwards.
During this period, studies started to investigate the use of results as
an element for organizational learning, with notable work related to
agricultural RD&I, specically linked to the Consultative Group on In-
ternational Agricultural Research (CGIAR) (Horton & Mackay, 2003;
Mackay & Horton, 2003; Patton & Horton, 2009). From 2010 onwards,
1e branches organized by Alkin and Christie are: Branch of Methods, Branch of Values, and Branch of Use.
Para continuar leyendo
Comienza GratisAccede a todo el contenido con una prueba gratuita de 7 días
Transforma tu investigación jurídica con vLex
-
Accede a resúmenes de sentencias generados por IA, que destacan al instante los aspectos jurídicos clave.
-
Realiza búsquedas avanzadas con opciones precisas de filtrado y ordenamiento.
-
Consulta contenido jurídico completo de más de 100 jurisdicciones, incluyendo una amplia colección de libros y revistas a texto completo.
-
Disfruta de funcionalidades exclusivas como comparador de versiones de una ley, notas de vigencia, análisis de citas, y mucho más.
-
Con la confianza de más de 2 millones de profesionales, incluidas las firmas más importantes del mundo.
Accede a todo el contenido con una prueba gratuita de 7 días
Transforma tu investigación jurídica con vLex
-
Accede a resúmenes de sentencias generados por IA, que destacan al instante los aspectos jurídicos clave.
-
Realiza búsquedas avanzadas con opciones precisas de filtrado y ordenamiento.
-
Consulta contenido jurídico completo de más de 100 jurisdicciones, incluyendo una amplia colección de libros y revistas a texto completo.
-
Disfruta de funcionalidades exclusivas como comparador de versiones de una ley, notas de vigencia, análisis de citas, y mucho más.
-
Con la confianza de más de 2 millones de profesionales, incluidas las firmas más importantes del mundo.
Accede a todo el contenido con una prueba gratuita de 7 días
Transforma tu investigación jurídica con vLex
-
Accede a resúmenes de sentencias generados por IA, que destacan al instante los aspectos jurídicos clave.
-
Realiza búsquedas avanzadas con opciones precisas de filtrado y ordenamiento.
-
Consulta contenido jurídico completo de más de 100 jurisdicciones, incluyendo una amplia colección de libros y revistas a texto completo.
-
Disfruta de funcionalidades exclusivas como comparador de versiones de una ley, notas de vigencia, análisis de citas, y mucho más.
-
Con la confianza de más de 2 millones de profesionales, incluidas las firmas más importantes del mundo.
Accede a todo el contenido con una prueba gratuita de 7 días
Transforma tu investigación jurídica con vLex
-
Accede a resúmenes de sentencias generados por IA, que destacan al instante los aspectos jurídicos clave.
-
Realiza búsquedas avanzadas con opciones precisas de filtrado y ordenamiento.
-
Consulta contenido jurídico completo de más de 100 jurisdicciones, incluyendo una amplia colección de libros y revistas a texto completo.
-
Disfruta de funcionalidades exclusivas como comparador de versiones de una ley, notas de vigencia, análisis de citas, y mucho más.
-
Con la confianza de más de 2 millones de profesionales, incluidas las firmas más importantes del mundo.
Accede a todo el contenido con una prueba gratuita de 7 días
Transforma tu investigación jurídica con vLex
-
Accede a resúmenes de sentencias generados por IA, que destacan al instante los aspectos jurídicos clave.
-
Realiza búsquedas avanzadas con opciones precisas de filtrado y ordenamiento.
-
Consulta contenido jurídico completo de más de 100 jurisdicciones, incluyendo una amplia colección de libros y revistas a texto completo.
-
Disfruta de funcionalidades exclusivas como comparador de versiones de una ley, notas de vigencia, análisis de citas, y mucho más.
-
Con la confianza de más de 2 millones de profesionales, incluidas las firmas más importantes del mundo.
