🏅This project has now been completed and the ISC and the Centre for Science Futures continues its outreach to ensure impact.
The imperative to re-think the ways that researchers and research outputs are evaluated and assessed is increasingly clear and urgent. Research evaluation regimes and practices are having wide-ranging, complex and ambiguous effects, including on the culture of research, the quality of evidence informing policymaking, priorities in research and research funding, individual career trajectories and researchers’ well-being. These issues play out differently across scientific disciplines and regional contexts. Further, open science frameworks and moves towards mission-oriented science are changing traditional ways of doing and communicating research, requiring new thinking on research evaluation and assessment.
Innovative and progressive approaches to responsible research assessment are being developed and promoted by certain higher education institutions and research funders in various countries and regions of the world. However, a concerted global initiative is required to mobilize research and research funding communities and higher education institutions to develop and adopt ways of evaluating, assessing and funding research that enable research to fulfil its role as a global public good and to address today’s challenges in more efficient, equitable, inclusive, and cooperative ways.
An international Scoping Group has been formed to explore the pathways to impact for this initiative. The Scoping Group met in June 2021 and March 2022 and conducted a series of regional consultations from October to November 2021.
Anticipated impact
- Comparative understanding of the functioning and effects of research evaluation and assessment systems and approaches around the world.
- Agreement on a context-sensitive assessment approach that enhances the value of research in serving the public good.
Key Milestone
✅The Future of Research Evaluation: A Synthesis of Current Debates and Developments was published in May by the ISC’s Centre for Science Futures.
The Future of Research Evaluation: A Synthesis of Current Debates and Developments
A dynamic and inclusive research system is profoundly important for both science and society to advance fundamental knowledge and understanding and to address increasingly urgent global challenges. See the recommendations from this important collaboration on the Future of Research Evaluation.
The Future of Research Evaluation gives a review of the current state of research evaluation systems and discusses the most recent actions, response and initiatives taken by different stakeholders through several case examples from around the world. The goal of this discussion paper is to contribute to the ongoing debates and open questions on the future of research evaluation.
A summary of the issues identified, actions taken and remaining open questions based on the report can be found in the infographic (click to view in detail):
Contact
- Dr Tracey Elliott, ISC Project Lead (tracey.elliott@council.science)
- Johanna Mogwitz, GYA member of the secretariat (johanna.mogwitz@globalyoungacademy.net)
- Dr Peter McGrath and Dr Ourania Kosti, joint IAP members of the secretariat (mcgrath@ictp.it, okosti@nas.edu)
- Professor Sarah de Rijcke, Chair, Scoping Group (s.de.rijcke@cwts.leidenuniv.nl)
Scoping group
- Robin Crewe studied at the Natal University in South Africa before obtaining his Ph.D. at the University of Georgia, USA. From 1986 to 1996 he was the Director of the Communication Biology Research Group of the University of the Witwatersrand. He was a Vice-Principal of the University of Pretoria from 2003 until his retirement from this position in June 2013. He is the chair of the Special Projects Committee of the South African Council for Natural Scientific Professions, Convenor of the APIMONDIA Africa Working Group on Honey Standards and Adulteration, Member of APIMONDIA Working Group on Adulteration of Bee Products. He is a Fellow of the Royal Entomological Society of London, a Fellow of the Royal Society of South Africa, a Fellow of the World Academy of Sciences (TWAS), a founding member and a past president of the Academy of Science of South Africa, a fellow of the African Academy of Science and a Foreign Associate of Hassan II Academy of Science and Technology in Morocco. He was awarded the Gold Medal of the Zoological Society of South Africa and honorary life membership of the Entomological Society of Southern Africa. He is currently a Senior Research Fellow in the Centre for the Advancement of Scholarship at the University of Pretoria.
- Clemencia Cosentino is the Director of Evaluation at the Food and Agriculture Organization of the United Nations. Before this, she was the Chief Evaluation Officer and Evaluation and Assessment Capability Section Head of the US National Science Foundation (NSF). A Senior Fellow and Director of STEM Research at Mathematica and the Director of the Program for Evaluation and Equity Research of the Urban Institute. Clemencia is widely recognized as an expert on science, technology, engineering, and mathematics (STEM), including the underrepresentation of minorities and women in STEM-related degree programmes, training, and careers. Over the past decade, she has been focusing on designing and developing complex data systems that leverage existing data and serve multiple functions (monitoring, research, and evaluation). She received her master’s and doctoral degrees in sociology from Princeton University, with a focus on education and international development.
- Sarah de Rijcke is Professor of Science, Technology and Innovation Studies and Scientific Director at CWTS, Leiden University, and Co-Chair of the Research on Research Institute (RoRI). Sarah specializes in social studies of research evaluation, which she considers in relation to epistemic cultures, knowledge infrastructures, valuation processes, and roles of research in and for society. She has a strong international public academic presence with global outreach activities in science policy, speaking frequently on the topic of research evaluation and metrics uses. She recurrently acts as expert advisor in European and global science policy initiatives. Most recently, she was invited to represent the Netherlands in a high-level UNESCO Expert Group to write a global recommendation on Open Science. Her present research is funded by a grant from the European Research Council (ERC). Her team regularly collaborates in research consortia funded by the European Commission’s Framework programmes and national research councils across Europe and the UK.
- Carlo D’Ippoliti is associate professor of economics at Sapienza University of Rome, where he co-coordinates the Minerva Laboratory on Gender Equality and Diversity. He is editor of the open access economics journals “PSL Quarterly Review” and “Moneta e Credito”; a member of the Global Young Academy; and the 2018 winner of the A. Feltrinelli Giovani prize awarded by the Accademia dei Lincei. A heterodox economist concerned about the stunting of diversity and pluralism within the social sciences, Carlo specializes in the history and sociology of economic research, and on European economic policy. He has published on research evaluation in economics and its impact on the subsequent development of economics research, and has coordinated two large research projects on this topic, financed by the Institute for New Economic Thinking (USA) and the Rebuilding Macroeconomics network (Economic and Social Research Council, UK).
- Shaheen Motala-Timol is a chemistry-trained higher education professional and quality assurance practitioner. She holds a PhD in Polymer Chemistry. She is presently heading the Regulatory Affairs and Accreditation Division at the Higher Education Commission, the authority that regulates the higher education sector in Mauritius. She also works as an independent higher education consultant for government agencies and serves as an external specialist for international regulatory agencies, such as the Oman Academic Accreditation Authority, the Malta National Commission for Further and Higher Education, and the Turkish Higher Education Quality Council, amongst others. Under the InterAcademy Partnership (IAP) Africa Fellowship, she conducted an internal assessment of the centres of the African Institute for Mathematical Sciences. In 2018-2019, Dr Timol was a member of the Executive Committee of the Global Young Academy, where she led the communication portfolio. She was a Hubert H. Humphrey Fellow under the Fulbright Exchange Activity and spent the 2016-2017 academic year at Pennsylvania State University, USA, researching different areas of international higher education. She was a Visiting Scholar at the Center for International Higher Education, Boston College. Her current research addresses the challenges and opportunities of cross-border higher education and internationalization.
- Noorsaadah Binti A. Rahman is Professor of Chemistry and Deputy Vice-Chancellor (Research & Innovation) at the University of Malaya (since 2015). She is responsible for the development and strategic enhancement of the University’s research and innovation profile of the university, aimed at enhancing research quality, capacity and capability across the University. She is a member of the Council of the Academy of Sciences Malaysia and chairs the Malaysian Open Science Alliance.
Professor Rahman received her BA in Chemistry degree from California State University, her MSc from the University of California, Irvine in the US, and her PhD from Cambridge University, UK. She has been received numerous international awards such as the Japanese Society for Promotion of Science (1992) fellowship, the JWT Jones Travelling Fellowship, the Royal Society of Chemistry (1995) award, the Chevening Award (1996), the Fulbright Scholar Award (2001) and CNRS Fellowship (2001/2002). Prof. Noorsaadah is also a recipient of the Top Malaysian Research Scientist (TRSM) Award. - Laura Rovelli is a political scientist and PhD in Social Science from the University of Buenos Aires, Argentina. Adjunct researcher at the National Scientific and Technical Research Council (CONICET) and faculty at La Plata National University (UNLP) in Argentina. She coordinates the Latin American Forum for Research Assessment (FOLEC) from the Latin American Council of Social Sciences (CLACSO) and is a member of the Advisory Board of DORA. Recently co-authored, with Dominique Babini, the book “Recent trends in open science and open access in scientific policies in Ibero- America” and has been an observer on behalf of CLACSO at the UNESCO intergovernmental meeting to elaborate a draft Recommendation on Open Science. At present, is conducting at FOLEC a research project funded by the IDRC and called: “Research quality and allocation of research funds in the Global South: Research assessment in change: inclusivity in science systems and mission-oriented projects in research funding initiatives from the Global South. Reshaping quality evaluation through grounded and progressive methodologies”.
- David Vaux is a molecular biologist whose research focusses on the mechanism of cell death, the physiological process used to remove unwanted cells. He trained in medicine in Melbourne, Australia, and was a post-doc at Stanford in the US before returning to The Walter and Eliza Hall Institute in Australia. Apart from his research, he is interested in issues related to research integrity. He a member of the Center for Scientific Integrity (NY), which acts as the board for Retraction Watch. He served on the Committee for Freedom and Responsibility in the Conduct of Science, a committee of the International Science Council. He has participated in most of the World Conferences in Research Integrity and gave the plenary speech at the 2010 meeting that produced the Singapore Statement on Research Integrity. He has encouraged journals to improve their policies on use of statistics and images, and their handling of concerns about possible errors. He is an associate member of the Committee of Publication Ethics (COPE). A major current effort is to have Australia establish a national ombudsman or office for research integrity to replace the current self-regulation model.
- Koen Vermeir (@KoenVermeir) is a Research Professor at the French National Research Center (CNRS) and the University of Paris, France. A theoretical physicist turned historian and philosopher of science, he is also active in the science-policy nexus. He has focused on science advice, science as a public good and the reform of research assessment and he has collaborated with the European Commission, the GScience Academies, the United Nations, UNESCO, cOAlition S and other national and international stakeholders. Koen is past Co-Chair of the Global Young Academy, member of the IAP Open Science working group and member of the ISC writing group on Freedoms and Responsibilities in Science. As GYA Co-Chair, Koen strove for a more inclusive science ecosystem and aimed to empower young scientists globally.
- Yupeng Yao is Director of the Bureau of Policy at the National Natural Science Foundation of China and a research professor in geology. He obtained BA (1988) and MS (1991) degrees with majors in mineralogy and petrology from Nanjing University, and a PhD (1999) in the field of petrology and tectonics from the Chinese Academy of Science in collaboration with Stanford University. He has four years of experience in the Chinese Geological Survey as an assistant research fellow working on Antarctic geology. Since then he has been working in the department of Earth Science at the National Natural Science Foundation of China for more than 20 years, as programme director of geology, division director of strategic planning, and deputy director of the department, successively. He is a Council member of the Chinese Association for Quaternary Research (CHIQUA) and petrology committee member of Chinese Society of Geology. He has initiated and coordinated bilateral/multilateral collaborations with other funding agencies in France, Germany, UK and USA, and others, in the fields of geoscience. He has published more than 80 papers covering geological research, funding strategy and scientific policy.