The ICSU Committee for Freedom and Responsibility in the conduct of Science (CFRS) is concerned that some of the policies and practices currently being adopted by scientific institutions and journal publishers may inadvertently be undermining the integrity of the scientific literature. This is compounded by the uncritical use of publication metrics, as a replacement for independent peer-review, in evaluating scientific performance. By pointing out these concerns to ICSU Member organizations, it is hoped that they will take action to ensure the quality of the scientific record and promote a cautious and critical approach to the use of publication metrics in research assessment.
Issues of concern:
- In making career appointments and awarding grants, the publication record of applicants, and in particular the number and supposed impact of publications, is often a main criterion. This inadvertently creates incentives for duplicate publications, split publications, publications of little value, and honorary authorship.
- Journal impact factors give an indication of the average number of citations to be expected for a paper in a particular journal, but these numbers are easily misinterpreted. For example, they can be influenced by the ratio of reviews to primary papers. Citations for any individual paper may be much higher or lower than that expected from the impact factor.
- While the number of citations can give an indication of the quality of a publication, this is not necessarily so. Truly novel and very important papers may well attract little attention for several years after their publication. A paper that is incorrect can generate multiple citations from papers that rebut it, and citation numbers can be inflated by self-citation. Citation numbers also tend to be greater for reviews than for primary papers.
- The value attributed to publication and citation records can pressure scientists to publish too early, or exaggerate and over-interpret results. It can also, in extreme situations, promote scientific misconduct, in which results are fabricated, falsified or plagiarised.
- As a result of the pressure to publish, a growing number of manuscripts are submitted to multiple journals before they are ultimately accepted. This increases the burden on reviewers, which ultimately diminishes the thoroughness of the review process. Multiple submissions and reviews of the same work can delay the communication of important results and it can be argued that multiplying the number of reviews rarely prevents the eventual publication of poor quality work.
- Alternative means of publication in un-reviewed or non-expert reviewed electronic archives, on individual or institutional home pages or blogs, are becoming increasingly popular. While these media open-up new possibilities and enhance general accessibility to publications, they also increase the availability of less significant or possibly misleading information.
CFRS is concerned that current policies and practices may be having serious effects on the quality of scientific work in general, and increasing the burden on journal reviewers. Any unnecessary increase in the volume of scientific publications threatens a proper reviewing process, which is essential for maintaining standards of accuracy and integrity.
In addition to its role in scientific publishing, the Committee regards rigorous and unbiased peer review as being the most important mechanism for judging the quality of scientific work and scientific projects. Establishing and maintaining good peer review processes is in itself a challenge and it is recognised that there can be benefits in using quantitative measures and metrics as complements to this process. However, the apparent simplicity and attraction of such numerical indices should not conceal their potential for manipulation and misinterpretation and they should therefore only be used with considerable caution.
Because norms for publication number, authorship conventions, and citations differ from field to field, judgements and policy are often best made by peers with expertise in the same area. CFRS urges ICSU’s member organisations to stimulate discussion of scientific evaluation criteria, career indicators and publication records, with the aim of promoting a system that can better serve science in general. Rather than learning to survive in a ‘publish or perish’ culture, young scientists should be encouraged and supported to produce high quality scientific communications that make a real contribution to scientific progress.
Questions for consideration, regarding the use of metrics in assessing research performance, include:
- In judging applications for grants and positions, what is the optimal balance between direct peer reviewing, including reading of relevant articles, and the use of quantitative measures based on publication records?
- In assessing publication records and performance, what weighting should be applied to publication number, h factors, journal impact factor, citation number, primary publication versus review?
- Noting that conventions vary considerably from one field to another, how much credit should be given to first authorship, last authorship, middle authorship, or corresponding authorship? In some fields the prevalence of ‘ghost authorship’ is also an issue of considerable concern*.
- What credit should be given for pre-prints and other electronic publications, whether they are peer reviewed or not? Should impact indices such as clicks, downloads, or links be taken into consideration?
- Should the number of publications that count towards grants or appointments be capped? For example, should only the best three publications per year be taken into consideration? Should scientists even be penalised for authorship on more than, say, 20 publications per year? [Such limits may seem counter-intuitive but would help to promote a culture in which all authors have genuinely contributed to the publications on which their names appear.]
- What weighting should be given to other quantitative measures of research output, such as patent applications, patents granted or patents licensed?
The Committee is aware that discussion of many of these issues is already well underway in some countries and some areas of science, but suggests that the debate is widened to the full international scientific community. By sharing options and strategies, it is hoped that the global science community can benefit from experiences of individual organisations.
*See, for example, Ross et al, JAMA 299, 1800-1812 (2008).
Addendum
Whilst this statement was being developed, the International Mathematical Union released the Citation Statistics report, which is a detailed critical analysis of the use of citation data for scientific evaluation. A main conclusion of this report is: While having a single number to judge quality is indeed simple, it can lead to a shallow under-standing of something as complicated as research. Numbers are not inherently superior to sound judgments.
About this statement
This statement is the responsibility of the Committee on Freedom and Responsibility in the conduct of Science (CFRS) which is a policy committee of the International Council for Science (ICSU). It does not necessarily reflect the views of individual ICSU Member organizations.
Related Items
President of the International Science Council, Sir Peter Gluckman, addresses EU Ministers
15.02.2024
Indigenous Environmental Defenders are critical for nature and for science, but face serious risk
09.08.2023
One year of war in Ukraine: exploring the impact on the science sector and supporting initiatives
04.08.2023
Attacks on Environmental Scientists: Implications for the Free and Responsible Practice of Science
06.07.2023
From Antarctica to Space: updates from Affiliated Bodies on Day 3 of ISC Mid-term meeting
15.05.2023
One year on: Statements, offers of assistance and resources on the current war in Ukraine
24.02.2023
Concern over ongoing legal proceedings against the Greek economist and statistician Andreas Georgiou
19.12.2022
Science in Times of Crisis Episode 3 - The Fallout of Conflict: The Arctic and Outer Space
15.12.2022
New report reveals professional challenges, preferences and plans of Ukrainian researchers abroad
12.12.2022
Science in Times of Crisis Episode 2 - The Current Clash: Science and the National Interest.
08.12.2022
The state of HIV/AIDS research in Africa: An interview with Dr. Joyce Nyoni for World AIDS Day
01.12.2022
Combating predatory journals and conferences through systemic change in scientific publishing
24.03.2022
A contemporary perspective on the free and responsible practice of science in the 21st century
10.12.2021
Daya Reddy highlights importance of core values of science at close of 2nd ISC General Assembly
15.10.2021
Ten outstanding individual scientists and organizations celebrated in first ever ISC Awards
13.10.2021
International Commission on the Clinical Use of Human Germline Genome Editing releases report
04.12.2020
Planned execution of disaster medicine scholar Ahmadreza Djalali must be halted immediately
01.12.2020
ISC showcases academic freedom and inter- and transdisciplinary research at Sida Science Days
20.05.2019
CFRS supports scientists whose human rights are under threat as a result of scientific activity
10.12.2014
Call for nominations: ICSU Committee on Freedom and Responsibility in the conduct of Science
14.10.2014
Work Plan for the Committee on Freedom and Responsibility in the conduct of Science (CFRS) 2014-2017
14.10.2014
Organisation of international scientific meetings: recommendations from ICSU policy committee
11.01.2013
