English
Putting science into the assessment of research
There is a pressing need to improve the ways in which the output of scientific research is evaluated by funding agencies, academic institutions, and other parties.
To address this issue, the group of editors and publishers of scholarly journals listed below met during the Annual Meeting of The American Society for Cell Biology (ASCB) in San Francisco, CA, on December 16, 2012. The group developed a set of recommendations, referred to as the San Francisco Declaration on Research Assessment. We invite interested parties across all scientific disciplines to indicate their support by adding their names to this declaration.
The outputs from scientific research are many and varied, including: research articles reporting new knowledge, data, reagents, and software; intellectual property; and highly trained young scientists. Funding agencies, institutions that employ scientists, and scientists themselves, all have a desire, and need, to assess the quality and impact of scientific outputs. It is thus imperative that scientific output is measured accurately, evaluated wisely.
The Journal Impact Factor is frequently used as the primary parameter with which to compare the scientific output of individuals and institutions. The Journal Impact Factor, as calculated by Thomson Reuters, was originally created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article. With that in mind, it is critical to understand that the Journal Impact Factor has a number of well-documented deficiencies as a tool for research assessment. These limitations include: A) citation distributions within journals are highly skewed [1–3]; B) the properties of the Journal Impact Factor are field-specific: it is a composite of multiple, highly diverse article types, including primary research papers and reviews [1, 4]; C) Impact Factors can be manipulated (or “gamed”) by editorial policy [5]; and D) data used to calculate the Journal Impact Factors are neither transparent nor openly available to the public [4, 6, 7].
Below we make a number of recommendations for improving the way in which the quality of research output is evaluated. Outputs other than research articles will grow in importance in assessing research effectiveness in the future, but the peer-reviewed research paper will remain a central research output that informs research assessment. Our recommenda-tions therefore focus primarily on practices relating to research articles published in peer-reviewed journals, but can and should be extended by recognizing additional products, such as datasets, as important research outputs. These recommendations are aimed at funding agencies, academic institutions, journals, organizations that supply metrics, and individual researchers.
A number of themes run through these recommendations:
• The need to eliminate the use of journal-based metrics, such as impact factors, in funding, appointment and promotion considerations;
• the need to assess research on its own merits rather than on the basis of the journal in which the research is published, and
• the need to capitalize on the opportunities provided by online publication (such as relaxing unnecessary limits on the number of words, figures, and references in articles, and exploring new indicators of significance and impact).
We recognize that many funding agencies, institutions, publishers, and researchers are already encou- raging improved practices in research assessment. Such steps are beginning to increase the momentum toward more sophisticated and meaningful approaches to research evaluation that can now be built upon and adopted by all of the key constituencies involved.
The signatories of the San Francisco Declaration support the adoption of the following practices in research assessment.
General Recommendation
1. Do not use journal-based metrics, such as journal impact factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion or funding decisions.
For funding agencies
2. Be explicit about the criteria used in evaluating the scientific productivity of grant applicants
and clearly highlight, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
3. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
For institutions
4. Be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting, especially for early-stage investigators, that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
5. For the purposes of research assessment, consider the value and impact of all research outputs (including datasets and software) in addition to research publications, and consider a broad range of impact measures including qualitative indicators of research impact, such as influence on policy and practice.
For publishers
6. Greatly reduce emphasis on the journal impact factor as a promotional tool, ideally by ceasing to promote the impact factor or by presenting the metric in the context of a variety of journal-based metrics (eg. 5-year impact factor, EigenFactor [8], SCImago [9], h-index, editorial and publication times, etc.) that provide a richer view of journal performance.
7. Make available a range of article-level metrics to encourage a shift toward assessment based on the scientific content of an article rather than publication metrics of the journal in which it was published.
8. Encourage responsible authorship practices and the provision of information about the specific contributions of each author.
9. Whether a journal is open-access or subscription--based, remove all reuse limitations on reference lists in research articles and make them available under the Creative Commons Public Domain Dedication [10] .
10. Remove or reduce the constraints on the number of references in research articles, and, where appropriate, mandate the citation of primary literature in favor of reviews in order to give credit to the group(s) who first reported a finding.
For organizations that supply metrics
11. Be open and transparent by providing data and methods used to calculate all metrics.
12. Provide the data under a licence that allows unrestricted reuse, and provide computational access to data, where possible.
13. Be clear that inappropriate manipulation of metrics will not be tolerated; be explicit about what constitutes inappropriate manipulation and what measures will be taken to combat this.
14. Account for the variation in article types (e.g., reviews versus research articles), and in different subject areas when metrics are used, aggregated, or compared.
For researchers
15. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics.
16. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due.
17. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs [11].
18. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote and teach best practice that focuses on the value and influence of specific research outputs.
References
1. Adler, R., Ewing, J., and Taylor, P. (2008). Citation statistics. A report from the International Mathematical Union. Avalaible at URL: <www.mathunion.org/publications/report/citationstatistics0>.
2. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. In BMJ. 15; 314 (7079): 498–502.
3. Editorial (2005). Not so deep impact. In Nature 435, 1003–1004. Rossner, M., Van Epps, H., Hill, E. (2007). Show me the data. In J. Cell Biol. 179, 1091–1092.
4. Vanclay, J. K. Impact Factor: Outdated artefact or stepping-stone to journal certification. In Scientometric 2012; 92: 211–238.
5. The PLoS Medicine Editors (2006). The impact factor game. In PLoS Med 3 (6): e291 doi:10.1371/journal.pmed.0030291.
6. Rossner, M., Van Epps, H., Hill, E. (2007). Show me the data. In J. Cell Biol. 179, 1091–1092.
7. Rossner M., Van Epps H., and Hill E. (2008). Irreproducible results: a response to Thomson Scientific. In J. Cell Biol. 180, 254–255.
8. http://www.eigenfactor.org/
9. http://www.scimagojr.com/
10. http://opencitations.wordpress.com/2013/01/03/open-letter-to-publishers
11. http://altmetrics.org/tools/
Original Signers (Organizations)
Academy of Sciences of the Czech Republic; Altmetric LLP; American Association for the Advancement of Science; American Oil Chemists’ Society; American Society for Cell Biology; American Society of Agronomy; Association for Psychological Science; Austrian Science Fund; Biology Open; British Society for Cell Biology; CBE – Life Sciences Education; Cell Structure and Function (a journal published by Japanese Society of Cell Biology); Centro Nacional de Análisis Genómico; Crop Science Society of America; Czech Mathematical Society; Department of Cell Biology, University of Texas Southwestern Medical Center; Development; Disease Models & Mechanisms; ECS – The Electrochemical Society; eLife; EMBO; EMBO Reports; EuCheMS; European Association of Science Editors; European Association of Social Anthropologists; European Astronomical Society; European Atherosclerosis Society; European Council of Doctoral Candidates and Junior Researchers; European Crystallographic Association; European Education Research Association; European Federation for Medicinal Chemistry; European Glaucoma Society; European Mathematical Society; European Molecular Biology Laboratory; European Optical Society; European Society for Soil Conservation; European Society for the History of Science; European Sociological Association; Faculty of 1000; FEBS Journal; FEBS Letters; FEBS Open Bio; Federation of European Biochemical Societies; Fondazione Telethon; Garvan Institute of Medical Research; Genetics Society of America; Gordon and Betty Moore Foundation; Higher Education Funding Council for England; Howard Hughes Medical Institute; ImpactStory; Institute for Molecular Bioscience, Brisbane, Australia; Institute of Mathematics, Academy of Sciences of the Czech Republic; Journal of Cell Science; Journal of Neurochemistry (Society Journal of the International Society of Neurochemistry); Linguistic Society of America; Macedonian Journal of Chemistry and Chemical Engineering; Medical Research Council Laboratory of Molecular Cell Biology; Molecular Oncology; Molecular Biology of the Cell; Molecular Systems Biology; PeerJ; Proceedings of The National Academy of Sciences; Public Library of Science; Society of Chemists and Technologists of Macedonia; Society of Economic Geologists; Soil Science Society of America; Spanish Crystallographic Association; Swiss Academy of Medical Sciences; The American Physiological Society; The Anatomical Record; The Association of Australian Medical Research Institutes; The Bionics Institute; The Company of Biologists; The European Society for History of Science; The EMBO Journal; The International Society of Addiction Journal Editors; The Journal of Cell Biology; The Journal of Experimental Biology; The Macfarlane Burnet Institute for Medical Research and Public Health (Burnet Institute); Victor Chang Cardiac Research Institute; Wellcome Trust.
Original Signers (Individuals)
Euan Adie, Altmetric LLP, London, UK; Elizabeth M. Adler, Executive Editor, The Journal of General Physiology, New York, U.S.A.; Sharon Ahmad, Executive Editor, Journal of Cell Science, Cambridge, UK; Kurt H. Albertine, Editor-in-Chief, The Anatomical Record; Bruce Alberts, Editor-in-Chief, Science, Washington, U.S.A.; José M. Amigó, Professor Emeritus, Unity of Crystallography and Mineralogy, Department of Geology, University of Valencia, Spain; Parker Antin, Editor-in Chief, Developmental Dynamics; Simeon Arseniyadis, Research Director, CNRS, France; Detlef Axmann, Professor, Department of Prosthodontics and Medical Materials, Eberhard-Karls-University, Germany; Tonci Balic-Zunic, Associate Professor in Mineralogy and leader of the Crystallography & Mineralogy Group, Natural History Museum, University of Copenhagen, Denmark; Joel Bernstein, Professor, Department of Chemistry, New York University Abu Dhabi, United Arab Emirates; Stefano Bertuzzi, Executive Director, American Society for Cell Biology, Los Angeles, U.S.A.; Ted Bianco, Acting Director, Wellcome Trust, London, UK; Joël Bockaert, Professor, University of Montpellier 1, France, Member, Académie des Sciences; Elena Boldyreva, Novosibirsk State University, Institute of Solid State Chemistry and Mechanochemistry, Siberian Branch of Russian Academy of Sciences, Russia; David Botstein, Founding Editor-in-Chief of Molecular Biology of the Cell, Director Lewis-Sigler Institute for Integrative Genomics, Princeton university, U.S.A.; Nouzha Bouhmaida, Professor, Laboratoire Sciences Des Matériaux, Faculté Des Sciences, Marrakech, Morocco; Roque J. Calvo, Executive Director, ECS – The Electrochemical Society; Michael Caplan, Professor and Chair, Dept. of Cellular and Molecular Physiology, Yale University, Connecticut, U.S.A.; Julio E. Celis, Editor-in-Chief, Molecular Oncology; Martin Černohorský, Rector emeritus, Silesian University in Opava, Professor emeritus, Masaryk University, Brno, Czech Republic; Vicki Chandler, Gordon and Betty Moore Foundation, California, U.S.A.; Daniel Choquet, Research Director, CNRS, France, Director of the Interdisciplinary Institute for Neuroscience, Director of the Bordeaux Imaging Center, Member of the Academy; Don Cleveland, President, American Society for Cell Biology, Distinguished Professor and Chair, Dept. of Cellular and Molecular Medicine, University of California, San Diego, U.S.A.; Françoise Combes, Observatoire de Paris and Académie des Sciences, France; Paul Courant Harold T. Shapiro, Professor of Public Policy, University of Michigan, U.S.A.; Brendan Crabb, President, Association of Australian Medical Research Institutes, Director, The Burnet Institute, Melbourne, Australia; Ana Maria Cuervo, co-Editor-in-Chief of Aging Cell, Professor, Albert Einstein College of Medicine, New York, U.S.A.; Stephen Curry, Professor and Chair, Department of Life Sciences, Imperial College, London, UK; Antonella De Matteis, Telethon Institute of Genetics and Medicine, Naples, Italy; Tracey DePellegrin, Executive Editor, GENETICS and G3: Genes|Genomes|Genetics, Maryland, U.S.A.; Michel Desarménien, Research Director, CNRS, France; Danny Dolev, Scientific Council, ERC, School of Engineering and Computer Science, The Hebrew University of Jerusalem, Israel; Athene M. Donald, Cavendish Laboratory, Cambridge, UK; David Drubin, Editor-in-Chief, Molecular Biology of the Cell, Professor, University of California, Berkeley, U.S.A.; Barbara Ensoli, Director, National AIDS Center, Italy; Wolfgang Eppenschwandtner, Executive Coordinator, Initiative for Science in Europe (ISE), Germany; Daniel Esteve, Quantronics group, SPEC-CEA Saclay, France; Pavel Exner, Scientific Director, Doppler Institute for Mathematical Physics and Applied Mathematics Prague, Czech Republic; Adam P. Fagen, Executive Director, Genetics Society of America; Sir Alan Fersht, FRS Associate Editor, PNAS, Washington, D.C., U.S.A.; László Fésüs, Chairman of Publications Committee, Federation of European Biochemical Societies; Marty Frank, Executive Director, The American Physiological Society, Maryland, U.S.A.; Toni Gabaldón, Centre for Genomic Regulation, Barcelona, Spain; Santiago Garcia-Granda, Professor, Physical Chemistry, University of Oviedo, Immediate Past-President, European Crystallographic Association, Spain; Juan Manuel García-Ruiz, Research Professor at the Consejo Superior de Investigaciones Científicas and University of Granada, Spain; Fernando Garzon, President, ECS – The Electrochemical Society; Marina Gebert, Group Leader Aquatic Cell Technology, Fraunhofer Institution for Marine Biotechnology, Luebeck, Germany; James Gentile, Dean, Natural & Applied Sciences, Hope College, Past President, Research Corporation for Science Advancement, former Editor-in-Chief, Mutation Research; Alexander Gerber, Managing Director, German Research Center for Science & Innovation Communication (INNOKOMM), Germany; Christian Gericke, Chief Executive, The Wesley Research Institute, Brisbane, Australia; Paul A. Gleeson, Head, Department of Biochemistry and Molecular Biology, The University of Melbourne, Australia; Bruce L. Goode, Editor, Cytoskeleton; Professor, Biology Rosenstiel Basic Medical Sciences Research Center, Brandeis University, U.S.A.; Sharona Gordon, Incoming Editor, Journal of General Physiology; Robert M. Graham, Executive Director, Victor Chang Cardiac Research Institute, Sydney, Australia; Peter Gunning, President, Australian Society for Biochemistry and Molecular Biology, Editor-In-Chief, BioArchitecture, University of New South Wales, Australia; John Gurdon, Former Chairman, Company of Biologists, UK; Lisa Hannan, Managing Editor, Traffic; Richard W. Hartel, Editor-in-Chief, Journal of the American Oil Chemists' Society; Carl-Henrik Heldin, Ludwig Institute for Cancer Research, Uppsala University, Sweden; Etienne Herzog, Interdisciplinary Institute for NeuroScience, Bordeaux University, France; Dennis W. Hess, Editor, ECS Journal of Solid State Science & Technology and ECS Solid State Letters; Heribert Hirt, President, European Plant Sciences Organisation (EPSO); Brian Hoal, Executive Director, Society of Economic Geologists, Colorado, U.S.A.; Jason Hoyt, Co-Founder and CEO, PeerJ; Fabian Huettig, Assistant Medical Director, Department for Prosthodontics with Section Medical Materials & Technology, Center for Dentistry and Oral Medicine, Tuebingen University Hospital, Germany; Steve Humphries, Editor-in-Chief, Atherosclerosi, Official Journal of the European Atherosclerosis Society; Tim Hunt, Fellow of the Royal Society, Chair, The Company of Biologists, UK; Howy Jacobs, Chief Editor, EMBO Reports, Finland; Reinhard Jahn, Department of Neurobiology, MPI for Biophysical Chemistry, EMBO Publications Advisory Committee (chair), EMBL Scientific Advisory Board (vice chair), Dean, Göttingen Graduate School for Neurosciences, Biophysics, and Molecular Biosciences, Germany; David James, Director, Diabetes and Obesity Program, Garvan Institute of Medical Research, Fellow, Australian Academy of Science, Canberra, Australia; Mark Johnston, Editor-in-Chief of GENETICS, Professor and Chair, Department of Biochemistry and Molecular Genetics, the University of Colorado School of Medicine, U.S.A.; Richard A. L. Jones, ex-Editor-in-Chief, European Physical Journal; Kozo Kaibuchi, Editor-in-Chief of Cell Structures and Functions (the official journal of the Japanese Society for Cell Biology); Alan Kraut, Executive Director, Association for Psychological Science; Karl Kuchler, Medical University of Vienna, Max F. Perutz Laboratories, Austria; Laurent Ladépêche, Interdisciplinary Institute for NeuroScience, Bordeaux University, France; Fernando J. Lahoz Director, Chemical Synthesis and Homogeneous Catalysis Research Institute, Spanish National Research Center, University of Zaragoza, Spain; Pekka Lappalainen, Executive Editor, Cytoskeleton, Research Director, Institute of Biotechnology, University of Helsinki, Finland; Rebecca Laurence, Publisher, F1000Research and F1000Posters; W. Mark, Leader Publications Director, American Society for Cell Biology; Thomas Lemberger, Chief Editor, Molecular Systems Biology; Maria Leptin, Director, EMBO; Anthony Linden, University of Zurich, Switzerland; Daniel Louvard, Director of the Research Centre, Institut Curie, France; Michael Lynch, President, Genetics Society of America; Michael Marks, Co-editor, Traffic, Professor, University of Pennsylvania, U.S.A.; Mark Marsh, Co-editor, Traffic, Director, Medical Research Council Laboratory for Molecular Cell Biology; Marc A. Marti-Renom, Associate Editor at PLOS Computational Biology, National Center for Genomic Analysis and Centre for Genomic Regulation, Barcelona, Spain; Thomas Marwick, Director, Menzies Research Institute Tasmania; Paul Matsudaira, Head, Department of Biological Sciences, National University of Singapore; Iain Mattaj, EMBL Director General; Satyajit Mayor, Director, National Centre for Biological Science, Bangalore, India; Tom Misteli, Editor-in-Chief, The Journal of Cell Biology; Thor Moeller, Researcher, Institut de Génomique Fonctionnelle (IGF), CNRS, France; Lucia Monaco, Chief Scientific Officer, Fondazione Telethon, Italy; Eric Murphy, Editor-in-Chief, Lipids, a Journal of the American Oil Chemists' Society; Valery Nakariakov, President, European Solar Physics Division, Physics Department, University of Warwick, UK; Susana Narotzky, Professor, Cultural Anthropology, University of Barcelona, Spain; Helga Nowotny, President, European Research Council, WWTF Vienna Science and Technology Fund, Austria; Paul Nurse, President, The Royal Society, London, UK; Henk Ottens, President, Association of Geographical Societies in Europe – EUGEO; Mark Patterson, Executive Director, eLife; Eva Pebay-Peyroula Professor, Joseph Fourier University, Grenoble, Member of the French Academy of Science; Pedro Pereira, Associate Researcher, IBMC – Instituto de Biologia Molecular e Celular, Portugal; Richard N. Perham, Editor-in-Chief, FEBS Journal; Alaine Peyraube, Director of Research at the CNRS, France; Olivier Pironneau, Professor, LJLL – Analyse Numérique, Université Pierre et Marie Curie (Paris VI), France; Heather Piwowar, Cofounder, ImpactStory; Olivier Pourquié, Editor-in-Chief, Development; Jacques Pouyssegur, Research Director at CNRS, Member of French & Europea Academy, Nice, France; Alberto Prestininzi, Editor-in-Chief, Italian Journal of Engineering Geology and Environment; Jason Priem, Co-founder, ImpactStory; Edward N. Pugh, Jr., Editor, Journal of General Physiology; Bernd Pulverer, Chief Editor, The EMBO Journal, Head of Scientific Publications, EMBO; Marianne Quiquandon, Researcher, CNRS, France; Jordan Raff, President, British Society of Cell Biology, Editor-in-Chief, Biology Open, Professor, Cancer Cell Biology, University of Oxford; Francisco X. Real, Spanish National Cancer Research Center and Universitat Pompeu Fabra; Alyson Reed, Executive Director, Linguistic Society of America; Kari Rissanen, Academy Professor, Department of Chemisty, University of Jyväskylä, Finland; Phillip J. Robinson, Head, Cell Signalling Unit, Children's Medical Research Institute; Mike Rossner, Executive Director, The Rockefeller University Press, New York, U.S.A.; Didier Roux, Member of the French Academy of Sciences; Anthony J. Ryan, Pro Vice Chancellor, Faculty of Science, The University of Sheffield, UK; Jean-Louis Salager, Editor-in-Chief, Journal of Surfactants and Detergents; Noel B. Salazar, President, European Association of Social Anthropologists; Michele Saviano, President of Italian Association of Crystallography, Director of Institute of Crystallography – CNR; Randy Schekman, Editor-in-Chief, eLife; Sandra Schmid, Cecil H. Green Distinguished Chair in Cellular and Molecular Biology, Professor and Chair, Department of Cell Biology, University of Texas Southwestern Medical Center, U.S.A.; Trina Schroer, Co-editor, Traffic, Professor, Johns Hopkins University, Maryland, U.S.A.; Ulrich Schubert, Professor, Institute of Material Chemistry, Vienna University of Technology, Austria; Jörg Schulz, Editor-in-Chief, Journal of Neurochemistry, Chair and Full Professor, Department of Neurology, RWTH Aachen University, Germany; André Sentenac, Member of the French Academy of Sciences, Former Director of a Department of Biology at the CEA (Atomic Energy Commission); Robert Shepherd, Director, Bionics Institute, University of Melbourne, Australia; Stuart Shieber, Harvard University, Maryland, U.S.A; Tom Stevens, Co-editor, Traffic, Professor, University of Oregon, U.S.A.; Jennifer L. Stow, Professor and Deputy Director, Research, Institute for Molecular Bioscience, The University of Queensland, Brisbane, Australia; Soňa Štrbáňová, Associate Professor, Centre for the History of Sciences and Humanities, Institute for Contemporary History, Academy of Sciences of the Czech Republic; Marlowe Tessmer, Senior Editor, The Journal of Experimental Medicine; Robert Tjian, President, Howard Hughes Medical Institute, Maryland, U.S.A.; Gerrit Van Meer, Dean of the Faculty of Sciences, Utrecht University, Netherlands; Petr Vanýsek, Editor, Journal of The Electrochemical Society and ECS Electrochemistry Letters, Czech Republic; Inder Verma, Editor-in-Chief, Proceedings of The National Academy Of Sciences (PNAS); Michael Way, Editor-in-Chief, Journal of Cell Science; Heiner Weber, Dean, Center for Dentistry and Oral Medicine, Chairman, Department of Prosthodontics, University of Tuebingen, Germany; Eric Westhof, Directeur, Institut de biologie moléculaire et cellulaire du CNRS, Strasbourg, France; Kathleen Wets, Publisher, F1000Prime; Felix Wieland, Managing Editor, FEBS Letters; Liz Williams, Executive Editor, The Journal of Cell Biology; Mitsuhiro Yanagida, Editor-in-Chief, Genes to Cells; Alpha Yap, Head, Division of Molecular Cell Biology, Institute for Molecular Bioscience, The University of Queensland, Australia; Mary Yess, Deputy Executive Director and Publisher, ECS – The Electrochemical Society; Marino Zerial, Max Planck Director, Max Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany; Ya-ping Zhang, Vice-President, Chinese Academy of Sciences, China; Jiří Zlatuška, Rector emeritus, Professor, Masaryk University, Brno, Czech Republic.