Research Impact

Before approaching research metrics, it is important to do so with caution. Comparing results from different databases (for example, SciVal and InCites) is not recommended as they draw on different datasets and work from different standards.

Source: xkcd comic

Before you place too much credence in the metrics you have found, consider these Research Metric Usefulness factors.

Discipline and sub-discipline

• There is significant variation between disciplines in the types of materials published, the number of publications, citation practices, and collaboration practices. For example, in the health sciences publications are often published more rapidly, with more collaborators, and cited earlier in their life than in the humanities

Wide variance in modes of research dissemination across fields

• STEM results will be very different to Creative School results, so comparing these factors is problematic

Researcher’s length of career

• Well-established researchers tend to be more highly cited than early-career researchers, regardless of the quality of the publication.

Challenges to orthodoxy

• Controversial takes can skew citations.

Reason for citation

• Articles may be cited for a variety of reasons including, review of interest in a research topic, to support an argument, or to remark on the poor quality of methods or analysis

Citation clubs, courtesy citations

• Citations gifted to titles not necessarily deserving, but designed to increase citation statistics

Ghost authorship and honorary authorship

• Authorship gifted to scholars not necessarily involved in the writing, but given credit as a courtesy

The citation tracking database being used

• The number of citations may appear to change, but it’s because the database’s scope of indexed materials has changed. No citation database is comprehensive. To get an accurate picture of citations, you may need to consult multiple sources

Bias

• Research suggests that research metrics are not free from bias, including with regards to gender and geographic location: women are cited less frequently than men; metrics are often biased towards western research, traditional research, research trends and fashions

Data Errors

• Misspellings of author names, author names that change, or incorrect attribution for authors that share a name can all impact metrics. ORCIDs can help disambiguate authors.

Mistakes

• Errors may mean some papers are not included in a search.

Metrics can often be so enticing to researchers that they may be used to encourage publication in predatory journals. As described in our Scholarly Communications guide, predatory journals will often:

• Use deceptive tactics to attract often young, inexperienced authors
• Almost always charge an APC, sometimes very low
• Make claims of “quick peer review,” which usually means no or minimal peer review

It will not help your career to publish in such journals.

Predatory metrics are central to the success of many of these disreputable publications, where researchers ignore the red flags for quick uptake and distribution. This ignores the fact that these articles are often poorly indexed (which means they are difficult to find and cite) and will not appear alongside major publications in Scopus or Web of Science.

The Global Journal of Environmental Science and Management has provided a list of "common looting publishers around the world" that should be avoided: https://www.gjesm.net/page_622.html

Qualifiers

• Open Access publishing is not predatory. Predatory publishers are misusing the open access publishing model.
• Sometimes what is identified as predatory publishing are emerging publishing industries in non-Western countries

This guide has been created by the Toronto Metropolitan University Library and is licensed under a Creative Commons Attribution International 4.0 License unless otherwise marked.