Metrics and Mechanisms: Measuring the Unmeasurable in the Science of Science

How does science work? What are its core mechanisms? In recent decades, research has increasingly explored such questions using a variety of quantitative metrics which can be easily calculated from publications. But some of these can be deceiving. Over time, the rate at which a paper gets new citations tends to decline, suggesting that older papers may lose their relevance and importance. But there’s another possibility – the once novel ideas a paper introduced may simply pass into common knowledge, and require no citation. In discussing the orbit of the Earth around the Sun, for example, few people now refer back to Galileo Galilei, but not because his published work has lost importance. Similarly, although statistics show that the most cited work of a scholar is more likely to appear early in their career, it’s a mistake to conclude from this that young scientists are more creative. Another persuasive argument holds that young scholars simply have more time to produce scientific work, and older scholars – more deeply enmeshed in the social relations of science – spend more time building social and intellectual capital for the community.
These are just two examples, as LML Fellow Hyejin Youn and colleagues argue in a recent paper, where conclusions drawn from elementary scientific metrics can be misleading. But they also argue that more sophisticated metrics, if used and interpreted with care, can offer legitimate insight into the mechanisms of science. Using several such metrics, their analysis offers a basis to classify existing theories and findings in several ways. First, by focussing on researchers’ attention, they classify science into hot and cold science. Second, considering scientists’ time and productivity, they divide science into fast and slow science, capturing how different fields move at different paces. Finally, with reference to reproducibility, they show how science can be split into soft and hard sciences.
Youn and colleagues join a growing number of scholars who argue for the more cautious use of metrics, with recognition that each metric comes with its own hidden “worldview” of science. They aim to motivate wider research interest in analysing the dynamic processes of science, and to support the more conscientious use of metrics by research institutions, funding agencies, and policymakers.
The paper is available as a pre-print at https://arxiv.org/abs/2111.07250

Leave a Reply

Your email address will not be published. Required fields are marked *