Theory & Practice in Peer Review

The bifurcation of science and funding was accelerated in the 1940s. The wartime effort escalated the need to unlock the secrets of the frontier. Science exploded. Moving from parlour game entertainments to a scaled up public enterprise attracted the bureaucracy.

When those allocating funding are not themselves experts in the science, an intermediary framework to evaluate funding opportunities emerged through the peer review system.

In the years since, with the rate of growth in the research community eclipsing the that of the funding pot available to support research, the system is operating under intense scrutiny, weight and manipulation.

At a macro level, policy reviews are frequently commissioned. Insights from every angle reveal the weaknesses in a system designed to uniformly rank proposals, so as to limit reliance on discretion. The relationship between the approach taken and research productivity is not far from the microscope when it comes to exploring how to allocate public funding in research. Most public funding bodies use one, or a combination of ex-post, ex-ante and fixed funding systems (Public Funding Science: An International Comparison). 

At a micro level, researchers have adapted to optimise their chances for success within the system. Key lessons for researchers trying to survive in the current system are:

  1. Publish a lot: Reward systems incentivise quantity more than quality, and novelty more than reliability (Quantity and/or Quality? The Importance of Publishing Many Papers).  Copious publications are driven by funding scarcity. In turn, funding dries up because the science isn't robust.  The result is te mass-scale generation of substandard, incremental papers that use poor methods and an increase in false discovery rates leading to a natural selection of bad science; and an overall reduction in the quality of peer review.

  2. Cite other well ranked researchers: Bibliometrics are widely opened for manipulation (Gaming the impact factor: where who cites what, whom and when?) and gamification (Goodharts Law: Are Academic Metrics Being Gamed?).  Citation metrics such as h-index, m-index, i10-index or g-index, number of papers, number of citations and impact factor, used in isolation can be misleading when applied to the peer review of publication output, as they do not describe the impact, importance or quality of the publication.  These traditional measures have become targets, and are no longer true measures importance/impact.

It’s acknowledged that the system is sub-optimal. It’s an open secret that theory and practice have long been divided.

The modern academic research enterprise, dubbed a “Ponzi Scheme” created the existing perverse incentive system (The disposable academic).  

Agencies are less likely to fund studies that straddle multiple fields.  Elena Bozhkova: (Interdisciplinary proposals struggle to get funded: Agencies are less likely to fund studies that straddle multiple fields, a study of Australian grants finds)

I’ve been on a number of search committees. I don’t remember anybody looking at anybody’s papers. Number and IF [impact factor] of pubs are what counts.  Professor Terry McGlynn  (twitter: @realscientists)

Despite demonstration that financial input is not linked to publication success in any measurable sense (Macro-level Indicators of the Relations between Research Funding and Research Output), a recent study (Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program) has concluded that the peer review system still outperforms what is perceived as the only alternative, being discretionary review by trusted experts.

Rather than letting go of metrics that encourage this perversity, intense effort is being invested in diversifying the mix of quantifiable (although equally artificial and corruptible) factors that can be included in the calculation. Publication count, citations, combined citation-publication counts, journal impact factors, total research dollars, total patents, now form part of the blended mix of readily countable metrics used in most awards, along with grey literature and outreach factors which allocate points for policy advisory roles, consulting appointments and lay-media. Some have made efforts to measure likes on Twitter for this purpose (The impact agenda has led to social media being used in a role it may not be equipped to perform).

These efforts are seen as preferable to a governance structure that affords diversity, and which is permitted to adapt to the context of the science, the users that may be impacted by the science and the resource providers who will likely engage with that science.

Alternative models have attracted much attention in the process of examining the flaws in the current system (Contest models highlight inherent inefficiencies of scientific funding competitions). These models invite exploration as to:

  1. the extent to which the aggregate investment in proposal preparation negates the scientific impact of the funding program?

  2. whether alternative mechanisms for awarding funds that advance science could improve efficiency in research?

Whilst many scientists stress the need for academic freedom, allowing a wide net to be cast with research resources, those providing the resources are seeking to demonstrate accountability and returns on allocations and therefore seek to constrain that freedom, however they are rarely called to account for the results of the decisions made in the context of research funding.  The value of the surface level justification is seen as more important that the substantive contribution being made to the collective knowledge base.

In order to displace that priority, we need to know how and whether we can remove the investment blindfold required to be worn when making decisions about science funding.   Without clearer understanding of research efficiency, and better ways to recognise data signals through the noise, amidst a complex system of perverse incentives, the potential to encourage sustainable research funding is limited.