The Slavery of the h-index—Measuring the Unmeasurable

The pros and cons of the h-index

The initial idea of Hirsch was to discriminate the investigators who are persistently productive from those who experienced an isolated auspicious moment in their scientific life, and who currently only cut coupons from their popularity roll. Nevertheless, it assumes that researcher A, who published a breakthrough story that was extensively cited, should deserve less respect than researcher B who publishes often and regularly; however, the outcome of the latter's work has not contributed yet to any remarkable discovery. A good example is the inventor of the RNA isolation method, Piotr Chomczynski. He has in total over 65 000 citations to which he contributed almost exclusively (92.9% of all citations) with one single paper regarding the method he introduced in 1987 (Chomczynski and Sacchi, 1987). His current h-index is 23, relatively low for such a prodigious number of citations. Yet, could we imagine working with RNA over the past number of years without possessing this simple technique that is now the principle of virtually every commercial protocol related to RNA extraction? Notwithstanding, it might be argued—depreciating that discovery—that because this method is so simple, someone else would have been discovered it sooner or later. In response, let me cite my former mentor, professor Günther Schütz, who would disprove similar arguments with a simple statement: If you say this is so trivial, why was it not you who discovered PCR? Breakthrough discoveries are not solely dependent on sophisticated science.

Another problem with the h-index is the impossibility of comparing the investigators during different stages of their careers (even assuming comparisons among those representing the same field, which is another ambiguous factor). There is a certain correlation between the age of an investigator and h-index. Clearly and in any case, some of articles will accumulate citations and this number will increase over the time since they were first published. However, even the comparison between investigators at a similar career stage may often be misleading, particularly among young post-docs whose careers have just begun. We must be honest and acknowledge the fact that it is a rare occurrence that right after completing a PhD, such an investigator is able to independently decide about own career development. Mostly, the scientific achievements at this stage are primarily a derivative of the power of the PhD mentor and the reputation of the hosting institution.

Another issue contributing to h-index limitations is that many research groups have different regulations regarding authorship. It is assumed that a researcher's name will be added to the authorship list only after considerable contribution has been made to the published work. However, what occurs fairly often is that being a “middle man” on the listing does not necessary reflect the significant contribution and, worth to be emphasized, the h-index does not differentiate between article authors who hold the most valuable first and last authorship position and those wherein the author's name appears as one among perhaps even 100 authors, as occurs with articles containing vast meta-analyses of clinical data.

Although, some of these concerns were raised in the discussion part of the initial paper published by Hirsch (2005), they were overshadowed by the enormous popularity of this tool and its indiscriminate application. Despite of many critical yet unofficial discussions about the h-index, its limitations, and perhaps dangerous influence on science, the topic—if only tackled by publications—is usually narrowed to the pursuit of more and more sophisticated bibliometrics and proposals of new indicators (Bharathi, 2013; Biswal, 2013; Díaz et al., 2016; Würtz and Schmidt, 2016). The critical voices seem to be less represented, nevertheless there are of course existing articles pointing out that past achievements of the scientists may not necessarily be correlated with future success and that all such rankings need context which means that the best method to gain an impression of the quality is still simply reading the papers (Wendl, 2007; von Bohlen Und Halbach, 2011). Perhaps the combination of looking into the context of the particular paper and the journal reputation gives the best assumption in this matter, however even this approach can be biased toward personal preferences and requires considerable amount of time.

Last but not least, when examining the reasons why the h-index is not trustworthy, we should not neglect issues regarding fraud. Because the h-index does not discriminate self-citations, it is not difficult to predict that even the investigators who are poorly cited by others but who publish prodigiously, citing mostly themselves, will easily increase their h-index in the long run. Moreover, some evidence of misuse has been reported involving artificially pumped h-indices based on i.e., regular cross-citations between good friends (Kotov, 2010).

The h-indices of nobel prize winners

Great discoveries require time for experiments. Furthermore, quality investigational pursuits require time for planning, critical thought, discussion, repetition. This process cannot be conducted in haste and under pressure to simply publish all that has been collected. Isaak Newton was said to think twice and to examine all of his discoveries time after time until he finally decided to publish them. Currently, with such an approach, he would most likely have been released nowadays from the university due to the lack of progress in his research. Seemingly, this example perhaps would be difficult to prove. However, consider instead scenarios that are closer to reality—the Nobel Prize laureates in Physiology & Medicine, an unbiased selection of preeminent scientists in life sciences. Do they actually possess h-indices above 100 or thereabouts? Indeed and not surprisingly, many of them do. However, there are also laureates who would have inevitably encountered roadblocks had their h-index been the decisive factor in their nomination. If we consider the last 25 laureates from the period of the recent 10 years, the outcomes are somewhat surprising. The h-index values vary among these esteemed professors within the range of 24–139 (Figure 1A), although no one would rightfully propose that one of these laureates should be regarded as a 5-fold better scientist than another. Clearly, this may again have reflected the time when the Nobel-worthy discovery was made and simply the age of the laureate, another problematic issue raised in one of recent commentaries in Nature journal (Fortunato, 2014). Nonetheless, even when comparing the laureates by their h-indices at the same age of 35 years, the average age of a young post-doc (Figure 1B), or the year before the laureates' famous discoveries were first announced in top-flight journals (Figure 1C), no correlation exists and moreover, examples may be found of investigators who had virtually no bibliometric track records. This is also true among relatively young Nobel laureates (those born >1960) whose careers were certainly influenced by the IT era from the very beginning (Figures 1A–C, green bars). It could be surmised that in many cases the moderate h-indices at this stage are simply correlated to the persistence in pursuing the hypothesis that ultimately was proven, thereby yielding a scientific breakthrough.

A lesson from poland

We have had a unique experience in Poland, arising from the Communist period of our history, when all work efforts were required to be quantified by numbers, that were the most important, even though the outcomes of that work were far from expected. That was a time during which, e.g., a car factory would proudly announce that it had produced the 1-millionth vehicle, although it completely disregarded that it was barely possible to start the engine in most of the cars or to exit a parking spot because of a variety of mechanical failures.

It appears that looking through the lens of the h-index, we are perilously reverting back to those times. What is conveyed in the article, the quality of the work and the actual impact on the topic in focus does not appear to be important. What counts is quantity, not quality. With more articles and more citations, the magic “h” letter will increase. As a result, this activity will considerably increase the approval chances of grant application or will lend favor to tenure position evaluation. Yet, what will be the feedback regarding these articles? Can anything be learned from them? Apparently, this is out of the question—it does not affect the h-index.

Poland is now enjoying a good period in its history. As a result of maintaining the gross domestic product in the black, despite worldwide economic crises, the monies spent on research are continuously increasing. Moreover, several attempts and programs are underway that aim to reorganize science in Poland to be more competitive and efficient. However, all these efforts may backfire if we remain stuck to the easy and old-fashioned way of evaluating our work.

Unquestionably, we scientists must somehow be evaluated as our job is based on spending public monies and such work should be accomplished in an efficient manner. However, the critical evaluation of science requires a great deal of effort and cannot be done using a simple comparison of numbers. As stated by the Hirsch himself, the single number can give only the rough approximation to an individual's researcher profile with many other factors contributing as well (Hirsch, 2005). The overall assessment is also affected by an average number of citations per paper, the quality of journals where the work was published, and goes far beyond the publication track records including other scientific achievements i.e., invited lectures, international experience (post-doctoral fellowships, sabbaticals, collaborations), project managements.

Another issue is the system of evaluation built upon assumption that future success of the project depends on the previous achievements of the applicant. Thus, it remains tempting for some of the grant reviewers to assess the applicants solely through the prism of their respective h-indices, and such a relentless pursuit of bibliometric factors may bring global science to its knees, leaving no room for independent research that may lead to the elucidation of fundamental biological quandaries.

Conflict of interest statement

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.