"The true use of these metrics [twitter metrics] is to measure the distinct concept of social impact. Social impact measures based on tweets are proposed to complement traditional citation metrics. The proposed twimpact factor may be a useful and timely metric to measure uptake of research findings and to filter research findings resonating with the public in real time.
"It should be noted that prediction of citations is not necessarily the end goal, and that lack of correlation is not necessarily a failure, because it is clear that these metrics add a new dimension of measuring impact."
"Correlations between Twitter coverage and Twitter citation rates with traditional bibliometric indicators for journals were positive and significant, with rates between .223 and .312. Comparing formal citations and Twitter citations for all papers published in 2011, we found a low but positive correlation of .183, which suggests that, although both indicators are somewhat related, they mostly measure a different type of impact.". (Haustein 2013)
It should be stressed again that one should neither expect nor hope for perfect correlation. Tweetations should be primarily seen as a metric for social impact and knowledge translation (how quickly new knowledge is taken up by the public) as well as a metric to measure public interest in a specific topic (what the public is paying attention to), while citations are primarily a metric for scholarly impact. Both are somewhat correlated, as shown here, but tweetations and citations measure different concepts, and measure uptake by or interest of different audiences (Figure 12). The correlation and mutual interaction between these audiences is illustrated in Figure 12 with bidirectional arrows. (Eysenbach 2011)
- first, they excluded JMIR articles, where an association has previously been found, and also other important journals (possibly all electronic-only journals, which have articleIDs rather than page numbers), which renders some of their data (table 1, table 2, fig 1) invalid.
- secondly, they did not look beyond Spearman correlation coefficients and missed an opportunity to analyze the data in the way I did, which is to get rid of the noise by dichotomizing the data (highly tweeted/highly cited vs lower tweeted/cited). Social media signals are messy, and signals can be missed if you don't look at them carefully.
If JMIR is missing from this dataset, then what else is missing? Where is Plos One in all this, the medical journal which publishes the most articles? Plos journals are nowhere to be found in their paper, and this omission cannot to be explained by their exclusion criteria. The apparent unexplained omission of data (including the most tweeted journal - JMIR - and the journal with the most articles - Plos One -) is disturbing and in my view justifies a retraction or correction. The authors acknowledged in an email to me "technical problems", and usually this would require them to correct or retract the paper, if discovered after publication.
Update 1 (11/Dec/2013): Haustein and colleagues now confirmed in an email to me that they not only excluded JMIR (the journal with the most tweets per article, around 13.000 tweets to articles published between 2010-2012, which would constitute about 4% of the tweets they looked at) because of "technical issues", but also Plos One (the worlds largest medical journal, which I believe publishes 30.000 article this year), or all BMC journals (the largest publisher of open access articles). I suspect that their "technical problems" is an omission of articles that have article identifiers (e...) instead of page numbers, as is the case for all electronic-only journals, because they did a sloppy mapping of altmetrics vs Web of Science data. If this would be the case, this would exclude all electronic and/or open access articles, which would introduce a serious bias, as I suspect it is mainly OA articles that are tweeted . Asked whether these errors wouldn't justify a retraction or at least a correction of the paper, they told me they wouldn't because "it would not change the general picture or the conclusions". It's always interesting to come across scientists who know the conclusions before looking at the data... What I do know is that their tables and figures are not worth much without including the most tweeted journals.
Update 2 (11/Dec/2013): I was trying to think of an example to illustrate the differences in the analytical approach - Haustein et al. used Spearman correlation coefficients (ranking articles by tweets, and ranking articles by citations, expecting that every rank matches on a global article level), while I in my JMIR paper dichotomized the data (highly tweeted vs less tweeted, trying to predict the most highly cited articles in a journal), adjusted by journal. I did this because I did not expect a linear correlation between tweets and citations, but I do think the fact that an article is highly tweeted has predictive power for how well an article does in terms of citations, relative to other articles in the same journal.
Why do these different analytical approaches yield different conclusions? An example may illustrate how absurd their approach is: Consider the correlation coefficient between the number of chocolate bars eaten per year by any specific person in the world, and body weight (e.g. make a table with two columns for each person: Number of chocolate bars eaten, and weight. Then rank them by each criterion and see if any given person is on the same rank by both criteria). The Spearman correlation coefficient of this would be very low on a global level, because people don't only get fat by eating chocolate, and different countries have different "normal weights" and eating habits (in some countries chocolate may not be drug of choice), not to talk about the different age groups included in the table. This is basically what they did - look at a "global" correlation. But now stratify the data where you look at a country-level or even city level, adjust by age, and group all people of the same age who are the top 25% chocolate eaters (25th percentile). Are these people more likely to be in the top 25% percentile of weight within their age group and city? I am sure they will be - the "odds" to be in this higher weight group will be much higher for the top chocolate eaters. In any tweet/citation analysis, the data must be adjusted (stratified) by journal. In my tweets vs citation analysis (for one specific journal), I found the odds ratio to be 11, i.e. highly tweeted articles were 11 times more likely to be in the highly cited group. This is a pretty strong association, even though the initial correlation of unadjusted data was low (Spearman correlation coefficient in my dataset: 0.22). Now, the Spearman correlation coefficients in their data was higher for some journals, yet they conclude tweets are useless to identify scholarly important papers. With tweets being sparse events (and the majority of articles not getting many tweets, which introduces a considerable "noise" at the bottom-end of less tweeted articles) the correlation coefficient is expected to be low. But this does not mean that highly tweeted articles don't have any correlation to high scholarly impact.