Altmetrics and publishers
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Zijistra, Hans
Lin, Jennifer
Roelandse, Martijn
Mulvany, Ian
- Timo Hannay of Digital Science opened the session a with a brief introduction. He pointed out that the metrics related to only journals highlight only one dimension of the impact. Futhermore the metrics related to the article level are “much more than metrics”. - After introducing himself Hans Zijistra illustrated the work done by Elsevier in the field of altmetrics. His presentation was very interesting for me: he spoke of passion and enthusiasm, to bring to bridge the communication gaps between old and new researchers about the issue of alternative metrics. For him Altmetrics are: “another angle of paper not the only way to impact”. This is a very important aspect in those who, like me, are trying to introduce the existence of altmetrics in their institutions. Hans points out how often the term Altmetrics “confuses people”. He suggested to promote, educate, explore these metrics and make them known to scientists with a “holistic approach”. But, in the meantime, what do Elsevier do? They have been running a pilot across Scopus and ScienceDirect in which they are testing different versions of the Altmetric donut, and soon will be announcing further developments soon. At the end of his presentation, Hans has highlighted the importance of quality over quantity, and than when you click the donut to reveal the full original data, Almetrics becomes “more sexy”! - Jennifer Lin brought the experience of open access publishers. PLOS was the first publisher that introduced article level metrics for their journals. She started her presentation with an Albert Einstein citation: “Not everything that can be counted counts, and not everything that counts can be counted”. Jennifer revealed how to construct such a system of metrics is very difficult. Researchers want to find the relevant papers, the most bookmarked, and navigate through the literature. In the other words, what happens behind the numbers and the data, is altmetrics. PLOS is focusing on the article usage by subject area and the use across all the materials. The future development of the Article Level Metrics,-ALM- for PLOS will be to tell the “stories behind the articles, and stories across article features”. - Martijn Roelandse talked about the Springer and BioMed Central experience. He started with an important question: which counts more: an article with 0 citation but published in a top ten journal, or an article published in lower-impact factor journal but with many citations? In the same way, is more important if an article has many citations, or an article with a lot of downloads? Hard to answer! Springer has compiled a list of the top article downloads for 2013 – what they found was that the most downloaded articles belong to journals without Impact Factors. What Martijn proposed about the idea of the evolution of metrics was interesting: beginning with article level metric, passing by journal metric, until finally reaching person level metric. In another words, a different way to look at the scientific community. - Last but not least, Ian Mulvany gave the eLife perspective. Firstly, he spoke about the project and the mission that characterizes eLife. Secondly he explained the platform and the work done. A very important aspect was the platform host and the bugs that complicated the visualization of the portal making it “a black box” without data. But don’t worry: they have changed their platform host and now e-life is running with a new version of LAGOTTO platform (on the basis of PLOS ALM). In this way they have solved the problem of black box and provide usage data. In the future e-Life-aims to have its own local repository of data and with a project call DET they will be able to provide own metrics. For years the aim of the Publisher was to maximize the impact factor of their journals. The revolution continues! It will be interesting to observe how publishers adopt alternative metrics to encourage the growth of science and knowledge.
 information beforehand. My name is Hans. I'll give you a little overview first and then start my presentation. As I said, my name is Hans, working for a marketing project with the LCP Science, specifically in the STM Journal Search Division. Basically covering all science areas, but really focusing on journal and article metrics. So not just article metrics, also journal metrics. And authors are a very important audience in that case. We're very much involved with that. The key thing for me is really the passion for bridging communication gaps. I think that's a role that altmetrics also can take here. I mean, altmetrics is a real new development that can bridge that gap between non-scientists and scientists, between alphas and deltas. And even like our ancestors tried to do in the past with getting communications across to animals with this horse known in the 1907 experiment, there's a lot to do in science communication. Altmetrics, I think, in essence are a good way of doing science communications. And that's something we all can do a lot more than we've done so far. So a few observations. What I do think is that scientists spend a lot of time on their article. And basically hardly any on the rest of it. They don't want to spend time on marketing. They don't want to spend time on promoting their article anymore. If it's been published, that's basically it. I think good science is not good enough. There's really a battle going on for attention. I mean, there's a lot of information. Yesterday we heard Mr. Farrar starting, kicking off the day, saying, isn't there an overburden coming up already? So it's a little bit finding the balance between a lot of information and finding it and filtering it through tools like altmetrics. Funding bodies, we've seen it yesterday, are more and more demanding. They want to see the societal impact of research. How can you prove that? More and more you see also, and I think especially in the Netherlands, I've seen a couple of examples for that, universities building up marketing and communications departments. I mean, initially they never thought that would be their role, but it's really something coming up. They're building up these departments with a goal. And there we are. Publishers and marketers can help in this area. There's really a role for publishers and marketers to play here. And if we don't play it, you and I can always help you out. So what role do altmetrics play? I see that it's the birth right now. Scientists want to be part of it. They want to know what's going on. What article hits the news. And how can I be part of that? I'll better create great insights and networks. I mean, if you go into the details of a detailed page, for example, on almetric.com or an impact story, you do see a great number of people tweeting about your work. Who are they connected to? Where are they from? All these kinds of interesting network data are there one mouse click away based. And it's just a new angle to impact. It's not the only way of looking at impact, but it's really a new way of looking at impact. And I think we need to treat it that way. The reach beyond academia is clear, but will it also change science? I mean, we heard these examples the day before yesterday at the heck day. Our scientists are going to change the titles of their articles without a question mark. Is it better? Should they do different things? Is that really changing science? Or is it just a side effect? But also, I think, can confuse people. I mean, I had a lot of feedback over the last couple of months while we implemented almetric services around our article, then not everybody is positive about it. I mean, there's also people really saying, what is this? I don't understand it. Please explain. So I really think publishers can play a role here. And what can we do? What can we do with almetrics? I think, essentially, it's really helping scientists to promote their work. Because that's something I said before. They don't like to do as easily. Maybe they have to do, but we really need to help them there. And the core role of a publisher historically, and I think should ever be is advancing science by listening to them, by collaborating, explaining things, and also educating. Educating systems like these tools to work with science, and again, bridge these communication gaps. And it's not just between alphas and debt. It's really early career scientists versus, let's say, older scientists. Clinical versus basic science. There's a lot of silos going on in science that need to be bridged. And I think, again, almetrics is a great tool in achieving that. We need to support sharing. I mean, simply by sharing with one click button to one click to Twitter to man-lay all kinds of repositories that brings your article out in the open. Don't keep it in the dark. We can really help you in getting it out. And again, this holistic approach to metrics, that's important. I mean, there's different situations where you use different metrics. I mean, a journal metric can be very helpful if you're just orienting on what's an interesting journal for me. What is the situation around that journal? And after you've published your article, you're much more interested in this article level, because that's really talking about you, talking about your article. And that's really what you're interested at that moment in time. I think, really, great science deserves great real-separate ability. And that's not always the case. So we need to help in visualizing that. Visualizing numbers, visualizing great science is, I think, a great tool to get the message across. And again, the link to journal metrics is still important. So what did we do at Elsevier? Basically, pilots. Pilots are a very important tool for us to get things going, because if you start with great plans, take months to do it, it doesn't work. You should take a small group of people, very enthusiastic, very engaged, and go for it. And just try things out. Don't worry about making failures and see what happens. And that's what we've done. We piloted in different platforms, science direct scopus. We have a specific platform for journal homepages, which is more of a catalog for journals. And we did a lot of campaigns. I'll show you a few in a minute. And really, engaging here in conferences and discussions is an important way of hearing and feeling and what's going on in science. What's important to scientists where publishers can help them with? My colleague Chin Mei was the guy who did the pilot on science direct. Just a few journals, 27, mainly in the medical area. He was testing the donut for us as a brick display, because I mean, the donut everybody knows from Maltmetric.com is that the only way displaying it is that the only visualization is that the most important one. Well, we found out that it's not really making a big difference in clicks, for example. I mean, clicks are not the holy grail, but it's at least a measure of seeing what affects the one visual ass against the other. I think Martin yesterday tweeted, whereas Habib, well here he is. Virtually with us. Michael Habib is the product manager for Scopus. For the people who don't know what Scopus is, am I still there? It's a database abstracts and articles and all this information. But Michael is really an evangelist. He's basically going to every Alp-Metric conference and he's been there from the start. I mean, when you and one of the other great grand challenges in 2011, I think Michael was there and he was one of the first to adopt the concept of Alp-Metrics at that time already. And he implemented it without any official contract or whatever. He just went for it. And I think that's a real nice approach. I took care of Alp-Metrics on journal homepages. I mean, you see this display. It's already bridging a bit of this gap between the article level and the journal level. Because a lot of people, let's say, initially go to Google, put in a few keywords and lend on journal on page. That's the first point of contact for a lot of people to see what's going on. And one way of showing what's going on with a journal is showing what's going on with their articles. And again, this is just one option you could see that a hit parade can make Alp-Metrics also a bit more sexy in a way. Because it's not just about numbers. It's also a bit like, how can we make sure that people start to like it, start to try it, start to find out what it's really about and start clicking things? Some statistics. Generally, what I already said, it's not a lot of extra traffic. You shouldn't do it for traffic generation as a publisher. You really should do it as adding quality to the science information. It's really about quality and not about quantity. An interesting pattern I found when we did some of these campaigns was if you have the full text article next to a donut, I would assume a lot of people would click the donut and, well, at least 50% of them. But interestingly enough, consistently, 86% of the page is clicked in the article and only 14% is visualisation. What does that mean? I'm not sure yet. Campaigns we did. I think also within a publishing company, it's important to bridge the gap between marketing people and, let's say, the more better oriented side of the business. And the Alp-Metrics explorer tool, for example, this analysis tool is a great way of getting the more alpha people to know what they can do with it. So what information is out there? What can they see on a journal level for all publishers, for all journals? And how can they use this in bringing this communication across? We've done a few things. And I think some of them are pretty creative. Virtual special issues, for example. There was an international archaeology day coming up. The person working in the archaeology field said, hey, let's make a selection based on Alp-Metrics, where we basically make the top 10 or top five selection of articles and display them in this listing, showing that archaeology is also a sexy field where a lot of things are happening. We do it with emailings, where you, again, see this visualisation next to articles. Publishers are using the information from the analysis tools in the editorial board meetings just to educate and make the editorial board enthusiastic about the concept of Alp-Metrics. Because it's still not that well known. I mean, everybody here knows what Alp-Metrics is, or at least the concept of article level metrics. But it's not like 80% of the scientists is aware of that. It's still a lot of work to do to make them at least aware and let alone let them play with it. And let them use it. One minute. I'll stop to go to my finish. Mandela, we've talked a lot about it already. We more and more bring that into life. We're integrating it more in our Alp-Metrics displays. We're bannering about it. And also in our sharing options. I mean, one of the easiest way of sharing your information is with one click of this sharing option buttons. Some statistics, really quickly. We've done a big mailing out in December. Just the first trial, I think the open and click based rates are very good. But at one point, we realized that a lot of information for a lot of journals is so small that you see scores of one that doesn't make sense to display. So we decided to put in a threshold to be a bit more selective on what we are displaying. Because it doesn't make sense to send out a message for a journal where you say, OK, this is a hit parade. And the best scoring article has a score of one. Wow. I mean, that doesn't work. Feedback, generally, people are very positive about it. They really feel there's something in it for them. They're also critical on usefulness. Societies are starting to demand it. I mean, we need to see it. We do want it as part of a contract or whatever. But also beware of controversial articles. Because we talked about it yesterday, sentiment is not really taken care of yet. I mean, a very negative article can end up high in a hit parade. And before you know it, you're promoting a very bad, a very controversial article. Feedback from individuals, I mean, it ranges from what you see on the top, very useful to what you see at the end, really unclear. I don't understand it. What criteria are you using? Some takeaways from this. I think, generally, we should be very receptive to talk to people who take at least the effort to contact you. I mean, if they want to talk about, automatically explain it, try to make them enthused yesterday, be creative. I talked about this virtual special issue. A bus competition. Prizes. I mean, prizes, plus friends were talking about it yesterday, prizes are a way to make it a bit more sexy, a bit more attractive to deal with altmetrics. And that's just another way of getting altmetrics out in the open, making people aware of it. And again, also be creative in things like remarkable articles or personal altmetrics. Final, final, final one. What's next, we will roll out across our different platforms. More integration will be there with usage and citations. And we'll be testing alternative displays. The buckets you heard about yesterday by my colleague, Eliza College, are going to be tested. Is that going to be working? Is it not going to be working? And we obviously keep listening. And again, as a closing statement, I think we really all should make big data small and try to visualize as much as possible, because that really makes this bridge happening. Thank you very much. One final tribute to the two Che Guevara's of altmetrics. You and Mike Taylor, I'm not sure he's in yet, is coming by bike. So probably some traffic jam coming up. I have these t-shirts made for you. And then they need to say, you're on my bike. Thank you. Thank you. Thank you, everyone. I think we'll press on without any questions, fans. But I'm sure you'll be able to find in at Copyright, if there's anything you want us for later. Next. I'd just like to write on the schedule I have here. But it will be Jennifer Lynn from PLOS. Hi, my name is Jennifer Lynn. Welcome to those of you who weren't able to make it yesterday. I will make a couple of references to some of the conversations we've had yesterday. So if you do get a chance and you have missed all of yesterday's really great proceedings, do check that out. I think that's available online. Already, Kat? Is that right? Yes, already. So my name is Jennifer Lynn. I'm from PLOS. We're an open access publisher. And I just want to share a little bit with you today about what we've been doing. So as you know, I mentioned yesterday, not only are we in front of building out the system that we originally set up back in 2009 as a pioneer of article of the metrics and alt metrics as a publisher, and henceforth really working on it to scale it up from a cottage industry to hopefully what will be in the future an entire ecosystem, that is the groundwork. So part of the larger groundwork for a scholarly communications infrastructure. I make one reference, this quote by Albert Einstein, that everything that can be counted counts, not everything that counts, can be counted. We do take this very, very seriously, especially being one of the first organizations to really start thinking about what these metrics are, what are appropriate uses of these metrics. Sometimes those discussions can actually be quite colorful, violent even per se. I do count myself to be very privileged and fortunate to be usually on the side of Martin Fennner, but in the event that it doesn't happen, it's not pretty. One of the things also that is part of what I want to share is that it's actually really, really hard to do this. Building the system itself, where we thankfully have Martin Fennner and a whole host of other very solid and strong engineers and visionaries in this area. But also the work which I would say follows behind is to deepen the ways in which we use it across our publishing organization. Of course, you can't do anything with metrics until you have them. So a lot of the work that we have done now and that we plan on rolling out in the near to midterm future, I'd like to take a little bit of time and share with you. This will be a dizzying array of very different random images. So I just want to set you up. Also in part because I have tables and tables are really never really that interesting. So the most obvious place to start is the expression of it on our open access articles. The front end. Pretty obvious. Authors want to see the reach of the article. So we display each of the metrics on the metrics tab. Researchers want to find the relevant papers. So as much as possible at this point in time, we've been integrating it into our search mechanisms or our display mechanisms such as on the home page, et cetera. In recognition of the fact that by and large, researchers do not go to a single journal's search engine to find, discover relevant papers for them, we really seek to collaborate with other parties. The places where researchers do go, in fact. And we think that that is ultimately going to be one of the very lynch pins of ALMs or altmetrics when those are readily made available in those spaces. Navigating through the literature, this is exactly what I'm talking about. We have these signposts to help users situate themselves when they're on an article. We'll be launching a new visualization of this kind in the next, hopefully before the end of the year or early next year. The back end. You thought I was going to share a different picture than you. Sorry. In the back end, we have built a CMS tool that makes it far easier for many of our editorial staff as in-house and externally to very quickly and easily transfer the curation content, the editorial information enrichment that they provide directly to our home pages and hopefully soon across the entire journal site. So we'll be using that, working on it right now. We're also partnering with Icon Factor Recommends, that service led by Devon West, fantastic technology that surfaces articles based upon a citation graph. We'll be applying some of the ALMs in our second iteration of our Icon Factor Recommends for article recommendations. So as I mentioned yesterday, you'll see the slide again. It's all about piping, not just in the context of the entire scholarly enterprise. The communications ecosystem. But in-house, right? It's the groundwork for all of these very different pieces, not just the front end, not just the back end, but across editorial, publication services, and marketing. For editorial, we're using it already to help enhance our curation work for promotion opportunities, to track dissemination of articles published. It'll give us a view into the type of channels that have potential, the rate of growth amongst many of the different subject areas. For our publication services, it supports our external editors. It provides data and content behind the metrics on the downstream impact of the publications, such as media coverage, blog coverage, et cetera. Just to keep in mind, and I think it has been the assumed assumption that behind a lot of the conversations today, as well as yesterday, when we talk about metrics, we're certainly talking about the number itself. Because the numbers, the raw data, rolled up, enriched, et cetera, they enable us to do far greater things. But what happens behind the numbers is also part and parcel of what metrics and ALMs really are. So surfacing that up, perhaps from our standpoint, in our workflow, we get the numbers first. And then we dig deeper into the things that happen behind it. Marketing can attract authors. We also have email campaigns that are set up to go notify authors, as well as other users who have signed up for notifications for newly published papers, what's being read, et cetera. One of the cool new things that we've recently started, it's going to be, in fact, a monthly series or possibly quarterly series, is something we call the PLOS 1 highlights. And it's essentially using ALMs to surface some of the top papers. And by top, this is a very highly contestable term. We don't really like to say top 10, et cetera, because there are very many use cases for what top means for each person. But what we've been doing is calculating a ratio of PDF downloads to HTML page views. What we do is we take papers that were published within the last four calendar months, and we calculate this and then surface the top view. It gives us a very interesting view as to what users across the globe, across all of the subject areas that we publish in for PLOS 1, are really looking at. And when we have this list, month by month, we can continue to track them, see what's happening to them as they age, as four months becomes a year, a year becomes two years, the citations, which generally lag come rolling in. This is just a very simple snapshot of the actual posting. I do encourage you to go online to PLOS 1's blog called Everyone to go check out the very first one that we did. As I mentioned, it's recurring, so there will be more coming down the line. And with the tracking of these papers, we can continue to visualize these top 10s, applying things in-house, such as our taxonomy or subject area, thesaurus, and begin to see the different areas, the hot areas, where people have expressed more interest in. The great thing about ALM is you can essentially mash up any other additional type of metadata. And this is in part why we say and continue to say that the possibilities for its use are absolutely endless. And we've only begun to scratch the surface. Another really cool editorial effort that is also underway is what we call the ALM article feature. It's a series that not only highlights individual articles based on a per-journal view, but it also celebrates in the end what is a core editorial functions of journals, which is to curate content that matters to their wider audiences. Each one of these articles were selected by our in-house as well as external editorial staff. And then what they did was they took you to the articles and they dove in to the ALMs, surfaced all the stories behind them. And they found out actually quite a lot of interesting things, things that maybe never would have come to light were it not for at least the beginning view of the numbers and at first glance. For one, the article here about prescription testosterone, the reception from the wider audience, one of the things that was quite interesting and provocative as through this story was that public viewing of science does matter. That might resonate to many of the funders, especially those who spoke yesterday. There are also very interesting stories across each one of these journal article features. I do encourage you to go look it up. We have a wrapper that lists all of the newly published article features that's on the PLOS tech blog. This is a picture of the sauropod gigantism. Sauropod dinosaurs were the largest terrestrial animals to roam the Earth. If you want to know more about it, we have a PLOS collection of articles all about the genetic evolution of this particular species. And ALMs, in fact, play a large part in running this collection. It's an ongoing series. Just another example of its use at PLOS. One of the things that I think that many publishers do end up, it's a perennial question, which existed before ALMs will continue to exist even with it, is what are we, in fact, publishing? The bibliometrics work, a little that we've done between Martin and I in partnership with other groups outside the experts like Mike Falwell yesterday, as well as a whole host of other new labs that are really beginning to start using our data and running numbers on it, beginning to explore what are the correlations between them, what is it actually mean when you have a tweet or a new article saved into your reference library or shared on Wikipedia, et cetera. You begin to get a lot of very interesting views of what it is that you're actually publishing. So I think this is a really strong component of what we're doing with ALMs at PLOS and something that we definitely want to invest more in in the future in collaboration with the bibliometricians out there. Moving on, what do we want as a publisher? As I mentioned, all of this work is underway, but a lot of this can't, in fact, be fully, completely, and completely realized until we have a number of things, one of them being structured metadata for the cleanest results. We've been working with Wellcome Trust for more than a year, in fact, showcasing some of the articles that their funded researchers have been publishing in our journals. And something that we knew, the very initial feedback that they gave us, which is important, is we can't run the cleanest reports until we know which articles were, in fact, published by Wellcome Trust funded researchers. So we need fund ref for that. We need orchids to identify authors and disambiguate them. And the same go for institutions with, say, Ringgolt. So structured metadata is really something that we want that will support our efforts. Another thing is, does anybody know what this little image looks like? I know it's kind of tiny, but this is a metaphor for what a raw count, say, the total page views and downloads for a PLOS article is. We get a very small view. If you look at it within the broader set of other ALMs that in fact the article is getting, you might get a bigger view. But what we want to know is what is actually going on? This little thing is just a smokestack on a bigger boat. The boat might be in Atlantic, it might be on the opposite side of the world. Context does matter, especially when this smokestack is on this boat. Context really matters. We need this raw data, but we also need this raw data to have context in order to really get meaning behind it. And so I don't think this can be done with us as a single publisher. We can't do it alone. Obviously, we've never envisioned it, and we've never acted on it based on this premise. That's just not in PLOS's DNA. So coordination between other publishers, whether it's technology etiquette for publishers, so that data is available and easily consumed, what we call data stewardship. Basic degree of coordination when it comes to how we're aggregating, how we're displaying, et cetera. All of these things are really, really critical. But at the end of the day, we can't really create that context unless we have open data that's accessible for all of us. And I think we'll hear a little bit more about this, I think, in the next session, as well as, perhaps, throughout the rest of the day, about how critical it is to start building, laying the pipes, as I mentioned, for this infrastructure of the future. Just to wrap up, one quick thing is that we are reissuing a call for papers in the PLOS collection of Altmetrics. We launched back in 2012 with an aim to provide this forum for the dissemination of innovative research on metrics. So I do encourage you, if you have been doing some work, write it up, and please submit it to the PLOS collection for Altmetrics. Thank you. Thank you. Thank you. Time for one question for Jennifer. Thanks for your stay. Thank you, Jennifer. Next up, we have Motay and Rolans from Springer Science and Business Media. Are you able to do this? I think I can handle that. It's going to be more interesting by the minute, so call us. Springer Green. Springer Green, yes. The sabotage, as in the two publishers with orange in the middle. Hello. Anyway, I want to talk about article-level metrics. Here's a quick outline of my talk. It's far up there. So I want to say a few words about the importance of article-level metrics to us, what we do as Springer, our ALM team, and also a few slides on what our friends from Shaper do with article-level metrics. So why do we talk instead of what was mentioned most of yesterday, ALM metrics, why do we talk about article-level metrics? Well, we are in the long tail of publishing. And I think for our authors, it's important to know how many citations does our article have. Because I think it's a valid question, which article made a bigger contribution? Is it the article in one of the top tier journals with zero citations and they do exist? Or one of the low impact journals with tens or sometimes hundreds of citations? And actually similarly you can ask the same question, which article made a bigger impact in article with many citations? Or maybe an article widely discussed on the social web? Or is it an article with many downloads or an article discussed on CNN? I will not make any judgment which one is more important. I think we should provide all possible metrics and leave it up to the reader or the evaluation committee which one is actually important to them. Just to give you an example that sometimes citations also don't matter too much. Here's the top downloaded articles from our Springer Plus journal. You can see it has 60,000 downloads even though it has no impact factor yet and I think at that time it wasn't in progress. So even though you don't have these which are considered sort of the ground rules for publishing, people do find the content and they do download the data when it's interesting. And it's also being picked up. So here are a number of, sorry for the donors, I think there's a bit of a donors overkill but there are a few examples. For example the top one appears in Melbourne in the newspaper. The second one in El Pais, the scientific American blogs about an article in Springer Plus and also Life Hacker wrote something about it in Springer Plus article. So how can article levels being used? I think it's a very useful tool to filter research for relevant impact and quality. It will give an alternative for the impact factor although I think the impact factor will remain to exist and I'll spend a few words on journal metrics data. But it will also help us to understand how research findings are disseminated and discovered. And I think going forward we should maybe broaden the scope from only article level metrics and basically any type of metrics. Whether that is journal metrics or chapter level metrics or book level metrics, data level metrics. Yesterday in the breakout session we discovered that actually a personal level metrics might be another alternative and well any other kind of metrics. And as long as you provide as many metrics as possible I think it's up to the reader or the evaluator or the author to decide which one he thinks is valuable. So what do we do? Here's a sample article on SpringerLink. We have the citations listed on the abstract trace and the shares that will link out to our platform citations of Springer.com which is based on cross-link data. It will show you where the citations are and if the content is within Springer you can also see the inside in text place where the citation is. The other link will go to the Althmetric article page where you can see a breakdown of where your article is being shared. Similar to the previous speakers we also take this information to our editorial board meetings. So we show the editor's and chief and the editorial board that there's also something else than the impact factor. We show them the top, download the top-sided and the top Althmetric articles in our meetings. The editorial staff and marketing has access to the Althmetric Explorer which has been mentioned earlier I think. And so we use that to first to monitor our own journals to see what's being said about the journals and if needed to take action and give some good respect to the people who mentioned some articles that are published in our journals. We also started to use the article Althmetrics in marketing campaigns to highlight top-sided and top-shared and top-down-ordered articles. Here's a wonderful read-it column. There's two of them that I would like to highlight. It's the Neurostars and Weeper Citations. So the Neurostars campaign was launched in 2012. I think it's one of the first and we showed the articles that were most cited, most downloaded and most shared from Springer in the discipline of neuroscience. So the journal where it was published in MetaWidges took all of the Springer neuroscience portfolio and then draw up a list of the top-sided, downloaded and shared articles. And we did a little countdown in the last week of December and it was very well received. We continued that this year also in conjunction with the Brain Awareness Week. And now we expanded that to our quarterly Neurostars list. So if you go to www.springer.com, you can see each quarter the most shared articles of our entire discipline. So also stepping away from the journal and focusing only on the article. The second example is a diversity in citation metrics to also to raise awareness that, well, basically an often heard question is how many citations do you have for your article? Well, I have 20 citations. Yes, that depends a lot on where your citations are. And so just before the impact factor, we had this campaign running and maybe a little bit teasing, Thomson Reuters. I cannot ignore that. And so again, we showed the high top-sided articles, but we also did some calculation and from the same set of articles in a certain journal, it doesn't matter too much. We looked at a number of citations on the different platforms. As you can see, and I think that was also shown yesterday by the Wellcome Trust, similar to the exercise, you can see the differences between the platforms are quite big. If you say you have 20 citations, and that might well be true if you refer to PubMed, but Google Scholar is a full-fold higher. And our friends from BioMessentral also wrote a very nice blog that I can recommend reading, breaking down some of the information that's behind it. Okay, moving on, we also informed the authors and the editors about ALM Twitter information on our websites in newsletters, et cetera. And to further power that within Spring, we introduced an article about Magics Virtual Team with people really from all parts of the company to take this really serious as a company. So those people from editorial, marketing, BioMessentral, Springer, actually I forgot one, which is papers. I don't know what this is. There's a thing somewhere. And we will further explore all the possibilities to do with our article of Magics. Okay, last part is about what our French for Papers have been doing. They also do work with article-level metrics. For those who don't know Papers, Papers is a reference manager tool that is available for many different platforms. And recently they also launched an online version of their application, which should work on all platforms. Now I think the cool thing here is they established a partnership with Altsmetric, the company. And now what you can see here is that you basically can take your Twitter handle, or basically anybody's Twitter handle, and put it in your Papers application, and then it will bring up the articles that are being discussed amongst your followers and friends. So rather than going to Twitter and seeing what being said and schooling all the way up and down your Twitter application, it will bring you here ready to go the articles that are being discussed in your network. Similar, you can do that also for a journal that you like. In this case it's the Cerebellum, and you can see these are the latest articles that are being discussed in the social web on the Cerebellum. Okay, some conclusions and a little outlook. I think article level metrics are a more accurate way of evaluating scholarly performance. They show the dissemination of an article to scholarly and non-scholarly communication. And I think they should be a new benchmark for employers, funders and collaborators. And last, they provide filters also to find the most relevant work. We from our side do our best to also educate everybody and make them aware that they exist in the process possible way. Getting back to the journal metrics, this is something we will be rolling out in the next couple of months to really show also on the journal level all possible metrics that are available on Springfield.com. I guess that's the timer. And so you can decide for yourself whether you're looking for a journal that has a high impact factor or has a quick turnaround time or has many happy authors. That's all I have. Thank you. Great, we have time for questions for Arani. I want to go to that. I'm from the Science Policy Institute of University of Sussex. There's been a lot of talk about the April level metrics versus the June level metrics. And I fully agree with that decision in the book. But I'm also worried thinking about science on the gobs to papers. I assume that the funders are actually just contributions of grants, not as papers, but as contributions to knowledge. And think of trying to understand the growth, scientific growth in careers, not as papers, but as building the knowledge of our research. So I think we still missing a point which is how to aggregate those article level metrics into socially meaningful development. And I wonder whether there have been any developments along these lines. Well, I think that's sort of the... If I understood correctly whether it's not all about articles, but also about the impact of the person on the scientific community. Is that more or less cutting a few corners? The questions to knowledge. I think that grants in trying to... Yeah, I think there's two... Well, I guess orchid will be very important there. Because there you find your fund information, you'll find your article information and any other information. And the other side I think will be some sort of a personal level metrics where you bring together all the information around the person and the impact that his work had on the scientific community. And I hope that will be something anytime soon. I think we have a session later today about how to side the article looking at other research efforts. I think that session should address those points a bit more. Thank you again. Have a great time. APPLAUSE The last speaker today isn't as you might have noticed from the panel group from Eli. We have a shed in the movement. Hi, I run technology at Eli. My goal here is to run through these slides pretty quickly while not dying. I'm going to give a very concrete example of some of the issues and challenges that we encountered when we tried to implement ALMs. So there's a really more grounded talk rather than a conceptual talk. So Eli is a new journal. There's a little blurb about us. The thing I'm going to focus on here is that part of our mission is to look at how to maximize the reach and influence of new research and showcase new approaches for the presentation and assessment of research. So from the get go, we knew that article level metrics and alt metrics needed to be important. That's kind of the timeline. It was announced in June 2011 and we were tasked to get the journal launched by the end of 2012. I joined in May of 2012 as the only technical platform staff person and we had to get that thing built by the end of 2012. So that's what it looked like when I joined. This is a little bit like what I felt with the task. And we launched the platform on the 13th of December 2012. So here's a joy. But we had to make some very pragmatic decisions about how we were going to implement article level metrics. So we have this nice way of getting into them. On the article page, this tab, you land on a page, you get these kind of metrics. But the way they get delivered to the reader is, oh, so, I wanted to just talk about that. From this quote by Mark Andreessen, he says, software is eating the world. And the counterpoint to that is that software is kind of buggy and there are lots of bugs in software. And it's an intrinsically irreducibly complicated thing to do. And actually, I'm astonished by the idea that so much of our world economy is handcrafted by people in basements. It's the only major industry which is still like a crafted, handcrafted industry. And I want to talk about kind of two problems that we've encountered. Not problems, but things that we learned through the process of implementing ALMs. And first is the black box problem. So when we show you our ALMs and you land on that page, we don't have them stored magically somewhere. On our web servers. We just call out these other providers who provide data as the page is loaded. And so we have our platform host who gives us our usage data. And when we launched, we worked with a company called Impact Story to give us our alt data, if you like. But having this provider give you that data as your page is loaded, one of the problems is there are black box. And in December of 2013, Impact Story told us that they were going to pivot their business and no longer supply this data to publishers. And so we no longer had access to that data. So it was a bit of a challenge. So what we decided we needed to do was move away from that kind of black box scenario. Let me just add. And we only moved away from this kind of black box scenario. And we implemented our own local version of the PLOS AMM app so that we had that data two-hand locally. Sorry, legato. Legato? Legato, thanks. Yeah, I only found out that name yesterday. I'm not up to date. Okay. Let's see what the next slide is. Okay, so we started out. I started out saying that we had this kind of scenario and that Impact Story was kind of a black box. But it gets a bit worse because it turns out that our platform host is also kind of a black box. And we were kind of one of the very first publishers that they worked with to try to deliver per article usage data. So we were a bit of like a canary in the coal mine. We were really working on very work in progress kind of APIs. And an example of this guy's meant to be yellow, by the way. One of the costumers was like, 12 months after we launched the platform, we noticed that our usage metrics were starting to evaporate and usage metrics on our articles were starting to go down. We were like, what's going on? This is crazy. And we dug in and we found that we had an API call, which was the API calls basically, give me all the usage of this article. But the way it was implemented was give me all the usage of this article for the last 12 months. And so we had to rectify that. This was something, this was just a bunch of issues we encountered. We fixed that by the way. So the kind of direction I would like us to move towards is instead of having even those usage data being provided by a black box provider, I'd like to just find a way to aggregate that into our own local data store and data warehouse. And actually, I think the talk that Jennifer showed gives you an example of the kind of stuff you can start to do an experiment with if you have local copies of that data. And we're not there yet, but this is the direction we'd like to go to. We'd like to take that use of data from our platform host, from Google Analytics, or from other providers. Okay. So I've often been asked why we didn't work with Altmetric, for example. There isn't any particular reason why we haven't worked with Altmetric yet. That's just prioritization. But I could easily imagine us connecting to the Altmetric data source in a similar kind of fashion. And I hope we can get there at some point. The second kind of thing I want to talk about is what I'm going to call the intelligent design issue around how you present Altmetric. So when we launched, we were told, of course, we're going to implement the view on the metrics just like PLOS, because since PLOS do it this way, that is the standard, and we need to get to standardization in how things are presented. And if you talk to anyone from PLOS, they think that was crazy. And so we fought really hard against the idea that we should just show things the same way, because we knew that that was just a first attempt to try and show these kind of usage data. So we didn't want to do that. So this is like the class way of showing that data, and our first attempt was, one of the things that was really important for us was not to show a cumulative graph. We wanted to show pair-time bucket events so that if there was a spiky event in readership, that would keep out at you. And we also wanted to show different types of usage data in relation to each other, so you could see if you could identify some kinds of correlations. And again, in December of 2013 at the PLOS at the ALM workshop in San Francisco, this guy Juan Pablo Alperin kind of popped up with a presentation where he just briefly criticized all of the ways that people were presenting this data. And his thing with ours was like, really? You really want to do a comparison there? That just makes no sense, because the scaling is totally off. And I was like, yeah, you got a point. And in that talk, he gave this thing that he had hacked together, where you could take two different kinds of usage data, but you would scale them. So the vertical numbers are not the same, but they are scaled so that you can eyeball whether there's a correlation. We thought, OK, this is really good. Let's use that. And he presented the code, made it openly available, and we thought, OK, we're going to use that code. But he had written that code just as a demonstration for a talk, and we found that there were some issues with code that we had to overcome when we got to using it. And we did. We worked through those issues, and we contributed back to the repository that that code lives in. And so we've now implemented that kind of view on our article-level metrics pages. So this is the kind of situation we're in today. I do hope we can get the platform host data running through some kind of central data warehouse that we have control over. But if I just focus in on how our PLOS-ALM app works, currently it's going out and querying these external sources, and it does that on a scheduled basis. So that Mark Patterson, my colleague was talking about yesterday, the DOI event tracker, when that comes online, we'll be able to just plug that straight into our own local instance. So we're quite excited about that, and it will mean that that should be an even more robust way of getting that data into our platform. So the kinds of things we've learned just so far. By actually implementing PLOS-ALM, I think we've helped improve the deployment scripts through the PLOS-ALM tool. I think having a completely independent body try to implement that has made that piece of software better. I think we've significantly improved the code for visualization that we encountered, and we've worked with one, we've worked with Mark and on that. We now have our own local repository of data to experiment with. We haven't done much of that experimentation yet, but it's right there. We will be releasing an ALM module into the Drupal community, so if any other publisher is running on Drupal, and they have an ALM app, that module will connect those two things very quickly, and when DEP is released, we'll be able to tap directly into that. And I think so far with our experiences with PLOS-ALM tool and with Impact Story, we're the only publisher to have integrated two major alt-metric providers, so hopefully we can extend that and work with other people as well. And thank you. APPLAUSE I'd like to thank Aga for standing in and putting together that great presentation in such a quick time. He's working feverish here on it yesterday, and also for bringing this back on to you. But to blow that, if anybody does have any questions for you. No? OK. Thank you. You will be around for the rest of the day. Right, now to hand over to Ewan Aide, who's been... had quite a few mentions already today who will be feeding the next session. APPLAUSE All right, guys, so on our next session, when we're...
 information beforehand.  My name is Hans.  I'll give you a little overview first and then start my  presentation.  As I said, my name is Hans, working for a marketing  project with the LCP Science, specifically in the STM  Journal Search Division.  Basically covering all science areas, but really  focusing on journal and article metrics.  So not just article metrics, also journal metrics.  And authors are a very important audience in that case.  We're very much involved with that.  The key thing for me is really the passion for bridging  communication gaps.  I think that's a role that altmetrics also can take here.  I mean, altmetrics is a real new development that can  bridge that gap between non-scientists and scientists,  between alphas and deltas.  And even like our ancestors tried to do in the past with  getting communications across to animals with this horse  known in the 1907 experiment, there's a lot to do in  science communication.  Altmetrics, I think, in essence are a good way of  doing science communications.  And that's something we all can do a lot more than we've  done so far.  So a few observations.  What I do think is that scientists spend a lot of time  on their article.  And basically hardly any on the rest of it.  They don't want to spend time on marketing.  They don't want to spend time on promoting their article  anymore.  If it's been published, that's basically it.  I think good science is not good enough.  There's really a battle going on for attention.  I mean, there's a lot of information.  Yesterday we heard Mr. Farrar starting, kicking off the  day, saying, isn't there an overburden coming up already?  So it's a little bit finding the balance between a lot of  information and finding it and filtering it through  tools like altmetrics.  Funding bodies, we've seen it yesterday, are more and more  demanding.  They want to see the societal impact of research.  How can you prove that?  More and more you see also, and I think especially in the  Netherlands, I've seen a couple of examples for that,  universities building up marketing and  communications departments.  I mean, initially they never thought that would be their  role, but it's really something coming up.  They're building up these departments with a goal.  And there we are.  Publishers and marketers can help in this area.  There's really a role for publishers and  marketers to play here.  And if we don't play it, you and I can always help you out.  So what role do altmetrics play?  I see that it's the birth right now.  Scientists want to be part of it.  They want to know what's going on.  What article hits the news.  And how can I be part of that?  I'll better create great insights and networks.  I mean, if you go into the details of a detailed page,  for example, on almetric.com or an impact story, you do see  a great number of people tweeting about your work.  Who are they connected to?  Where are they from?  All these kinds of interesting network data are there one  mouse click away based.  And it's just a new angle to impact.  It's not the only way of looking at impact, but it's  really a new way of looking at impact.  And I think we need to treat it that way.  The reach beyond academia is clear, but will it also  change science?  I mean, we heard these examples the day before  yesterday at the heck day.  Our scientists are going to change the titles of their  articles without a question mark.  Is it better?  Should they do different things?  Is that really changing science?  Or is it just a side effect?  But also, I think, can confuse people.  I mean, I had a lot of feedback over the last couple of  months while we implemented almetric services around our  article, then not everybody is positive about it.  I mean, there's also people really saying, what is this?  I don't understand it.  Please explain.  So I really think publishers can play a role here.  And what can we do?  What can we do with almetrics?  I think, essentially, it's really helping scientists to  promote their work.  Because that's something I said before.  They don't like to do as easily.  Maybe they have to do, but we really need to help them there.  And the core role of a publisher historically, and I  think should ever be is advancing science by  listening to them, by collaborating, explaining  things, and also educating.  Educating systems like these tools to work with science,  and again, bridge these communication gaps.  And it's not just between alphas and debt.  It's really early career scientists versus, let's say,  older scientists.  Clinical versus basic science.  There's a lot of silos going on in science that need to be  bridged.  And I think, again, almetrics is a great tool in  achieving that.  We need to support sharing.  I mean, simply by sharing with one click button to one click  to Twitter to man-lay all kinds of repositories that brings  your article out in the open.  Don't keep it in the dark.  We can really help you in getting it out.  And again, this holistic approach to metrics, that's  important.  I mean, there's different situations where you use  different metrics.  I mean, a journal metric can be very helpful if you're just  orienting on what's an interesting journal for me.  What is the situation around that journal?  And after you've published your article, you're much more  interested in this article level, because that's really  talking about you, talking about your article.  And that's really what you're interested at that moment in  time.  I think, really, great science deserves great  real-separate ability.  And that's not always the case.  So we need to help in visualizing that.  Visualizing numbers, visualizing great science is, I  think, a great tool to get the message across.  And again, the link to journal metrics is still important.  So what did we do at Elsevier?  Basically, pilots.  Pilots are a very important tool for us to get things going,  because if you start with great plans, take months to do it,  it doesn't work.  You should take a small group of people, very enthusiastic,  very engaged, and go for it.  And just try things out.  Don't worry about making failures and see what happens.  And that's what we've done.  We piloted in different platforms, science direct  scopus.  We have a specific platform for journal homepages, which is  more of a catalog for journals.  And we did a lot of campaigns.  I'll show you a few in a minute.  And really, engaging here in conferences and discussions  is an important way of hearing and feeling and what's  going on in science.  What's important to scientists where publishers can help  them with?  My colleague Chin Mei was the guy who did the pilot on  science direct.  Just a few journals, 27, mainly in the medical area.  He was testing the donut for us as a brick display, because  I mean, the donut everybody knows from  Maltmetric.com is that the only way displaying it is that  the only visualization is that the most important one.  Well, we found out that it's not really making a big  difference in clicks, for example.  I mean, clicks are not the holy grail, but it's at least a  measure of seeing what affects the one  visual ass against the other.  I think Martin yesterday tweeted, whereas Habib, well  here he is.  Virtually with us.  Michael Habib is the product manager for Scopus.  For the people who don't know what Scopus is, am I still  there?  It's a database abstracts and articles and all this  information.  But Michael is really an evangelist.  He's basically going to every Alp-Metric conference and  he's been there from the start.  I mean, when you and one of the other great grand  challenges in 2011, I think Michael was there and he was  one of the first to adopt the concept of Alp-Metrics at  that time already.  And he implemented it without any official  contract or whatever.  He just went for it.  And I think that's a real nice approach.  I took care of Alp-Metrics on journal homepages.  I mean, you see this display.  It's already bridging a bit of this gap between the  article level and the journal level.  Because a lot of people, let's say, initially go to  Google, put in a few keywords and lend on journal on page.  That's the first point of contact for a lot of people to  see what's going on.  And one way of showing what's going on with a journal is  showing what's going on with their articles.  And again, this is just one option you could see that  a hit parade can make Alp-Metrics also a bit  more sexy in a way.  Because it's not just about numbers.  It's also a bit like, how can we make sure that people  start to like it, start to try it, start to find out what  it's really about and start clicking things?  Some statistics.  Generally, what I already said, it's not a lot of extra  traffic.  You shouldn't do it for traffic generation as a  publisher.  You really should do it as adding quality to the science  information.  It's really about quality and not about quantity.  An interesting pattern I found when we did some of these  campaigns was if you have the full text article next to a  donut, I would assume a lot of people would click the donut  and, well, at least 50% of them.  But interestingly enough, consistently, 86% of the  page is clicked in the article and only 14% is  visualisation.  What does that mean?  I'm not sure yet.  Campaigns we did.  I think also within a publishing company, it's  important to bridge the gap between marketing people and,  let's say, the more better oriented side of the business.  And the Alp-Metrics explorer tool, for example, this  analysis tool is a great way of getting the more alpha  people to know what they can do with it.  So what information is out there?  What can they see on a journal level for all publishers, for  all journals?  And how can they use this in bringing this  communication across?  We've done a few things.  And I think some of them are pretty creative.  Virtual special issues, for example.  There was an international archaeology day coming up.  The person working in the archaeology field said, hey,  let's make a selection based on Alp-Metrics, where we  basically make the top 10 or top five selection of articles  and display them in this listing, showing that  archaeology is also a sexy field where a lot of things  are happening.  We do it with emailings, where you, again, see this  visualisation next to articles.  Publishers are using the information from the analysis  tools in the editorial board meetings just to educate and  make the editorial board enthusiastic about the concept  of Alp-Metrics.  Because it's still not that well known.  I mean, everybody here knows what Alp-Metrics is, or at  least the concept of article level metrics.  But it's not like 80% of the scientists is aware of that.  It's still a lot of work to do to make them at least  aware and let alone let them play with it.  And let them use it.  One minute.  I'll stop to go to my finish.  Mandela, we've talked a lot about it already.  We more and more bring that into life.  We're integrating it more in our Alp-Metrics displays.  We're bannering about it.  And also in our sharing options.  I mean, one of the easiest way of sharing your  information is with one click of this sharing option buttons.  Some statistics, really quickly.  We've done a big mailing out in December.  Just the first trial, I think the open and click based rates  are very good.  But at one point, we realized that a lot of information for  a lot of journals is so small that you see scores of one  that doesn't make sense to display.  So we decided to put in a threshold to be a bit more  selective on what we are displaying.  Because it doesn't make sense to send out a message for a  journal where you say, OK, this is a hit parade.  And the best scoring article has a score of one.  Wow.  I mean, that doesn't work.  Feedback, generally, people are very positive about it.  They really feel there's something in it for them.  They're also critical on usefulness.  Societies are starting to demand it.  I mean, we need to see it.  We do want it as part of a contract or whatever.  But also beware of controversial articles.  Because we talked about it yesterday, sentiment is not  really taken care of yet.  I mean, a very negative article can end up high in a hit  parade.  And before you know it, you're promoting a very bad, a  very controversial article.  Feedback from individuals, I mean, it ranges from what you  see on the top, very useful to what you see at the end,  really unclear.  I don't understand it.  What criteria are you using?  Some takeaways from this.  I think, generally, we should be very receptive to talk to  people who take at least the effort to contact you.  I mean, if they want to talk about, automatically explain  it, try to make them enthused yesterday, be creative.  I talked about this virtual special issue.  A bus competition.  Prizes.  I mean, prizes, plus friends were talking about it  yesterday, prizes are a way to make it a bit more sexy, a  bit more attractive to deal with altmetrics.  And that's just another way of getting altmetrics out in the  open, making people aware of it.  And again, also be creative in things like remarkable articles  or personal altmetrics.  Final, final, final one.  What's next, we will roll out across our different  platforms.  More integration will be there with  usage and citations.  And we'll be testing alternative displays.  The buckets you heard about yesterday by my colleague,  Eliza College, are going to be tested.  Is that going to be working?  Is it not going to be working?  And we obviously keep listening.  And again, as a closing statement, I think we really  all should make big data small and try to visualize as much  as possible, because that really makes this bridge  happening.  Thank you very much.  One final tribute to the two Che Guevara's of altmetrics.  You and Mike Taylor, I'm not sure he's in yet, is coming by  bike.  So probably some traffic jam coming up.  I have these t-shirts made for you.  And then they need to say, you're on my bike.  Thank you.  Thank you.  Thank you, everyone.  I think we'll press on without any questions, fans.  But I'm sure you'll be able to find in at Copyright, if  there's anything you want us for later.  Next.  I'd just like to write on the schedule I have here.  But it will be Jennifer Lynn from PLOS.  Hi, my name is Jennifer Lynn.  Welcome to those of you who weren't able to make it  yesterday.  I will make a couple of references to some of the  conversations we've had yesterday.  So if you do get a chance and you have missed all of  yesterday's really great proceedings, do check that out.  I think that's available online.  Already, Kat?  Is that right?  Yes, already.  So my name is Jennifer Lynn.  I'm from PLOS.  We're an open access publisher.  And I just want to share a little bit with you today  about what we've been doing.  So as you know, I mentioned yesterday, not only are we  in front of building out the system that we originally  set up back in 2009 as a pioneer of article of the  metrics and alt metrics as a publisher, and henceforth  really working on it to scale it up from a cottage  industry to hopefully what will be in the future an  entire ecosystem, that is the groundwork.  So part of the larger groundwork for a scholarly  communications infrastructure.  I make one reference, this quote by Albert Einstein, that  everything that can be counted counts, not everything that  counts, can be counted.  We do take this very, very seriously, especially being  one of the first organizations to really start  thinking about what these metrics are, what are  appropriate uses of these metrics.  Sometimes those discussions can actually be quite  colorful, violent even per se.  I do count myself to be very privileged and fortunate to  be usually on the side of Martin Fennner, but in the  event that it doesn't happen, it's not pretty.  One of the things also that is part of what I want to  share is that it's actually really, really hard to do  this.  Building the system itself, where we thankfully have  Martin Fennner and a whole host of other very solid and  strong engineers and visionaries in this area.  But also the work which I would say follows behind is to  deepen the ways in which we use it across our  publishing organization.  Of course, you can't do anything with metrics until  you have them.  So a lot of the work that we have done now and that we  plan on rolling out in the near to midterm future, I'd  like to take a little bit of time and share with you.  This will be a dizzying array of very different random  images.  So I just want to set you up.  Also in part because I have tables and tables are really  never really that interesting.  So the most obvious place to start is the expression of  it on our open access articles.  The front end.  Pretty obvious.  Authors want to see the reach of the article.  So we display each of the metrics on the metrics tab.  Researchers want to find the relevant papers.  So as much as possible at this point in time, we've been  integrating it into our search mechanisms or our display  mechanisms such as on the home page, et cetera.  In recognition of the fact that by and large, researchers  do not go to a single journal's search engine to find,  discover relevant papers for them, we really seek to  collaborate with other parties.  The places where researchers do go, in fact.  And we think that that is ultimately going to be one of  the very lynch pins of ALMs or altmetrics when those are  readily made available in those spaces.  Navigating through the literature, this is exactly  what I'm talking about.  We have these signposts to help users situate themselves  when they're on an article.  We'll be launching a new visualization of this kind  in the next, hopefully before the end of the year or early  next year.  The back end.  You thought I was going to share a different  picture than you.  Sorry.  In the back end, we have built a CMS tool that makes it far  easier for many of our editorial staff as in-house and  externally to very quickly and easily transfer the curation  content, the editorial information enrichment that  they provide directly to our home pages and hopefully soon  across the entire journal site.  So we'll be using that, working on it right now.  We're also partnering with Icon Factor Recommends, that  service led by Devon West, fantastic technology that  surfaces articles based upon a citation graph.  We'll be applying some of the ALMs in our second  iteration of our Icon Factor Recommends for  article recommendations.  So as I mentioned yesterday, you'll see the slide again.  It's all about piping, not just in the context of the  entire scholarly enterprise.  The communications ecosystem.  But in-house, right?  It's the groundwork for all of these very different pieces,  not just the front end, not just the back end, but  across editorial, publication services, and marketing.  For editorial, we're using it already to help enhance our  curation work for promotion opportunities, to track  dissemination of articles published.  It'll give us a view into the type of channels that have  potential, the rate of growth amongst many of the different  subject areas.  For our publication services, it supports our  external editors.  It provides data and content behind the metrics on the  downstream impact of the publications, such as media  coverage, blog coverage, et cetera.  Just to keep in mind, and I think it has been the assumed  assumption that behind a lot of the conversations today, as  well as yesterday, when we talk about metrics, we're  certainly talking about the number itself.  Because the numbers, the raw data, rolled up, enriched,  et cetera, they enable us to do far greater things.  But what happens behind the numbers is also part and  parcel of what metrics and ALMs really are.  So surfacing that up, perhaps from our standpoint, in our  workflow, we get the numbers first.  And then we dig deeper into the things that  happen behind it.  Marketing can attract authors.  We also have email campaigns that are set up to go notify  authors, as well as other users who have signed up for  notifications for newly published papers, what's  being read, et cetera.  One of the cool new things that we've recently started, it's  going to be, in fact, a monthly series or possibly  quarterly series, is something we call the  PLOS 1 highlights.  And it's essentially using ALMs to surface some of the top  papers.  And by top, this is a very highly contestable term.  We don't really like to say top 10, et cetera, because  there are very many use cases for what top means for each  person.  But what we've been doing is calculating a ratio of PDF  downloads to HTML page views.  What we do is we take papers that were published within the  last four calendar months, and we calculate this and then  surface the top view.  It gives us a very interesting view as to what users  across the globe, across all of the subject areas that we  publish in for PLOS 1, are really looking at.  And when we have this list, month by month, we can  continue to track them, see what's happening to them as  they age, as four months becomes a year, a year  becomes two years, the citations, which generally  lag come rolling in.  This is just a very simple snapshot of the actual  posting.  I do encourage you to go online to PLOS 1's blog called  Everyone to go check out the very first one that we did.  As I mentioned, it's recurring, so there will be more  coming down the line.  And with the tracking of these papers, we can continue to  visualize these top 10s, applying things in-house,  such as our taxonomy or subject area, thesaurus, and  begin to see the different areas, the hot areas, where  people have expressed more interest in.  The great thing about ALM is you can essentially mash up any  other additional type of metadata.  And this is in part why we say and continue to say that the  possibilities for its use are absolutely endless.  And we've only begun to scratch the surface.  Another really cool editorial effort that is also  underway is what we call the ALM article feature.  It's a series that not only highlights individual  articles based on a per-journal view, but it also  celebrates in the end what is a core editorial functions of  journals, which is to curate content that matters to their  wider audiences.  Each one of these articles were selected by our in-house as  well as external editorial staff.  And then what they did was they took you to the articles  and they dove in to the ALMs, surfaced all the stories  behind them.  And they found out actually quite a lot of interesting  things, things that maybe never would have come to light  were it not for at least the beginning view of the numbers  and at first glance.  For one, the article here about prescription testosterone,  the reception from the wider audience, one of the things  that was quite interesting and provocative as through this  story was that public viewing of science does matter.  That might resonate to many of the funders, especially those  who spoke yesterday.  There are also very interesting stories across each one of  these journal article features.  I do encourage you to go look it up.  We have a wrapper that lists all of the newly published  article features that's on the PLOS tech blog.  This is a picture of the sauropod gigantism.  Sauropod dinosaurs were the largest terrestrial animals  to roam the Earth.  If you want to know more about it, we have a PLOS collection  of articles all about the genetic evolution of this  particular species.  And ALMs, in fact, play a large part in running this  collection.  It's an ongoing series.  Just another example of its use at PLOS.  One of the things that I think that many publishers do end  up, it's a perennial question, which existed before ALMs  will continue to exist even with it, is what are we, in  fact, publishing?  The bibliometrics work, a little that we've done between  Martin and I in partnership with other groups outside the  experts like Mike Falwell yesterday, as well as a whole  host of other new labs that are really beginning to start  using our data and running numbers on it, beginning to  explore what are the correlations between them,  what is it actually mean when you have a tweet or a new  article saved into your reference library or shared  on Wikipedia, et cetera.  You begin to get a lot of very interesting views of what it  is that you're actually publishing.  So I think this is a really strong component of what we're  doing with ALMs at PLOS and something that we definitely  want to invest more in in the future in collaboration with  the bibliometricians out there.  Moving on, what do we want as a publisher?  As I mentioned, all of this work is underway, but a lot of  this can't, in fact, be fully, completely, and completely  realized until we have a number of things, one of them  being structured metadata for the cleanest results.  We've been working with Wellcome Trust for more than a  year, in fact, showcasing some of the articles that their  funded researchers have been publishing in our journals.  And something that we knew, the very initial feedback that  they gave us, which is important, is we can't run the  cleanest reports until we know which articles were, in fact,  published by Wellcome Trust funded researchers.  So we need fund ref for that.  We need orchids to identify authors and disambiguate them.  And the same go for institutions with, say,  Ringgolt.  So structured metadata is really something that we want  that will support our efforts.  Another thing is, does anybody know what this little  image looks like?  I know it's kind of tiny, but this is a metaphor for what a  raw count, say, the total page views and downloads for a  PLOS article is.  We get a very small view.  If you look at it within the broader set of other ALMs that  in fact the article is getting, you might get a bigger view.  But what we want to know is what is actually going on?  This little thing is just a smokestack on a bigger boat.  The boat might be in Atlantic, it might be on the opposite  side of the world.  Context does matter, especially when this smokestack is on  this boat.  Context really matters.  We need this raw data, but we also need this raw data to have  context in order to really get meaning behind it.  And so I don't think this can be done with us as a single  publisher.  We can't do it alone.  Obviously, we've never envisioned it, and we've never  acted on it based on this premise.  That's just not in PLOS's DNA.  So coordination between other publishers, whether it's  technology etiquette for publishers, so that data is  available and easily consumed, what we call data  stewardship.  Basic degree of coordination when it comes to how we're  aggregating, how we're displaying, et cetera.  All of these things are really, really critical.  But at the end of the day, we can't really create that  context unless we have open data that's accessible for all  of us.  And I think we'll hear a little bit more about this, I  think, in the next session, as well as, perhaps, throughout  the rest of the day, about how critical it is to start  building, laying the pipes, as I mentioned, for this  infrastructure of the future.  Just to wrap up, one quick thing is that we are  reissuing a call for papers in the PLOS collection of  Altmetrics.  We launched back in 2012 with an aim to provide this forum  for the dissemination of innovative research on  metrics.  So I do encourage you, if you have been doing some work,  write it up, and please submit it to the PLOS collection  for Altmetrics.  Thank you.  Thank you.  Thank you.  Time for one question for Jennifer.  Thanks for your stay.  Thank you, Jennifer.  Next up, we have Motay and Rolans from Springer Science  and Business Media.  Are you able to do this?  I think I can handle that.  It's going to be more interesting by the minute,  so call us.  Springer Green.  Springer Green, yes.  The sabotage, as in the two publishers with orange in the  middle.  Hello.  Anyway, I want to talk about article-level metrics.  Here's a quick outline of my talk.  It's far up there.  So I want to say a few words about the importance of  article-level metrics to us, what we do as Springer, our  ALM team, and also a few slides on what our friends from  Shaper do with article-level metrics.  So why do we talk instead of what was mentioned most of  yesterday, ALM metrics, why do we talk about article-level  metrics?  Well, we are in the long tail of publishing.  And I think for our authors, it's important to know how  many citations does our article have.  Because I think it's a valid question, which article made a bigger contribution?  Is it the article in one of the top tier journals with zero citations and they do exist?  Or one of the low impact journals with tens or sometimes hundreds of citations?  And actually similarly you can ask the same question, which article made a bigger impact  in article with many citations?  Or maybe an article widely discussed on the social web?  Or is it an article with many downloads or an article discussed on CNN?  I will not make any judgment which one is more important.  I think we should provide all possible metrics and leave it up to the reader or the evaluation  committee which one is actually important to them.  Just to give you an example that sometimes citations also don't matter too much.  Here's the top downloaded articles from our Springer Plus journal.  You can see it has 60,000 downloads even though it has no impact factor yet and I think at  that time it wasn't in progress.  So even though you don't have these which are considered sort of the ground rules for  publishing, people do find the content and they do download the data when it's interesting.  And it's also being picked up.  So here are a number of, sorry for the donors, I think there's a bit of a donors overkill  but there are a few examples.  For example the top one appears in Melbourne in the newspaper.  The second one in El Pais, the scientific American blogs about an article in Springer  Plus and also Life Hacker wrote something about it in Springer Plus article.  So how can article levels being used?  I think it's a very useful tool to filter research for relevant impact and quality.  It will give an alternative for the impact factor although I think the impact factor  will remain to exist and I'll spend a few words on journal metrics data.  But it will also help us to understand how research findings are disseminated and discovered.  And I think going forward we should maybe broaden the scope from only article level  metrics and basically any type of metrics.  Whether that is journal metrics or chapter level metrics or book level metrics, data  level metrics.  Yesterday in the breakout session we discovered that actually a personal level metrics might  be another alternative and well any other kind of metrics.  And as long as you provide as many metrics as possible I think it's up to the reader  or the evaluator or the author to decide which one he thinks is valuable.  So what do we do?  Here's a sample article on SpringerLink.  We have the citations listed on the abstract trace and the shares that will link out to  our platform citations of Springer.com which is based on cross-link data.  It will show you where the citations are and if the content is within Springer you can  also see the inside in text place where the citation is.  The other link will go to the Althmetric article page where you can see a breakdown of where  your article is being shared.  Similar to the previous speakers we also take this information to our editorial board meetings.  So we show the editor's and chief and the editorial board that there's also something  else than the impact factor.  We show them the top, download the top-sided and the top Althmetric articles in our meetings.  The editorial staff and marketing has access to the Althmetric Explorer which has been  mentioned earlier I think.  And so we use that to first to monitor our own journals to see what's being said about  the journals and if needed to take action and give some good respect to the people who  mentioned some articles that are published in our journals.  We also started to use the article Althmetrics in marketing campaigns to highlight top-sided  and top-shared and top-down-ordered articles.  Here's a wonderful read-it column.  There's two of them that I would like to highlight.  It's the Neurostars and Weeper Citations.  So the Neurostars campaign was launched in 2012.  I think it's one of the first and we showed the articles that were most cited, most downloaded  and most shared from Springer in the discipline of neuroscience.  So the journal where it was published in MetaWidges took all of the Springer neuroscience portfolio  and then draw up a list of the top-sided, downloaded and shared articles.  And we did a little countdown in the last week of December and it was very well received.  We continued that this year also in conjunction with the Brain Awareness Week.  And now we expanded that to our quarterly Neurostars list.  So if you go to www.springer.com, you can see each quarter the most shared articles of  our entire discipline.  So also stepping away from the journal and focusing only on the article.  The second example is a diversity in citation metrics to also to raise awareness that, well,  basically an often heard question is how many citations do you have for your article?  Well, I have 20 citations.  Yes, that depends a lot on where your citations are.  And so just before the impact factor, we had this campaign running and maybe a little bit  teasing, Thomson Reuters.  I cannot ignore that.  And so again, we showed the high top-sided articles, but we also did some calculation  and from the same set of articles in a certain journal, it doesn't matter too much.  We looked at a number of citations on the different platforms.  As you can see, and I think that was also shown yesterday by the Wellcome Trust, similar  to the exercise, you can see the differences between the platforms are quite big.  If you say you have 20 citations, and that might well be true if you refer to PubMed,  but Google Scholar is a full-fold higher.  And our friends from BioMessentral also wrote a very nice blog that I can recommend reading,  breaking down some of the information that's behind it.  Okay, moving on, we also informed the authors and the editors about ALM Twitter information  on our websites in newsletters, et cetera.  And to further power that within Spring, we introduced an article about Magics Virtual  Team with people really from all parts of the company to take this really serious as  a company.  So those people from editorial, marketing, BioMessentral, Springer, actually I forgot  one, which is papers.  I don't know what this is.  There's a thing somewhere.  And we will further explore all the possibilities to do with our article of Magics.  Okay, last part is about what our French for Papers have been doing.  They also do work with article-level metrics.  For those who don't know Papers, Papers is a reference manager tool that is available  for many different platforms.  And recently they also launched an online version of their application, which should  work on all platforms.  Now I think the cool thing here is they established a partnership with Altsmetric, the company.  And now what you can see here is that you basically can take your Twitter handle, or  basically anybody's Twitter handle, and put it in your Papers application, and then it  will bring up the articles that are being discussed amongst your followers and friends.  So rather than going to Twitter and seeing what being said and schooling all the way  up and down your Twitter application, it will bring you here ready to go the articles that  are being discussed in your network.  Similar, you can do that also for a journal that you like.  In this case it's the Cerebellum, and you can see these are the latest articles that  are being discussed in the social web on the Cerebellum.  Okay, some conclusions and a little outlook.  I think article level metrics are a more accurate way of evaluating scholarly performance.  They show the dissemination of an article to scholarly and non-scholarly communication.  And I think they should be a new benchmark for employers, funders and collaborators.  And last, they provide filters also to find the most relevant work.  We from our side do our best to also educate everybody and make them aware that they exist  in the process possible way.  Getting back to the journal metrics, this is something we will be rolling out in the  next couple of months to really show also on the journal level all possible metrics  that are available on Springfield.com.  I guess that's the timer.  And so you can decide for yourself whether you're looking for a journal that has a high  impact factor or has a quick turnaround time or has many happy authors.  That's all I have.  Thank you.  Great, we have time for questions for Arani.  I want to go to that.  I'm from the Science Policy Institute of University of Sussex.  There's been a lot of talk about the April level metrics versus the June level metrics.  And I fully agree with that decision in the book.  But I'm also worried thinking about science on the gobs to papers.  I assume that the funders are actually just contributions of grants, not as papers, but  as contributions to knowledge.  And think of trying to understand the growth, scientific growth in careers, not as papers,  but as building the knowledge of our research.  So I think we still missing a point which is how to aggregate those article level metrics  into socially meaningful development.  And I wonder whether there have been any developments along these lines.  Well, I think that's sort of the...  If I understood correctly whether it's not all about articles, but also about the impact  of the person on the scientific community.  Is that more or less cutting a few corners?  The questions to knowledge.  I think that grants in trying to...  Yeah, I think there's two... Well, I guess orchid will be very important there.  Because there you find your fund information, you'll find your article information and any other information.  And the other side I think will be some sort of a personal level metrics where you bring together  all the information around the person and the impact that his work had on the scientific community.  And I hope that will be something anytime soon.  I think we have a session later today about how to side the article looking at other research efforts.  I think that session should address those points a bit more.  Thank you again.  Have a great time.  APPLAUSE  The last speaker today isn't as you might have noticed from the panel group from Eli.  We have a shed in the movement.  Hi, I run technology at Eli.  My goal here is to run through these slides pretty quickly while not dying.  I'm going to give a very concrete example of some of the issues and challenges that we encountered when we tried to implement ALMs.  So there's a really more grounded talk rather than a conceptual talk.  So Eli is a new journal.  There's a little blurb about us.  The thing I'm going to focus on here is that part of our mission is to look at how to maximize the reach and influence of new research  and showcase new approaches for the presentation and assessment of research.  So from the get go, we knew that article level metrics and alt metrics needed to be important.  That's kind of the timeline.  It was announced in June 2011 and we were tasked to get the journal launched by the end of 2012.  I joined in May of 2012 as the only technical platform staff person and we had to get that thing built by the end of 2012.  So that's what it looked like when I joined.  This is a little bit like what I felt with the task.  And we launched the platform on the 13th of December 2012.  So here's a joy.  But we had to make some very pragmatic decisions about how we were going to implement article level metrics.  So we have this nice way of getting into them.  On the article page, this tab, you land on a page, you get these kind of metrics.  But the way they get delivered to the reader is, oh, so, I wanted to just talk about that.  From this quote by Mark Andreessen, he says, software is eating the world.  And the counterpoint to that is that software is kind of buggy and there are lots of bugs in software.  And it's an intrinsically irreducibly complicated thing to do.  And actually, I'm astonished by the idea that so much of our world economy is handcrafted by people in basements.  It's the only major industry which is still like a crafted, handcrafted industry.  And I want to talk about kind of two problems that we've encountered.  Not problems, but things that we learned through the process of implementing ALMs.  And first is the black box problem.  So when we show you our ALMs and you land on that page, we don't have them stored magically somewhere.  On our web servers.  We just call out these other providers who provide data as the page is loaded.  And so we have our platform host who gives us our usage data.  And when we launched, we worked with a company called Impact Story to give us our alt data, if you like.  But having this provider give you that data as your page is loaded, one of the problems is there are black box.  And in December of 2013, Impact Story told us that they were going to pivot their business and no longer supply this data to publishers.  And so we no longer had access to that data.  So it was a bit of a challenge.  So what we decided we needed to do was move away from that kind of black box scenario.  Let me just add.  And we only moved away from this kind of black box scenario.  And we implemented our own local version of the PLOS AMM app so that we had that data two-hand locally.  Sorry, legato.  Legato?  Legato, thanks.  Yeah, I only found out that name yesterday.  I'm not up to date.  Okay.  Let's see what the next slide is.  Okay, so we started out.  I started out saying that we had this kind of scenario and that Impact Story was kind of a black box.  But it gets a bit worse because it turns out that our platform host is also kind of a black box.  And we were kind of one of the very first publishers that they worked with to try to deliver per article usage data.  So we were a bit of like a canary in the coal mine.  We were really working on very work in progress kind of APIs.  And an example of this guy's meant to be yellow, by the way.  One of the costumers was like, 12 months after we launched the platform,  we noticed that our usage metrics were starting to evaporate and usage metrics on our articles were starting to go down.  We were like, what's going on? This is crazy.  And we dug in and we found that we had an API call, which was the API calls basically,  give me all the usage of this article.  But the way it was implemented was give me all the usage of this article for the last 12 months.  And so we had to rectify that.  This was something, this was just a bunch of issues we encountered.  We fixed that by the way.  So the kind of direction I would like us to move towards is instead of having even those usage data being provided by a black box provider,  I'd like to just find a way to aggregate that into our own local data store and data warehouse.  And actually, I think the talk that Jennifer showed gives you an example of the kind of stuff you can start to do an experiment with  if you have local copies of that data.  And we're not there yet, but this is the direction we'd like to go to.  We'd like to take that use of data from our platform host, from Google Analytics, or from other providers.  Okay.  So I've often been asked why we didn't work with Altmetric, for example.  There isn't any particular reason why we haven't worked with Altmetric yet.  That's just prioritization.  But I could easily imagine us connecting to the Altmetric data source in a similar kind of fashion.  And I hope we can get there at some point.  The second kind of thing I want to talk about is what I'm going to call the intelligent design issue around how you present Altmetric.  So when we launched, we were told, of course, we're going to implement the view on the metrics just like PLOS,  because since PLOS do it this way, that is the standard, and we need to get to standardization in how things are presented.  And if you talk to anyone from PLOS, they think that was crazy.  And so we fought really hard against the idea that we should just show things the same way,  because we knew that that was just a first attempt to try and show these kind of usage data.  So we didn't want to do that.  So this is like the class way of showing that data, and our first attempt was,  one of the things that was really important for us was not to show a cumulative graph.  We wanted to show pair-time bucket events so that if there was a spiky event in readership, that would keep out at you.  And we also wanted to show different types of usage data in relation to each other,  so you could see if you could identify some kinds of correlations.  And again, in December of 2013 at the PLOS at the ALM workshop in San Francisco,  this guy Juan Pablo Alperin kind of popped up with a presentation where he just briefly criticized all of the ways that people were presenting this data.  And his thing with ours was like, really? You really want to do a comparison there?  That just makes no sense, because the scaling is totally off.  And I was like, yeah, you got a point. And in that talk, he gave this thing that he had hacked together,  where you could take two different kinds of usage data, but you would scale them.  So the vertical numbers are not the same, but they are scaled so that you can eyeball whether there's a correlation.  We thought, OK, this is really good. Let's use that.  And he presented the code, made it openly available, and we thought, OK, we're going to use that code.  But he had written that code just as a demonstration for a talk,  and we found that there were some issues with code that we had to overcome when we got to using it.  And we did. We worked through those issues, and we contributed back to the repository that that code lives in.  And so we've now implemented that kind of view on our article-level metrics pages.  So this is the kind of situation we're in today.  I do hope we can get the platform host data running through some kind of central data warehouse that we have control over.  But if I just focus in on how our PLOS-ALM app works,  currently it's going out and querying these external sources, and it does that on a scheduled basis.  So that Mark Patterson, my colleague was talking about yesterday, the DOI event tracker,  when that comes online, we'll be able to just plug that straight into our own local instance.  So we're quite excited about that, and it will mean that that should be an even more robust way of getting that data into our platform.  So the kinds of things we've learned just so far.  By actually implementing PLOS-ALM, I think we've helped improve the deployment scripts through the PLOS-ALM tool.  I think having a completely independent body try to implement that has made that piece of software better.  I think we've significantly improved the code for visualization that we encountered,  and we've worked with one, we've worked with Mark and on that.  We now have our own local repository of data to experiment with.  We haven't done much of that experimentation yet, but it's right there.  We will be releasing an ALM module into the Drupal community, so if any other publisher is running on Drupal,  and they have an ALM app, that module will connect those two things very quickly,  and when DEP is released, we'll be able to tap directly into that.  And I think so far with our experiences with PLOS-ALM tool and with Impact Story,  we're the only publisher to have integrated two major alt-metric providers,  so hopefully we can extend that and work with other people as well.  And thank you.  APPLAUSE  I'd like to thank Aga for standing in and putting together that great presentation in such a quick time.  He's working feverish here on it yesterday, and also for bringing this back on to you.  But to blow that, if anybody does have any questions for you.  No? OK. Thank you.  You will be around for the rest of the day.  Right, now to hand over to Ewan Aide, who's been...  had quite a few mentions already today who will be feeding the next session.  APPLAUSE  All right, guys, so on our next session,  when we're...
Altmetrics_and_publishers_0_215.png Altmetrics_and_publishers_216_2207.png Altmetrics_and_publishers_2208_2435.png Altmetrics_and_publishers_2436_2483.png Altmetrics_and_publishers_2484_4739.png Altmetrics_and_publishers_4740_11243.png Altmetrics_and_publishers_11244_11567.png Altmetrics_and_publishers_11568_13643.png Altmetrics_and_publishers_13644_16511.png Altmetrics_and_publishers_16512_17663.png Altmetrics_and_publishers_17664_20363.png Altmetrics_and_publishers_20364_21011.png Altmetrics_and_publishers_21012_23171.png Altmetrics_and_publishers_23172_26531.png Altmetrics_and_publishers_26532_26603.png Altmetrics_and_publishers_26604_26783.png Altmetrics_and_publishers_26784_26867.png Altmetrics_and_publishers_26868_26963.png Altmetrics_and_publishers_26964_27563.png Altmetrics_and_publishers_27564_30551.png Altmetrics_and_publishers_30552_30647.png Altmetrics_and_publishers_30648_30791.png Altmetrics_and_publishers_30792_31775.png Altmetrics_and_publishers_31776_32159.png Altmetrics_and_publishers_32160_33623.png Altmetrics_and_publishers_33624_36239.png Altmetrics_and_publishers_36240_37991.png Altmetrics_and_publishers_37992_38447.png Altmetrics_and_publishers_38448_39263.png Altmetrics_and_publishers_39264_41507.png Altmetrics_and_publishers_41508_42155.png Altmetrics_and_publishers_42156_43967.png Altmetrics_and_publishers_43968_44279.png Altmetrics_and_publishers_44280_45419.png Altmetrics_and_publishers_45420_46187.png Altmetrics_and_publishers_46188_46463.png Altmetrics_and_publishers_46464_46811.png Altmetrics_and_publishers_46812_47579.png Altmetrics_and_publishers_47580_48839.png Altmetrics_and_publishers_48840_49679.png Altmetrics_and_publishers_49680_49727.png Altmetrics_and_publishers_49728_50279.png Altmetrics_and_publishers_50280_52835.png Altmetrics_and_publishers_52836_53843.png Altmetrics_and_publishers_53844_55775.png Altmetrics_and_publishers_55776_55847.png Altmetrics_and_publishers_55848_56831.png Altmetrics_and_publishers_56832_57647.png Altmetrics_and_publishers_57648_59483.png Altmetrics_and_publishers_59484_61343.png Altmetrics_and_publishers_61344_61595.png Altmetrics_and_publishers_61596_63671.png Altmetrics_and_publishers_63672_63911.png Altmetrics_and_publishers_63912_64823.png Altmetrics_and_publishers_64824_65327.png Altmetrics_and_publishers_65328_65591.png Altmetrics_and_publishers_65592_67175.png Altmetrics_and_publishers_67176_67319.png Altmetrics_and_publishers_67320_68327.png Altmetrics_and_publishers_68328_69023.png Altmetrics_and_publishers_69024_69179.png Altmetrics_and_publishers_69180_73151.png Altmetrics_and_publishers_73152_73487.png Altmetrics_and_publishers_73488_73919.png Altmetrics_and_publishers_73920_74399.png Altmetrics_and_publishers_74400_74939.png Altmetrics_and_publishers_74940_76271.png Altmetrics_and_publishers_76272_76799.png Altmetrics_and_publishers_76800_77411.png Altmetrics_and_publishers_77412_77939.png Altmetrics_and_publishers_77940_78071.png Altmetrics_and_publishers_78072_78287.png Altmetrics_and_publishers_78288_79523.png Altmetrics_and_publishers_79524_80711.png Altmetrics_and_publishers_80712_82319.png Altmetrics_and_publishers_82320_82607.png Altmetrics_and_publishers_82608_83603.png Altmetrics_and_publishers_83604_83723.png Altmetrics_and_publishers_83724_84035.png Altmetrics_and_publishers_84036_84323.png Altmetrics_and_publishers_84324_84779.png Altmetrics_and_publishers_84780_84875.png Altmetrics_and_publishers_84876_84947.png Altmetrics_and_publishers_84948_86519.png Altmetrics_and_publishers_86520_87563.png Altmetrics_and_publishers_87564_87731.png Altmetrics_and_publishers_87732_88703.png Altmetrics_and_publishers_88704_88724.png