Day 1: Altmetrics: Now and Next and Altmetrics In Research Evaluation
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Adie, Euan
Piowar, Heather
Priem, Jason
Lowenberg, Daniella
Allen, Polly
Liu, Jean
Williams, Kate
Kennison, Rebecca
Kirk, Martin
Now and next session: - First up was Euan Adie (Altmetric) who gave us the “10,000 foot view” from Altmetric of the past year. Among the trends he highlighted included academic twitter’s growth year over year and the rise of pre-prints. Another trend perhaps alarming to this audience is that publisher meta-data is getting worse. Without the meta tags, it’s difficult for Altmetric and others to make connections and track mentions. - Following Euan, Heather Piowar and Jason Priem (ImpactStory) who discussed the importance of transparency and open source code, and provided a sneak peak at their new project PaperBuzz, based on CrossRef event and Unpaywall user data. The aim is to help tell you “what’s buzzing this week” by topic or scholarly/non-scholarly audiences, based on open data and code. What are the things people are talking about online and what they’re reading based on open access data. While the sources do not currently provide as much data or data as high in quality, the expectation is that this will continue to change and improve moving forward. - Next up was Daniella Lowenberg (California Digital Library) discussing the “Make Data Count” project. Begun in 2014, this project aims to bring metrics to the data level. Researchers were surveyed about which metrics were most important to them, and based on the survey, the project is now tackling how to best structure data presentation to enable tracking similar to journal articles. The project is in process of producing a draft for posting as a pre-print to get feedback from the community. The project aims to make usage trackage easier and engage researchers across communities, iterating recommendations as they learn from the process. - After Daniella, Polly Allen (Plum Analytics/Elsevier) spoke about Altmetrics and societal impacts – raising attention to issues involved in tracking actual impact (using as an example clinical citations). Polly also made important points regarding the use of big data in general and the biases that are still contained within it, despite the presumption by many that data removes such biases. She encouraged us to fully think through what success means and the societal implications of using any metric, while celebrating the wealth of information that remains to be explored. - Jean Liu (Altmetric) then provided an absolutely adorable narrative about Altmetric for books entitled The Donut’s Quest, detailing the challenges of tracking altmetrics for books. Tracking is surprisingly difficult because there are multiple domains where books live online. Nevertheless, Altmetric has seen an enormous amount of attention paid to books and continue to work to track it. Altmetrics in Research Evaluation session: - First Kate Williams (University of Cambridge) spoke about “Altmetrics in practice: understanding emerging cultures of evaluation,” and her research exploring the social impact of using societal impact metrics. Kate employs sociological and ethnographic analysis to better understand the “space between fields that shapes the culture of evaluation,” and assess impact in a wider context. She discussed how altmetrics and other new forms of evaluative capital can inform development of this framework, but more research is needed around how metrics take meaning in practice. Kate further explained this “space between” involves all stakeholders in the research ecosystem — from scholars to policy makers, publishers, and the public — and :shifting permeable borders to allow techniques to transfer and exist in this hybrid space.” - Next Rebecca Kennison (K|N Consultants) gave an update on the Mellon-funded HuMetricsHSS project, which explores the potential of altmetrics as value-based indicators. The goal is to create a framework addressing “all aspects of scholarly life well-lived,” nurturing values in practice, and empowering scholars to enrich their impact narratives. - The final presentation featured Martin Kirk, PhD, (University of British Columbia) on the current landscape of research evaluation metrics in Canada. Martin reiterated a common theme: current research evaluation conversations are “ALL about impact.” In Canada there is more demand to show definitive results from taxpayer funding, and a highly competitive climate for securing these funds.
 Can you come and sit here please? This works great. Yeah, go for it. Okay, thank you. Kat, I'm just going to do this for you. Okay, we have SuperWizClaire helping us out with slides and things so when I can't work the technology. Okay, hope everyone managed to get coffee and is ready for an exciting day of talks. Welcome to 4am, I'm Kat, I work at Altmetric and I'm one of the organisers of the conference so I hope you have a great time here. So I'm pleased to introduce you all of the speakers of our Altmetrics Now and Next session. And what we've really tried to do with the content in this session is to give you a bit of an overview of some of the latest thinking in Altmetrics. Some of the new ideas and the new technologies that are starting to emerge. And to really kind of set the scene for the next couple of days of discussion. We have a slight program change because I'm afraid Katie from PLOS couldn't make it in the end. But the good news is that that gives everyone else a little bit more time to talk. So I won't be cutting them off too quickly. I think we'll have everyone talk and then we'll do questions at the end. So if you do think of anything please try and hold on to it till then. And we'll get started. So first up we have Ewan Addy. Ewan is the founder of Altmetric. He was a former researcher and he's going to share with you some of the thinking that we've been doing over the last year and what we're thinking of moving forward with Altmetrics. Hi everyone. So what I was going to talk about is not so much anything to do with Altmetric.com. If you're interested now you can talk to people here like I'm here and Gina and Kat. It was more like things we've noticed as Altmetric.com. So obviously we get a lot of people talking to us about Altmetrics. And some things we can help with, some things we can't, but it's to give you an idea of the kind of queries we're getting and things we've noticed. So it's this 10,000 feet view from Altmetric.com. Is it showing up? Oh, the first thing I should say is that following Antly's very clever idea, I've also put my slides online because I realized you can't necessarily see the bottom of them. So they're there bitly Addy 4 a.m. So without further ado, jumping straight into it, the first thing is that we're seeing academic Twitter still growing. So this is a graph of tweets that mention articles, not books or data sets or software or anything like that, specifically articles. And the interesting thing is it's still growing by quite a high percent. So from 2014 to 15, it was 45% growth, 15 to 16, 32%. It looks like this year it's going to grow again by between 30 and 40%. And that by itself without context isn't necessarily the interesting part. The interesting part of the Twitter itself is only growing 10% a year. So for whatever reason, we're finding more articles being mentioned online. Now, why is that? Is it because more scientists are using Twitter? You know, they're overrepresented in that 10% of new users. It's because journals are getting better at social media. I don't know, but we've definitely seen the trend. Second interesting thing, which I think is a very positive thing, is that people are talking about preprints more. So this year has really been the year where we saw this kind of data take off. People have always talked about archive online, but they're always talking about bioarchive. So this is a very kind of loose analysis. I'd be interested if anyone else wants to take it further. But just quickly looking at plus one, I picked plus one just because it's, you know, a big journal has a lot of data. And looking at how many articles it published in 2016, and then comparing it to bioarchive, and then seeing how many mentions in total did it get, for every 100 papers, how many tweets did it get, how many blogs, how many policy citations, this kind of thing. So the interesting thing is with bioarchive, actually in general, and there could be all sorts of reasons for this, but it actually gets more blog posts than tweets, a pair of 100 papers, than plus one. And I could say that's because, you know, they're inviting people to submit preprints to bioarchive, and you're kind of selecting a certain type of research to go in there, or maybe it's, you know, researchers think, again, just like with open access at the beginning, it's a particularly controversial topic or something that's quite exciting, so they deliberately make it open as soon as possible. There's all sorts of reasons. But regardless, people are finding articles on preprint servers and talking about them. So the opposite, as you might expect, maybe, is intuitive, is you get fewer news stories, policy sites, Wikipedia sites from the preprints, from bioarchive, than you do plus one. So there's still more news stories written about things in published journals than there are in preprint servers, on bioarchive specifically, but they still exist. So you do still get news stories, and you do still get policy documents. So there's an example here about, it was a preprint about Zika virus on bioarchive, and it was already picked up by the CDC. So to be clear, I think this is a great thing that preprints is, you know, kind of taking off. I don't think that's a controversial view in this room, probably. What I think is maybe a problem is there's no difference in the way these things are cited. If you go to Wikipedia page and it cites a preprint, it cites it in exactly the same way as it cites peer-reviewed research. If you look at a news article, in the vast majority of cases, the way the news article talks about the research is exactly the same as it would talk about peer-reviewed research, with a few notable exceptions. So there's Spiegel there was one example I could find where the journalist actually goes, it's research that hasn't been peer-reviewed yet. And I don't necessarily think it's an old metrics community issue or responsibility to fix, but I think it's someone's responsibility to at least think about it and think, is this what we want? You know, we talked, I remember four or five years ago, there was a lot of the crisis in scientific communications, it was about, you know, people overhyping results and things in the mainstream media. If that was peer-reviewed research, how much worse is it going to be if, you know, it's accepted, as long as it's on a preprint server, it's now, you know, part of the scholarly record, I'm going to say whatever we want about it. It's not all negative, some of these things seem negative, they're not necessarily. But here is one that's unfortunate for us, especially. We think publisher metadata get worse. So Google Scholar, when it first came out, really kind of incentivized a lot of publishers to put meta tags on their page with good metadata in it, because if they weren't there, they wouldn't get indexed in Google Scholar, which is important for discovery and basically ties to your business as a publisher. Nowadays, we don't see that so often. If anything, we sometimes hear from publishers that they have meta tags, but it's only activated when it's the Google bot that crawls their pages. I cannot conceive of why this would be a good idea, but there you have it. There's a move away from the kind of traditional scholarly platforms, which for all of their strengths and weaknesses, you know, do have a record of putting good metadata on the pages, and towards more custom platforms, where often the metadata is an afterthought. So, you know, people say the metadata is the love letter to the future and things, so people kind of forget that. They don't include it in their first pass of their new publishing platform, no meta tags, no way for us and other metric providers to kind of link a paper with, or to rather connect an actual research output with a link. Another thing, books versus articles. So, I feel now pretty confident that we've got good data on article sharing on social media. So, you know, if you, there's, you know, it's never 100%, there's always problems. We talked about this a little bit in the workshop yesterday, but in general, if you tweet a link to a paper, I'm pretty confident that at least somebody in the all-media community is going to be picking it up properly. But books, and I'm not going to talk a lot about this because I know Gene has got something coming up on it, but books are very different. And again, it's to do with not just technological issues, but also kind of social ones, the way people cite books. So, there are some example tweets there. You can't really see them because they're down at the bottom. But it's, for example, people don't link to a book. If you think about it, you know, where is the canonical place for a book to live? Is it Amazon? Is it a publisher site? You know, is it your institutional repository? It's not a clear kind of accepted way to do it. There's other connected problems around books, not necessarily having the same level of quality metadata as articles. It's certainly not with good unique identifiers like DOI, so that is changing. So, it feels like it's a little bit behind. So, you know, we talk about data sets and software being kind of second-class research out there, citizens. But actually, from an all-metrics perspective, books are, you know, they're still out there as well. They're not quite at the same level as articles. So, moving more on to good things, I always, I think other people working in all-metrics live experience as well, but I always really like it when people actually know what all-metrics are. And occasionally, I meet a researcher through friends or whatever, and they'll be like, oh, yeah, the donut guy, I know you. And I always make it feel really good. And most recently, someone said to me, oh, how much I finally know you do because you're in the economist. That means that all-metrics has really made it. Okay, maybe, I don't know what that standard is, but there you go, that did happen this year. And I think it is a sign, amongst many other things, that awareness and update is still growing. So, I really wish we had hard data here. And I know it's Michael Habib around. Like, I know Scopus used to do that, you know, what percentage of people have heard about all-metrics. And I remember it was 0.5% the very first year. I wish I knew what that percentage was now if we had the same kind of longitudinal study. We don't, but we do know that, you know, increasingly, we have more researchers who know about all-metrics. More publishers, more funders and governments, especially from an all-metric.com perspective. We have many more publisher, sorry, many more funder and government customers than we have before. And in general, their approach to all-metrics is also quite sensible. So, the worry was always, you know, people are going to, you know, have their tenure practice judged on how many tweets they have and this kind of thing. And it never really seemed like a real worry. And actually, it isn't. Like, we've not worked with anyone who's really come to us and said, I really want this one number to judge people on. They're generally not like that. That's very good. One thing that we've realized as a company is that training and awareness and kind of pragmatic best practices as important an activity as the actual development of the all-metrics, which is easy to say is actually quite hard to put into practice. But we spend a lot of time. We have, I think, as a company, we have more people on the non-technical side than we do the development side. It's still pretty close, but a lot of the work we do now is in training and awareness. The flip side of this is, and I think maybe Rebecca, I don't know if this is the paper that Rebecca meant, but things like this, so there was this blog post recently in the LSE Impact blog, the click-bake and impact one, talking about a paper that had a high-up-metric score. But the content was about the case for colonialism in a TNF journal. And the authors say, well, you know, this article got published and had a lot of attention about it. And the highlighted bit here is, by the rules of modern academia, this is a triumph. And then following on from that, separately, earlier on, we've got two tweets from people saying, in this conversation going, you know, we're talking about this colonialism paper. And then you stop tweeting about it because it's adding to the metric score. You're just giving the guy extra metric points. And I don't know quite how to feel about this. On the one hand, people assume that our metrics are accepted. It's obviously not a niche thing. If people are worrying about it at this extent, on the other hand, obviously it's completely wrong. And the, you know, awareness and the training and stuff that we're trying to do is saying, it's not about the counts. You always have to look at the mentions in the context and things maybe it hasn't filtered all the way through. It's a ongoing challenge. Oh, okay. I'm almost done. So then because we're at an all metrics conference and, well, you know, we talk about indicators most of the time, I thought what better place to make up a new metric, which I'll call the e-index. It's precise to one decimal point. And it's calculated by looking at the number of hits on Google News for any search term. I thought, oh, let's do that size. Let's see if we can gauge how popular our metrics is in the public consciousness. So for all metrics, 375.0 hits on Google News. And for the impact factor, there's only 388.0 hits on Google News. So, yeah, that's pretty good going. And then just to bring everyone down to Earth, I look for CRISPR and it was 193,000 hits on Google News. So, you know, not sure about the public consciousness, but at least maybe in the places that matter, you know, we're getting somewhere. I'll leave it there. Thanks a lot. I'm looking forward to taking on the breaks. Great. Thank you, Yvan. We might have to have words about your new metric. We'll see. Okay. Next up, we have Heather and Jason, who are co-founders of Impact Story, and they are going to talk to us about the design and implementation considerations for an open-autometric UI and API, which sounds pretty exciting. Let me just find the slides. Do we need to make it full screen? Can I make it full screen? Yeah, I don't think I've done that right. Sorry. I see. Let me make it full screen. It is exciting. You're right. Okay. So, going to give you a sneak peek of a thing called Paper Buzz. And it's brought to you by Impact Story and PKP. I think Juan's in the audience. Wave one. There we go. Right there. So, he can answer questions later as well. So, all of us, I think, are the vast majority of us use data from Altmetric, and it's fantastic, and it's really good. It's fantastic for lots of reasons. It's comprehensive data. It's easy to use. It's free for APIs to call. It just makes me happy. It's so easy. It's free for research use, which a lot of us really value and makes it really different from lots of APIs in the scholarly space before now. And on its detail page, on the page you get to from the book market, many of those details are free for everybody. There's a lot there that's really great. But it's a company that has, as a business model, keeping their code closed-source and closed-data. And it also costs money for journals and startups and others to build things on top of this. So, because of that, Crossref has decided that they will build something with open data and make that available openly for people to use, open-source, open-data and free for everyone. And Joe Woss is in the audience right here, and he's going to be talking about that late on Thursday in more detail, the Crossref Event Data API on Thursday in one of the last sessions. So, stay tuned for that. So, unfortunately, because they're building a system which philosophically, they're not making judgments. They're not rolling things up. They're keeping it at a low level of abstraction, which is really useful for the ground infrastructure for something. But it means it's not that easy to use, actually, for lots of people, for lots of purposes in some ways. So, there's real value in having something that's a higher level of abstraction there. So, and one of the reasons there's value in building this higher level of abstraction is because open is so important, we need adoption here for this, for all the transparency issues that we've been talking about. So, early on, they were talking about the Almetrics providers. Sorry, just a second. So, all the Almetrics providers right now are closed source, the main four that are used, PlumX, Almetric, and even ImpactStory, because we're mostly getting Almetric data right now. It's closed source, and the methods we use for getting data these days are so complicated, you can't summarize them in brief sentences about how to call Twitter data. You really need the source to be able to document how it's actually done, which requires open source. So, the CrossFit API is all open source. You can see what's actually there. The stuff we're building on top is open source. It's the only way we're going to get to transparency. So, it's a real game changer for Almetrics as a platform and really important. The CrossFit Event Data API has 35 million events so far, though 30 million of them are Wikipedia. That's one of the things that makes it difficult to use. They're putting out a lot of low-level events, specifically right now, a lot of low-level events about Wikipedia. It makes it hard to get to some of the stuff that a lot of people want about Twitter and so on. And again, they're doing this for philosophical reasons, but it does make it harder to use. So, it's a PKKP, an impact story. We're building a higher level of abstraction on top of this, which we're really excited to show you. So, it's open source, open data, free for everyone, built on the CrossFit Event Data. The audience includes everyone. So, PKKP is interested because they want to include this data in OJS for philosophical reasons. Again, to embrace the open. So, it's going to be great for journals, researchers, tool builders, and then others that use tools built on that. The API simplifies events, it includes counts, it includes summary statistics by day, adds that sort of value on top of the CrossFit API. So, if you haven't seen the CrossFit Event Data API, here is a snapshot of what it looks like. There's a lot of detail here, as you can tell. Again, Joe will go into it in more detail. Here's what we're proposing instead. Excuse me, here's what we're proposing in addition. So, this, for people who don't want that level of detail, we just, we deduplicate events, summarize them, include more detail there, Jason will talk about it in a few more seconds, in a little bit more time. We also include metadata there. So, the journal title, the journal authors, things that you might want when you get the DOI. You could go into CrossFit and get that in a separate call, but lots of times people want that at the same time. So, we're again enriching that data to make it an easy to use API. We're also building a website, a bit like the details page you get to when you click on the altmetric bookmarklet, that's free for everyone, includes all the events, and is indexed by DOI. So, one of those pages exists for every paper. As a bonus, it includes a new source, which we're really excited about. So, Unpaywall Views. Let me tell you a little bit about Unpaywall real fast. It is a free browser extension. If you don't have it, you can go get it right now at Unpaywall.org. It's a browser extension that pops up this little tab on the corner of your paper when you're on a publisher site or somewhere with the DOI. And that tab is gray when it is closed access, but green when we have been able to find an open access copy of it anywhere on the web. So, we look in institutional repositories in PubMed Central and Archive and so on, turn that green and you can click on it and go straight to an open access copy. More than 100,000 people have installed it so far in the six months it's been available with more than 4.6 million uses. So, it's a really rich data set, growing fast, really exciting. The people come from 185 countries and we're able to see about 30% of them are from university IP addresses. We're releasing that view data with privacy issues taken into account. So, no IP addresses obviously, but we're releasing the country a rounded time stamp so it's not very identifiable and whether it's a college location. So, this information is also in our new API with one more bonus feature in that website about what's buzzing this week. So, we're using the CrossRoof event data and this Unpaywall data to see what are people talking about in reading this week. And on that note, Jason's going to give a demo. Thank you, Heather. So, I actually want to show this in Google Chrome. It's not super well tested. This is a very early beta. We wanted to show you guys because we're just really excited about it and really interested in hearing your comments because it is still in very active development. We're going to launch this officially at the end of this year. So, as Heather said, we think one of the really exciting strengths about paper buzz is that it's all open data and it's open code behind it. So, we've been talking a lot yesterday at the kind of other workshop, the all metrics workshop and also some today about the provenance chain behind the data. How do we know exactly where every single one of these events has come from? And so far that's been really difficult to do because we have this open platform now it's not so difficult to do. We can also, not only can we trace the provenance but for practical purposes, we can use this data anywhere we want, any way we want. We think that's a really powerful, and a lot of people think that's a really powerful attribute of open data. So, lots of people can build tools in this and we thought, well, we'll build a tool line. So, we built this thing called paper buzz hot buzzing this week. So, I'll just show you this real quick. You can go there, it's live on the web and you can play around with it a little bit. But we said, what are the things that people are talking about most online using CrossFit.net data and also using Unpaywall data so we're able to see what people are reading, who have this, the 100,000 people who have this extension installed, we're also able to see what people are talking about. So, we combine these in, get a little algorithm, again, it's all an open source code so you can track exactly how that algorithm is formed. And so, we can say, okay, what are the top ones? There's this thing about case for colonialism, there's another thing about being type 2 diabetes, that's pretty fun. Because we have open access data from the OADOI API, we can click through to read some of these, some of them we can't, but we're able to see which ones we're able to read. And then we can also sort of slice and dice the data, because again, we have these rich data sources behind it. So, we can pick a certain topic and say, well, I just want to read what are the top, you know, environment stories. And so then I can kind of see, these ones are obviously about the environment. Another thing that's fun is because we do have data about the audience because of the IP addresses that we have from Unpaywall, we can use that to help us figure out, okay, are mostly scholars reading this or are mostly people who aren't scholars reading this? Where scholars are sort of proxied by, are you coming from university IP? Obviously, it's not perfect, but by using this, we can, you can sort of see, I'll just use, I'll do it on all topics. If you look and see what are the mostly non-academic articles, the first one is one about tattoos, which you maybe understand, it's going to be a little more interesting to the wider population, maybe just as specifically, that set of scientists who are studying tattoos, probably pretty small, where it's quite a lot of the general public. There's a lot of them, we found it was interesting, a lot about people's diseases, right? Because a lot of times if someone has this disease, they're staying very up on the research for that. So, there's one story about this therapy, there's a couple more about, right, so type 1 diabetes. So, people who have type 1 diabetes, of course, are very interested in following this research, perhaps even more than the researchers who are doing the science about it. And then of course, because open access is really important to us, we kind of filter by open access, and then as you go through and read, everything is something that a person, even without university library access, is able to read. So, you can go through, as you see, there's always like blue read buttons, means that you're able to just go ahead and read that, open access, either open access on the publisher site, or like I said with the OADOI database, which has 90 million articles in it, we can look at and see, hey, is it somewhere else free to read online, so we can let people click through there. So, I'll just click on one of the score details, and this is the part that I think is a little more relevant for a lot of the journal users, is we can actually get this breakdown of the altmetrics that we're getting from CrossFit Event Data by source, as well as the Unpaywall data. So, as you can see, there's 138 event, 131 events for this, I just kind of picked this one at random, so quite a lot of tweets, and you can filter and just see the Unpaywall view, so you can see where they're from, there's one from Romania, that's pretty cool. Maybe they were at our recent last conference, and then we got a red link there, and of course we have the other types of links as well, like Wikipedia and stuff like that. Eventually, people will be able to subscribe to get alerts for this, but quite prominently places the view on API, because again, we feel like the open data behind this is a real big part of the advantage, so you can look through here, and without any trouble, you can build your own apps on top of our data. The Hotness API is also free for anybody to use. So, in a lot of ways, this stuff that I'm showing you here is not actually as big a deal as what Heather was talking about, because anyone can build this, right? Like, we built it, it was fun, it took us maybe a week or something like that. Anyone, a lot of people here in the room have the ability to build an app like this on this data, and that's what we think is really powerful, and I've been interested in all metrics for a long time, and this is definitely one of the things that I'm most excited about, because now we have an open data set for altmetrics that anyone can build stuff on top of, and anybody can do research. Certainly one of the weaknesses, which needs to be acknowledged, is there is not nearly as much data, and the quality of data is not nearly at the level that you would get from altmetra.com or from Plum right now. They've got a lot more years behind it, and they've been collecting it for a lot more years. So, we even have a little warning on this, it's pre-2017, where the data is not as complete. So, we're looking forward to that improving, that's definitely something that I would kind of consider as a weakness of what we're doing, but again, I would really consider the strength being this incredible source of open data. We're super excited about it, really happy to talk to any of y'all at the break, and thanks for your attention. I'll just go to that. Thanks very much, Jason and Heather. I think that was an amazing amount of information in 10 minutes, so I'm sure there will be lots of good questions at the end for you. So, next up we have Daniela Lohenberg from the California Digital Library. Daniela is a research data specialist and product manager, and she is going to talk about making data count. Hi, so today I'm representing Make Data Count, which is a project that should develop data level metrics and elevate data to a first class research output. So, we're representing our team, which is a Sloan-funded grant collaboration between Data Site, Data One, and the California Digital Library, and you may recognize some altmetric faces included on this project. So, before I go into what we're working on right now, I wanted to give a little bit of a history of how this project has come since the beginning in 2014. NSF gave a grant to PLOS, the California Digital Library, and Data One, to do a pilot into data level metrics, and this was really done by John Kratz and Carly Strasser, where they did a survey out to the research community, both looking at what would it take to develop data level metrics and what do researchers value in data level metrics. And what we found from what will we need to do to actually make data level metrics is that data are much different than articles, and it's a much more complex process. Data have multiple granules, they're more complex than a PDF, they can be versioned, and data can be derived from other data. So, there was a lot more components to look into when trying to develop data level metrics, even learning from when we built article level metrics. And what they found from the research community is that people valued citations, downloads, and views, similar to articles, and also touched on social media mentions also being important to them. So in 2016, Moore funded a meeting grant for California Digital Library, Data Site, and Data One to come together and look at this survey and what steps would need to be taken to take this project to the next level of actually developing data level metrics. And what was acknowledged is that journal articles already have an infrastructure for this, but data really need an infrastructure to make, to elevate them to a first-class scholarly output and give credit to researchers for data like they do for their articles. Even though data play a larger role in the recent process, there just hasn't been an infrastructure for this so far. Which takes us to now. In 2017, Data Site, Data One, and California Digital Library were funded by Sloan for a two-year project that we're now working on to develop these data level metrics. So what are we working on? The first is a formal recommendation for measuring data usage. We're working with Counter because we know that we need a standardized way of actually counting data metrics. So right now we've come up with a draft, and that's on a Google Doc that we're hoping to get community input and feedback on. And then we'll be incorporating that, putting it up as a preprint and iterating off of the feedback that we get from the community. But we'll also be starting to build out the rest of this from this first initial draft of the counter recommendations we've come up with. Working from that, we're going to develop a hub for these data level metrics similar to article level metrics using Legado. We'll be using a similar framework for DLMs. We want to make usage tracking easier. We want to drive adoption and show how it can be done easily by doing mass adoption across repositories and publishers. And we want to engage researchers across all communities and iterate as we learn from this project within the next two years. What will this look like? Well, it'll mostly look in the back end. What we will be doing is taking with the hub. We'll be taking citation usage information in, and then repositories can be feeding information and log crunching similar to article level metrics that we then can display out on any repository that wants to use our consistent way of doing data metrics. This will be all open source. Everything will be documented on GitHub. And anyone who wants to be involved in working on this from the repositories, we'd love to get earlier adopters. We'll be beginning with the data one and the California Digital Library repositories first to show how log crunching can be done. But we're looking also for other early adopters interested in getting a start on displaying this. And by the end, we hope to have a standardized widget that also could be on repositories showing citations, downloads, views, and mentions of social media for every dataset, just like articles. So when we entered this space, though, we knew that we couldn't be the only people working on data metrics. So we wanted to do a survey to see what else is going on in this space. So we talked with anyone who got in touch with us when we released a survey about what we're working on. A lot of publishers, societies, journals, repositories, maybe a lot of you that are in this room are watching the live stream. And what we learned when talking with them is that we need a better system to monitor and track usage and impact metrics for data. Right now, there's not a standardized way. People want to know what's going on with their data, and it led to more community input into what we can do to make data count and data level metrics really useful. So ways to get involved and where we're headed with this. As I mentioned earlier, the countercutter practice for research data is really where we're looking for community input. I know a lot of you in the room now could give really valuable feedback on this, and we're hoping that as many of you can contribute as possible. We also have a link that I can send out for the survey. We're looking for people who want to be early adopters in this. We have a list of them now that we're trying to put together, but the more the better, the more community input we can get into this. And also help with community outreach. Once we do have this up and running for data level metrics, both researcher, publisher, all metric, all views, to make sure that people are citing their data and talking about data like they do of articles. And also on our website, we have a roadmap for where we'll be headed and where you can meet with us. We really need the community to be a part of this, to make it a success, and we're really excited about this. So please get in touch with us, follow us, and find me at the break to talk more about it. Thank you. Thanks very much, Danielle. I have some really nice suggestions at the end there, I think, for how people can get involved. Next up, we have Polly Allen. Polly is Director of Product Management at Plum Analytics, and she is going to talk about all metrics in societal input. Hi, everyone. How's everyone doing? Okay, I'll stretch it out here. So my name is Polly Allen, and I was formerly Director of Product Management for Plum Analytics. We were acquired by Elsevier in February 2017, so I'm now Director of Product Management for Research Metrics at Elsevier. And we're talking today about altmetrics and societal impact. So thinking back to the early days of the altmetrics conference and how the movement was born, how one of the bright and shining hopes for altmetrics has always been that it could provide some way to measure social impact, or at least proxies for social impact, some way to know in an area where we have not a lot of data, can we drive better data-based decisions in the whole area as a whole? And we've made a lot of really great strides in the area so far, and there's, of course, so much more to do. It's exciting to see that we're moving from an era of looking at just looking for the signals in the online exhaust that exists out there and things like social media and mass media, and looking now more in specific places where people are saying, can we measure social impact in a particular field and find out where my research is being read and referenced in areas like policy documents and clinical citations. And it's some really exciting developments that I think provide a lot more usefulness, while at the same time we're improving the robustness of the underlying data, the different ways it can be gathered, gathered and counted. And I wanted to go through an example today of societal impact. What are we talking about? What does that, what do we mean by that? And how are we measuring it compared to how it goes on in the real world? So let's consider the example of clinical guidelines and clinical citations. So for a piece of research to gain clinical impact, what has to happen? So, of course, it has to be published, and then it has to be read, consumed in some way by someone, and largely, you know, those activities are moving online to a greater degree than they were in the past. The dissemination of that research is still largely at conferences and people, you know, making sure other people in their field are aware of it. Again, more on person, but there are ways to detect signals for this online with platforms like SlideShare, et cetera. So once things are begun to be disseminated, there's the amplification of those messages, and that's a really interesting place to see where people are choosing to amplify their message. Does this differ, you know, from different fields? And does the newsworthiness of an article really indicate this is an, you know, amplification-worthy message, or is this something that just happens to be fuzzy and something of interest to the lowest common denominator? These signals are harder and harder to tease out, but we started to, you know, look at the work of what do these signals mean. And the goal, ultimately, for a lot of researchers is for their research to become part of the consideration set for the people who make decisions around clinical reviews and clinical guidelines to say, you know, please read this work, become aware of it, and then actually reference it in your work so that I can know and include in my funding application that my work actually had, you know, an impact when it finally got to the stage of being included in a clinical review, a guideline, a bedside assistance tracker, or part of physician education even, and made its way to the clinic ultimately. So what we do today is that we're able to put little, it's hard to see on the screen, there's little gauges on a bunch of different spots, so we take our little gauge thermometers and we can stick them in at various points in the pipeline to get these proxies of impact because actually measuring impact quantitatively is very hard, and that's what people are asking for in order to make decisions. So we have this ability to gather events and count them, and what I'm excited to see now is even though there's so many other areas, this is one example where we're looking at, this is one pipeline we can stick things into, gather data from APIs and crawling, there's a lot more, there's a whole world of, sometimes it feels like, to discover in terms of what are the other areas, can we start looking at patents, can we start mining legal documents for economists and social scientists, and really try to see where in all of this online chaos we can actually try and make some order and pull some numbers out in a robust way that people agree is counting something useful. But when we talk about getting to the next stage, it really is getting beyond counting and how do we count things. And it's about designing systems that help people make decisions and provide feedback loops the way your speedometer does on a car, so that you can make decisions in more real time, and so that you can assess yourself, do all the things that people are eager to do to improve their own performance in various ways. This, of course, is a double-edged sword. So there's power, great, great power in providing, you know, having systems that use data, especially data at scale, underlying them to make decisions. But there's also great, great danger. And in Catherine O'Neill's book, Weapons of Math Destruction, she describes a lot of really interesting scenarios in place today where systems that have big data underlying them have been designed ostensibly for efficiency and also to make sure that we're reducing the human bias and human error in a system. And she shows how so many of them have actually done the reverse, where they're actually encoding the errors that exist in the system today and reproducing them in a massive scale, creating these rich-get-richer situations. And I think it's important that we make sure as we're designing systems and starting to move forward in that arena that we're making sure we recognize the underlying biases in a lot of the data. When we're looking at Wikipedia metrics, for example, I think it's important to remember that less than 15% of Wikipedia editors are female. What kind of biases are we introducing by counting these metrics and not counting others? I don't think the U.S. News and World Report could have anticipated in 1983 when they first, as a struggling publication, decided that they would take on an ambitious project to rank 1,800 U.S. universities in terms of excellence, their definition, that they would have had such a huge impact on academia as a whole. And so when they chose, originally they started doing the rankings, they were doing qualitative surveys and the complaints arose, this is too biased. Who are you phoning? Why are you phoning those people? This is just opinion-based and it's actually affecting our bottom lines of universities. You need to get more data, you need to get more quantitative. And it's another example where how do you measure excellence? How do you put that number on that and rank people based on that? So from a system where people might have heard from word of mouth that such and such a school has the following positives, negatives, find a good fit for them, we moved more to a system where because of the attributes they chose to use when they moved to a quantitative system in 1988, it was very interesting to see that they chose, they wanted their ranking for people to believe these are the schools that are excellent. When people think of excellence, what schools do they think of? They think of Harvard, they think of Stanford, and what attributes do they share? They share very low acceptance rates, they share alumni go on to prestigious jobs with high pay, what they didn't include or rank highly in those early days were things like admission fees or the percentage of minorities that are going to the school. And what we saw between 1985 and today is that tuition rates have risen 500%, four times the rate of inflation, partially driven by schools need to improve their rankings in a system where it was designed for winners continue to win and losers unless they get a huge injection of cash, unless they start making sure they have more applicants and people then that they can take, how do we drive that? A lot of them did this by raising money to provide great sports teams and get a lot of attention in the media. So I think it's a cautionary tale about how the metrics we use today are being used and how especially when you have a system that's designed using proxies, they're more open than ever for gamification and we have to be wary, but also think ahead so it's not a reactionary thing afterwards and we've designed the system and it's in motion, but really thinking through what are the implications of the underlying biases in a lot of the systems we're designing. So it's not all doom and gloom. I want to reassure you I really am excited about the area and I'm incredibly optimistic. I think a lot of the problems we see in underlying big data systems that have these problems today, they're born of laziness, they're born of the human tendency to use what's available and if they have a quick number they can grab to use for whatever, even if it's not fit for purpose, there's a danger that people will use it lazily and not in appropriate ways. So it's an ongoing effort for education, working closely with the people who are making decisions based on our metrics and making sure that we're conscious going forward, not just of what do we use to measure social impact, but what is the social impact of using those measures. Thanks very much. APPLAUSE Thanks very much, Polly. Some really pertinent points in there I think that we all need to consider as we go through the next couple of days. Next up we have Jean Lu. Jean is Product Development Manager Altmetric and she is going to talk about Altmetrics for Books. Hello everybody. My name is Jean and I lead all new product development at Altmetric. So we've been talking a lot about metrics this morning and I think what we should also remember is that the second aspect is the narrative. So the mentions, for example, of research outputs make up those metrics that we are talking about. So I'm going to tell you a story about Altmetrics for Books and if you were in the workshop yesterday, that was one of the types of things that might be perhaps coming up in next year's workshop. How do we start to talk about Altmetrics data for more than just journal articles and what does that even look like right now? Well, bear with me. So in The Donuts Quest, a book that I personally authored, is a thrilling tale of Altmetrics for Books that begins with a little donut. Once upon a time there was a little sentient donut who wondered what people were saying about scholarly books online. The little donut embarked on an epic quest across the scholarly landscape hoping to map the online locations of all the world's books. He crossed the Taylor and Francis Mountains and ambled through the Springer Nature woodlands. He traversed the Elsevier Rapids, canoeed around the venerable Wiley Islands and surfed on the bustling Amazon River. So I'm going to pause here for a minute because these are all obviously publisher names. So when we start looking for Altmetrics for Books, one of the things we need to do is know where to look. So we need to know what kinds of records we actually want to track. That is actually a surprisingly difficult task because every single publisher will have lots and lots of different domains that they host book records. So when you think about a book, do you think about the physical version of the book? Do you think about the e-book? Do you think of the e-commerce page? An institutional repository page? Those are all valid places that books live. And to actually find them and actually make sure we're tracking them properly is actually quite difficult. And for each publisher we've ever worked with, some of them have at least two to three domains, if not up to eight. And we have to make sure that our tracking software can actually recognize every single one of those web page formats so we can recognize that that's a book and not just a random page on their website. So let's keep that in mind. He even followed the mysterious nomadic books who somehow managed to live in many places all at the same time. So this is exactly what I mean when I say they live in different places but they are actually the same book because the concept of the book is what we're tracking. We're not tracking necessarily the physical copy. We could be, but it's not online. We want to, for example, look at the e-book version or just the e-commerce landing page. But all of those represent the book. And the author themselves may not use one particular version to refer to their book when sharing the link. So we need to be able to get that book wherever it lives and you might have heard recently that AllMetric.com started tracking Amazon. That was actually quite an undertaking because it turns out there are tons of mentions of Amazon and when we're working with data providers such as Twitter that gets expensive very fast to actually track all that information parse it down to the exact books that you want because we're not necessarily wanting to go and track Harry Potter. We want academic books. So that separation is quite difficult. And these books, these nomadic books, they live all over the place so we need to identify where they are. But we have a little secret weapon you see. So the little doughnuts searched tirelessly for the magical ISBNs. Special numbers that identified each unique book. So ISBNs are really interesting because they come in two forms. A 10-digit form and a 13-digit form and you can convert between the two. However, 10 digits is also a phone number. So recognizing an ISBN from a phone number or any other random string of numbers is quite difficult as well. You need to know the context. You need to be able to find the ISBN out of a sea of numbers from a policy document, for example. You need to know that it is an ISBN in that case. Also, sometimes, you had to try and avoid his foe, the bad metadata snowman. The bad metadata snowman like to mess with the little doughnuts metadata reading goggles, making it hard to track books and chapters. So Ewan already talked about this in his presentation, that publisher metadata seems to be getting worse. That's definitely true of journal articles. For whatever reason, some publisher metadata seems to be disappearing. With books, oftentimes, meta tags are not even available at all. And so when you don't have meta tags to rely on, what you have to do instead is scrape numbers like ISBNs and titles and authors and everything out of the page directly. Which sounds like perfectly valid. It is fine. You get that information anyway. You can build a book record. But as soon as that web page layout changes, that breaks the scraper. And you can imagine in a recently small team that we still have it all metric, we have fewer developers and we have non-technical staff. It's pretty unsustainable to go and fix every single page that breaks to try and recapture that data. So what we really want is for book publishers to add meta tags to all of the different places that their books live. And one of those meta tags has to include the ISBN. So this is where metadata can really mess with things because as soon as we don't know what the book is, it's hard to match the attention to a specific record. But the little donut remained determined, knowing that ISBNs were the keys to identifying the books, especially the ones that lived in many places. So like I said, if you put a book record on a repository and you put an ISBN there, that's great. As long as we see the ISBN somewhere else, we might be able to say, oh, that is the same book. So we're building that book concept. We know that all these different records all over the web refer to one book. ISBNs are a little bit different than DOIs. They don't resolve to a specific URL. So we don't have a canonical place to say, this is the one place all the books actually came from. So that makes our lives difficult. Some books do get DOIs, but that is not certainly the universal thing. And ISBNs, you can have multiple for a specific book because different ISBNs refer to different versions of a book. So as you see already, this is very complicated compared to tracking journal articles, which has a very established infrastructure, a very established way of putting meta tags on a page, assigning DOIs, registering them, and having them resolve to real URLs. So it's quite a complicated mess that you end up in when you're trying to juggle different ISBNs and find them correctly. But in this story, we're not going to focus too much on the challenges, you see. It is a nice story. So finally, with a map of online book locations, the little donut returned home and logged into the fantastical Altmetrix database. So obviously, if you've ever followed altmetrix.com, you know that actually we don't go looking for the output records first. The signal to start looking actually comes from our attention data. So we see a tweet come in, and it links to a domain that we recognize. If we recognize that's a book publisher domain, we go and get that book. But we need to know that domain first. We need to have that map of all the different domains where content we care about lives in order to make that connection. And once we do, we have a huge database of great stuff. But you see how very fragile it can be if the books don't have the proper metadata. So suddenly, he was flung into a rainbow of online attention, flying past over one million mentions from mainstream news, policy, social media, syllabi, and more. And so, yeah, already in the short time we've been tracking altmetrix for books. I'd say about two years now. We officially launched altmetrix for books last year. But for about two years now, we've been collecting data. We've at least been following a lot of the domains that we know host book content. And it's very astonishing, I think, because we didn't even start by tracking attention sources that were specific to books. These are all sources that we know mentioned journal articles. And even within those, they were already, as you can see, over a million mentions of these different books and occasionally chapters, too. It was quite astonishing. And, well, the donut agrees. He says, this is wonderful stuff, exclaimed the little donut. I must tell the authors. Because one of the things that I think strikes me when we were talking about altmetrix for books is that there is a really strong need in the books area compared to journal articles to actually have some kind of meaningful data. There isn't the same saturation of citation metrics that you see with books and chapters. If anything, the researchers are spending months, years, on this content. And then they get really no feedback afterwards. Can you imagine publishing a blog post and not checking Google Analytics? It's kind of an odd feeling to not have any feedback to know that the book's been read, utilized, et cetera. So this is something that we think authors really must see. And so what we're doing, and this is our little donut's mission, too, is to make that visible. So he baked batches of rainbow donuts and set off again, this time with a new mission, to show hard-working authors all the online attention paid to their scholarly books. Thank you very much. APPLAUSE Thank you, Jean, for that very different presentation. We're a little bit tight on time, but are there any questions for any of the panelists? Yeah, would you mind coming to the mic if there isn't one? Can you hear me? Are we good? Just to make things more complicated and to add, especially to Jean's task, there are a few fields where people publish anthologies, and so books by themselves are not enough. We need to be able to track the chapter in the same way that we track a journal article. Can you talk to that? Yes, so... OK, so chapters are actually... Whenever they do get their own webpage, very often they are assigned a DOI. So usually chapters are listed in a table of contents on a book website, but for some publishers, so Taylor and Francis was one publisher we piloted, Altmetric for Books with, and they had assigned DOIs to all of their chapters. So in that case, it was possible for us to get chapter-level attention data, and what we opted to do was roll that into a big Altmetric details page where you could actually toggle through the table of contents. I won't speak too much to the way we scored it, but we did actually make sure we... When we gave an Altmetric attention score to the book, we actually just pooled all the mentions for the chapters and then didn't sum it up for... So because that would be double counting, and you could just, for example, mention loads of different chapters to game your score. So we actually didn't let people do that, we just pooled it and then counted it as if it was one entity to calculate the score that way. So it was not as biased, but for some publishers that have the DOIs, it is definitely possible, but that's where that publisher awareness of adding DOIs to their different bits of content, that really needs to start happening. Because the technology is there, it's almost like we're waiting for the publishers to kind of pick up the pace and upgrade their systems. I wanted to mention on the Plum Analytics side, we really love the roll-up you guys are doing on the Altmetric side, trying to, you know, there's still open questions for a lot of the metrics of if someone is just tweeting about a chapter and you roll those up, that's not double counting to count those between chapters. So again, the devil's in the details in a lot of these metrics of that roll-up task and how do you properly attribute things to the right places. Again, I do think it takes work from the publishers to get on board to make sure that the metadata is there and usable too. Thanks, next one. Thank you for your presentations. My question involves, so we've heard about three or more platforms and systems and all these types of things. I wonder about the relationships between these platforms. Are we partners? Are we adversaries? Are we working together? And then the relationship, for me, for instance, as a researcher, how should I choose between one or another to do my research on? And for instance, as a university, how do I make a decision of which one to buy? Do I need to buy all of them? I mean, I'd like to see some answers to. Who would like to speak first? Jason? I think coming from sort of the bibliometrics world, I think that's a great opportunity for research as a researcher. So quite a lot of bibliometrics research as new bibliographic databases have come online have been in looking at the properties of these various databases against one another. So originally there was this Web of Science, and then Scopus, and then Google Scholar, and then additional ones as well. And there's quite a lot of literature now from the bibliometrics community saying, well, what are the strengths and weaknesses of these different databases for different kinds of research? And I think that's a terrific job for the research community to accept that me and Heather are occasional publishers. That's something that I think we could be publishing on. I think the different services can be publishing on it. But I think it's a great research question and one that is probably never going to be done, because especially in the early days of this particular world, there's a lot of changes going on really quickly. And I think that's one of the... I always try and contrast this between the early days of bibliometrics, where it started quite similar to all metrics, in that this was a set of metrics that a lot of people had a lot of concerns about. We, when Eugene Garfield proposed this idea of mining citations to try and track the progress of ideas through the literature, he was met with a tremendous amount of skepticism. Couldn't get anybody to listen to his ideas for years. And then when he finally did, the research world was, in many cases, extremely antagonistic this idea. Like, how dare you reduce what we're doing to a number? And that antagonism has certainly not gone away entirely, but in many ways the research community has reached sort of a piece with the idea of citation metrics. That took decades. And I think in many ways we in the all-metric community are way ahead of that because we can at least look at that example and say, okay, well, here are some of the questions people have. Here's the kind of research needs to be done. One thing I'm really excited about is there are multiple organizations who are doing this work, so at least we can start doing that compared to contrast. Again, Web of Science is the only game in town for quite a long time. So I think trying to draw lessons from that, my two-part answer is trying to draw lessons from early progress of the bibliometrics world and then, like I said, relying on the research community to look at the pros and cons. As for the commercial benefits of one another system, I think that's probably best left to others. If we could keep comment short, but would anyone like to add anything? I'll think you stabbed it. I mean, I mentioned before about the awareness and the training and stuff being equally important to the technical development. So first of all, I'd say, is early enough, certainly, for all of us to benefit. Frankly, the more people involved in the metrics, the more people learn about it, and the more scope there is for when it comes to the commercial side, where it would be incorrect to say we're definitely competing with plum commercially. That doesn't mean we would never work together or we hate each other or anything like that. It's not that kind of relationship. But in general, on the non-commercial side, I'd say it's much friendlier, certainly than other places, other industries. I think the main point, like I said earlier, it feels like there's this open world that is ready for exploration that we've only started to explore. And I was really excited to see Reddit highlighted in the keynote speeches like this is an area that we've been gathering metrics for ages and there's very little research on it yet. So these platforms are available for research. We're very happy to work with researchers who approach us on our side. I know Altmetric has their API available publicly. And I think it's going to take collaboration. It's going to take open data systems. It's going to take commercial partners as well to get where we need to go in the field. Okay, I think for the sake of keeping me on time, do we have one more question at all? Yeah, in the back. We'll try and get some more mics. Hi, my question is in the, it's in the debate on societal impacts and I think it's mostly directed at Polyengine. And it's about country level metrics. Have you tried and attempted to grasp the issue of country level metrics? And if yes, could you please talk about some initiatives or some things that you've done in this work? Regional, like Regional 2 a country or like? Yeah, like any research that has been produced. Like if you try to aggregate the work of researchers of a country and try to draw some comparisons, for example. So I, we don't have specific research that we're doing yet, but what we're trying to do is surface some of the country level information we do store in our database. So for example, policy sources, news sources, Facebook, I think in Twitter are the four that we currently have available in Altmetric and we actually see the country level data. What we want to do a bit further. So we started this project early in the summer to actually make maps so that you could look at, for any given search query you do in Altmetric, you'd be able to see which countries had mentioned you the most in those sources. And it's quite interesting too, because it also reveals to us the coverage gaps. So when we look at the whole database, for example, and then we look at the maps of news, we can see there are some parts that we're not covering very well, some parts that are very well, I mean, I'm Canadian, so we've added a lot of Canadian news sources. Let's just say that. But, you know, so there are definitely some regions of the world that we're not able to cover yet, but we can visualize it. So the first step for us is to visualize the next step is to delve deeper, but all that data is available in our API. So we would want to surface that and get some really interesting research coming out about that. So we're actually going to talk about this this afternoon actually and see a little bit about these maps and some. Okay. Now that we're part of Elsevier, Elsevier produces SciVal. So that's been used in the bibliometric field for a long time. And we're definitely, you know, speaking with them long term about how we can apply the same kinds of analyses to Altmetric data, where it's appropriate and where it makes sense. And then the second point is that what's really interesting to me at the country level and the region level is that it becomes important to track completely different sources when you're looking at different regions of the world. And specifically China, it came up in the workshop yesterday where a lot of these social media sources are blocked. What can you use again for proxy? So we've definitely been working with partners worldwide to try to find usage data, citation data from regions that typically are underserved and don't have metrics in this area at all. It's difficult actually because you need to aggregate the data from the article level or the research level. And so you need a good coverage for that. Of course. Okay. Okay. Thank you very much. A big thank you to all of our panelists. I'm sure if anyone did have further points that they'd like to pick up, everyone will be around during the lunch break. So please do go up and have a chat with them. Okay. I'm now going to hand over to our next chair for the Altmetrics and Research Evaluation Session. Okay. Hello, everybody. Hello, everybody. My name is Dono Kubo and I am pinch-hitting for Katie Hinckling. She ran into some travel issues. So, you know, as one of the organizing members, I know that she really wished she would have been here. You know, I know she's done a lot of work in this space, but so instead you have me for a little bit. So, this session is going to be on altmetrics and research evaluation. And so, I'd like to introduce the first speaker and this is Kate Williams from the University of Cambridge, who will lead us through an understanding of emerging cultures of evaluation. Hello, everyone. My name is Kate. I'm a research fellow at the University of Cambridge and also at Harvard. And I have an ESRC funded grant on altmetric indicators of research impact in development research organizations. And this project involves mixed methods, so it includes ethnography, interviews, and quantitative data analysis. But I want to talk to you today about some preliminary development of a theoretical framework which arising from this project. So, it's quite a departure from what we've been talking about today. Although it links to Mike's observations about whether we need a term for altmetritions and to Paulie's question about the social impact of using social impact metrics. So, research is obviously crucial in addressing complex environmental, social and economic challenges. As such, research institutions are eager to develop efficient ways to measure the wider impact in order to inform strategies around research focus, funding, communication and reporting. Yet, the measurement of impact is highly contested and expertise in the area is under theorised. So, informed by a sociological perspective, today I hope to intervene in this setting by reframing impact assessment as a space between fields which shapes cultures of evaluation. And I hope this contribution will provide new directions for the study of impact and altmetrics. So I have five aims. The first, I begin very briefly by situating the recent focus on research impact in its wider contracts. And then I outline three of the core ideas. The first is that impact assessment occurs within a space between fields. The second is that altmetrics are making available new types of capital or resources. And the third, that this framework can be used to understand emerging cultures of evaluation in research organisations. And I conclude by outlining an example from the case study of the World Bank. This work diverges from previous work because of its field theoretic approach which is a distinctly sociological perspective to elaborate this hybrid nature of the space between fields. So recent years have seen a reformulation of science systems towards more applied socially accountable enquiry produced by an increasingly diverse set of research institutions with greater emphasis on the costs, output and quality of research. And recently this focus has extended to attention on wider societal impact. And accordingly, new techniques of impact assessment are emerging. Here amongst these, of course, are altmetrics. The available academic literature on altmetrics largely focuses on the statistical comparison of altmetric data to establish sources, the validation of measurements and the identification of biases. Thus, there's a current need for greater understanding about how these metrics take on meaning in practice. Given the significant recent uptake of altmetric technologies, research is required that considers the specific cultural and institutional effects of these new tools, set in the context of nuanced understanding of cultures of evaluation around research impact. So using a sociological framework, this paper seeks to address this gap in two specific ways. First, although there's a body of sociological research on traditional metrics of research quality or scientific impact, work specifically on altmetrics occurs primarily within the fields of scientific metrics and library or information science. Against this backdrop, however, two recent relevant sociological publications argue that altmetrics are of growing importance in contemporary research systems. And these studies indicate a growing sociological interest in altmetrics and provide potential avenues for theoretically informed research. Second, relatively few studies have focused on the sociological effects of altmetrics rather than the application of altmetric methodologies to questions of sociological interest. Therefore, there's a need for distinct but related work, including ethnographic studies, on the institutional effects of research evaluation that pushes novel insights on impact assessment and altmetrics out to different fields while also contributing new perspectives to information science and research evaluation and policy. So onto my first core idea. In the study of expertise, the production of knowledge has been dealt with as a function of relatively institutionalised conditions. So there's an assumption that areas of expertise will grow into more defined forms as they evolve, whereby they coalesce into formal structures such as academic disciplines and professional organisations and thereby establish control over the production and certification of expertise. So for example, in law. However, having only featured in Britain and Australia since around 2004, the area of impact assessment does not conform to this ideal typical notion of a bounded profession, disciplinary intellectual field of inquiry. And one explanation is that impact assessment is an emergent field which will coalesce into formal structures which perhaps underpins a lot of current altmetric research. The assumption here is that it will mature into an institutionalised field, a discipline with strong, highly regulated boundaries that offer autonomy from other fields and contain a specific logic and fixed capitals or resources. However, an alternative explanation is that impact assessment occurs in a space between fields because it comprises interactions of different arenas, not simply different criteria within a self-contained field. The available resources or capitals that make up this space are drawn from various fields with their own implicit and explicit rules, activities and norms. Thus impact assessment can be conceived as a non-standard field of expertise. And these types of fields are located within a multi-dimensional social space that contains a number of actors, stakeholders and organisation types rather than tied to a particular community. For example, in assessing impact, the borders are blurred between university departments, research centres, publishers, libraries, academic networks, professional societies, the media, government and the private sector. Each group can make a valid claim for expertise in this area. The key feature here is shifting permeable borders that permit people, ideas and techniques to travel between multiple fields. For the range of actors who converge in these spaces, this poorly articulated hybrid space permits freedoms not available within established fields. So actors are able to draw on the expertise, ideas and language from outside their own field while remaining credible and authoritative within their distinct arena. For example, impact assessment requires technical skills, so bibliometrics or metrics or academic expertise, expert reviews, narrative case studies. And to be useful, it must be communicated or implemented in particular ways. So performance management, institutional evaluations, public accountability documents or national evaluations. And thus it requires an ongoing balance of technical requirements, communications skills, orientation to value for money and an orientation to policy, practice or management. Accordingly, this is also true of the demonstration of impact by research organisations. To be impactful, research must contain a definitive academic identity yet that produces a range of products for receptive audiences and positive outcomes for diverse groups, which can be assessed by specialist evaluators through a predefined set of practices. So as shown here on the slide, the measurement of impact, the capitals or resources of several fields are at play. Specifically, this might include the field of knowledge production, the rigor and methodologies of academia, politics, the media and economics. So this idea offers an alternative perspective whereby to assess research impact, evaluators and audiences draw upon various types of capital from different fields and accordingly to demonstrate impact research organisations now must also do the same. So the second idea here is that in addition to these fields that I showed before, we're also an emerging, we're also observing an emerging field of metrics, which is the skills to create and interpret rankings and indicators. In contemporary research systems, metrics are imbued with significant power. In addition to influencing resource generation and allocation and the recruitment, assessment and promotion of staff, they also change the way we do politics, assess value, allocate guests, resources and justify worth. So this emergent field of metrics links various actors to users or clients using the language of numbers. Yet the field of metrics itself is quickly evolving. Scholarship and academic communication is rapidly changing with many of the scholarly processes occurring online, which creates a wealth of data that can be used to inform scholarly and impact metrics. Especially for applied policy and practice focus fields of study, alternative metrics are attractive and are sensibly helping scholars and institutions to find and observe what impact looks like. And as evidenced by the fact that we're all here today, we're seeing large changes in the field of metrics. As Halstein Bowman and Costas put it, what is considerably different, however, is that altmetrics capture events on platforms that are in a constant state of change and whose use and user communities are new, diverse and not entirely understood. Thus, the increasing interest in and importance of altmetrics makes new types of capital available. For example, on the research production side, it's at least theoretically possible for a researcher to gain legitimacy by demonstrating a huge social media following or to bolster credentials by combining citations and altmetrics on a CV or in a performance evaluation. And on the evaluation side, funders, publishers and research institutions can also gain legitimacy by demonstrating value for money and policy orientation by investing heavily in altmetric technologies. So in this context, altmetrics are now a significant factor in the space between fields. Given these recent challenges, recent changes, it's important to consider the specific cultural and institutional effects of new metrics and technologies. This requires an examination of how actors are involved in the process of gaining and maintaining legitimacy from various fields. And highly relevant here is the work of Michel LeMont, who shows that sociological research in particular can facilitate unveiling the evaluation criteria and bringing to light the devices, institutions or cultural and social structures that support or enable them. So the theoretical development that theoretical framework developed here could potentially allow us to examine the intersection of cultures and logics of different fields. So here I move to an example of research production and evaluation in an international development institution, namely the World Bank. The bank was selected because of its dominance in global policy research and because of the tension between its research and policy functions. Over a three-month ethnographic period, I examined the context and structures of impact at the institutional and individual level, examining the acquisition of various capitals through existing and emerging cultures of evaluation. And I'll provide briefly some early insights in order to demonstrate the potential for this theoretical framework for understanding how cultures of evaluation are changing. So at the bank, as elsewhere, the increasing focus on research impact is a structuring force in the construction of valued and legitimate knowledge. The bank is an interesting case because it's engaged in a balancing act where it seeks to dominate the language and resources of metrics, media, academic, political and economic fields. And as a result, the cultures of evaluation reflect these diverse portfolio of capitals held in constant tension with one another. As shown by some examples here, what I've seen in particular is that altmetrics are starting to change contests for capital or resources, but they're not in full force yet. There's great interest and sizable investment from leaders, managers, communications and publishing teams in these new technologies, and researchers are aware that they need to be developing the corresponding new skills. As one researcher stated, in this day and age with this technology, it's no longer acceptable for us just to be speaking to a small community, whether that's just an operational community or just an academic community. There's always a sense that there needs to be some kind of above and beyond. So my tentative impression is that we're seeing changing contests for capital, whereby legitimacy can be gained in new ways, which will require research with a strong theoretical framework to unpack. The implications of the framework I've outlined for the altmetrics community might be around specific types of metrics, for example, that policy and media citations are particularly valued in policy research contexts like at the bank, because they speak specifically to political and media fields, where a great deal of legitimacy is sought and amassed. In addition, the field of metrics is still emerging, so we can expect to see shifts as the techniques and theories develop, which will be reflected in the cultures of evaluation that rely on legitimacy gained from various fields. Thank you. Thank you, Kate, for a very insightful presentation. The next speaker I'd like to introduce is actually a former colleague of mine, Rebecca Kennison, and she is from K&N Consultants, and she'll be discussing Humemetrics HSS Initiative, Rethinking humane indicators of excellence in the humanities and social sciences. Kate, I see your sociology and I raise you philosophy. We're clearly the theoretical framework. I don't know what you're talking about. We are a melon-funded team that is being led by the Dean of Arts and Letters at Michigan State, who is a philosopher, so that's why I say that. You'll see a lot of that in what we're talking about here. Like Kate's piece about theoretical frameworks, we're also proposing a framework, and we're starting with this problem. It's all good to do all these great things, but you're hardly making any impact on social media, and this is the world in which increasingly, particularly administrators, and again, our team is led in part by a Dean, so we're really focused on what we can do to help him. So the goal of the Humemetrics HSS Initiative, which is humane metrics, is to create and support this framework for understanding and evaluating all aspects of the scholarly life well-lived. There. You guess how your work could out for you now? For promoting and nurturing these values in scholarly practice, and for empowering scholars to tell more texture stories about the impact of their work. So we're taking altmetrics and kind of flipping it around, and really taking a look at what values do we want to look at? What do we want to encourage? And then how can we find a way to do that, rather than what can we measure, and now we hope something's going to come out of that? So we're flipping it on its head in a reverse engineer kind of way. So our preliminary values-based framework is this, which is what we've thought a great deal about what we would like to see in the academy, equity and openness and collegiality and quality and community. And we've started to test that in conversation with people. And next week we'll be having our very first workshop where we'll bring together 21 academics from Humanities and Social Science. We're also just concentrating right now in Humanities and Social Science with the argument that if we can get it right there, we can get it right in STEM. So we'll see. So what does this mean in terms of what we're looking at? So equity, we're defining that as the willingness to undertake study with social justice and access to research and the public good in mind. There are a variety of different components of that that you can see here. Fairness, inclusivity, diversity and so on. We're also interested in openness of all kinds. And by that we don't just mean open access and open source, but also transparency and candor and accountability. And again, willingness to change and adapt. These are all kinds of openness. So really thinking about what the quality of openness might be. What does collegiality mean? We're rapidly beginning to understand that without collegiality, none of these other values will actually matter. There's no equity without collegiality. There's no openness without collegiality. There's no collaboration without collegiality and so on. And collegiality, the practice of kindness and generosity and empathy towards yourself and others, really hard in a competitive environment like the academy. What about quality? So quality very important, not just in replication and reproducibility and soundness, but also in intentionality, really thinking about why you do what you do. These are all quality markers. And finally, we're looking at community. So engagement with and leadership of communities of practice and also with the public. And again, variety of elements that fall under that kind of value, including solidarity. Okay. But what does that have to do with altmetrics? So that's all very great, wonderful. We should all strive towards those values. Well, here's where we think that we have shared goals with the altmetric community and with the work that many people are doing and thinking about. Because what we're hoping to do is expose and highlight or recognize and reward this scholarship that goes into all research activities and not just publications in all the diversity of publications, but really looking at things like peer review and mentoring and public scholarship of all kinds in order to be able to reward all of those things. And so again, some things are easier to measure. And in fact, this morning we found out not so easy to measure, right? So not so easy to measure. And now what we're suggesting is we need to expand this beyond that to the work that scholars do entirely. So I'm going to use an example of the syllabus as one of those scholarly activities. Because creating a syllabus is a scholarly activity. Not just, it's not just about teaching, but teaching is also a scholarly activity and we don't want to lose sight of any of those things. So what might we look at in a syllabus? I know that altmetrics is working with the open syllabus project. And one of our team members is Stacy from Altmetrics. So we're very keen at looking in the first instance of what a syllabus might what might we look at. So here's some elements that we think we can pull out in a machine readable kind of way, but we'll find that out as we get into that. The course description and the discipline and titles and authors and hopefully people are using DOIs and ISBNs, but as we just heard, that's a challenge. And class scene, but also taking a look at things like class engagement and student activities and assignments. How much time are you spending within a course on a particular text, for example? That should tell us something. What are you doing within your class about that? That should tell you something about engagement. And tell us something different than when you're citing it in a published work that you're doing. Because what are some of the things that we can learn by looking at how a syllabus is created? Also tells us something about the class, the person, and so on. So what can we discover about circulation of ideas? Is there a difference between what's being cited in a syllabus and what's being cited in a published work? We have theories about that, but we haven't really looked to see. I mean, our theory is that what you're presenting within a syllabus is the best work possible out there. And that's not necessarily the reason why you're citing something within an article or a book. Let's be honest about that. So that's one thing that we want to take a look at. Are there differing ways to take a look at that? What are the forms of scholarship that are included and by whom? What roles do the students play in the scholarly conversation? And as I already mentioned earlier, I'm really interested in engagement indicators and how much time is something is being spent and whether we can learn something about context within that. So how might this then encourage those values that I started out with? Just going to use some examples as you think about syllabi and how they're created. A question like, am I including scholars and works from all backgrounds? That would be equity. So really thinking about that and that would start to encourage the value of equity. Have I made this syllabus open to other instructors? That would be a kind of openness. Do I have a code of conduct that encourages kindness? Is this the best word possible? And how can I encourage engagement? And there are many, many, many questions that I'm sure all of you can think about as well that might fit into things that we could look at that then once we can uncover them and unpack them and really think about how you expose them, that might then encourage people to be more intentional for these kinds of values as they're creating their syllabus. So this is the work that started all that off. We were at the Triangle SCI last October. I cooked this all up. And we've gone on from then to continue to think about this. So we'd love to hear from you. We'd love to engage with you at all times. We're only just beginning, as I said, next week, as our very first workshop to take a look at this. So please follow us and engage with us. We have a blog where we're blogging as much as possible. Please come and comment. We welcome emails. And we really, really want to engage with the community. Thank you. Thank you, Rebecca, for your philosophical side of things. Our next speaker will be Martin Perk from the University of British Columbia. And he'll be telling us about research evaluation in Canada. So I'll say I'm the chemist raising from the sociology and the philosophy, or maybe raising lowering, I'm not sure. But I am from a very different cohort from most of the rest of us. So I'm part of the University Administration at a very large research university, number two in Canada, and have a very different perspective on metrics and particularly altmetrics. So I thought I'd talk a little bit about what's happening around us in Canada, a sense of what we're up against here. So it is all about impact. We worry about that a lot at universities. Our stakeholders demand nowadays more than ever to know, you know, what have we done lately and with all the taxpayers' funding that we use in the process of our research. Somebody said earlier that we're also, even in the competitive process of securing research funding, it's become a lot more important to show impact and potential knowledge mobilization. So it's a competitive aspect as much as anything. But there are obstacles to using altmetrics and metrics in general. And I'm part of that administrative decision-making body that decides what metrics we worry about at UBC and what systems and products that we implement to help us. One of our, I would say, main obstacles is in Canada we do not have a REF or an ERA. So we don't have any source of block funding, which in other jurisdictions forces universities to be very mindful of things like metrics and altmetrics. So we don't have the same economic drivers towards using metrics and systems as other jurisdictions. Now there are some funders. The good news is there are some funders that are looking at specifically metrics. CFI, our Canada Foundation for Innovation, is a big infrastructure funder. And they are quite big in metrics. But many of the other funders, not so much, are health research funders, EHR, Canadian Institutes of Health Research, are looking at things like research fish. But it's very early days for us. And we have not really got a head around how the funder is going to hold us accountable in terms of research metrics. So that's the basic context. Lack of block funding, it's all project based. Everything is project based here. And the funders are not really pushing us too hard yet on the metric side of things. The provincial governments do push us fairly hard. And so we do produce reports on a yearly basis that involve quite a bit of bibliometrics to this point. But I think we're starting to think about altmetrics. This is a look at the funding landscape. Basically, a list of funders in this country. CFI is a research infrastructure, NSERC, Science and Engineering, CHR, Health Research, SHERC, Arts, Social Science Humanities. And we've got various other niche programs and chair programs. So I think we used to be a very well funded jurisdiction. We've lost quite a bit of ground over the last 10 years. We used to be the top funded per capita in university research. I think we're sort of middle of the pack in OECD now. So instead of talking about cool metrics and cool systems, I think what we as administrators look at more is, you know, what's the compelling business case? Why do we need systems and tools? So I'll put my technology poll hat on instead of technology posts. And these are the kind of things that we worry about as university leaders. You know, we need to convince people, our stakeholders of the research ROI, that's still very difficult to do. And I think the tools that help metrics developing and plumb and other groups are really helping us get to that. But it's still early days. Research as an economic development activity is huge for us, especially at the provincial level. I'm sure that's true for most. Collaboration, what are the ideal impacts? How do we build interdisciplinary teams? Who's missing from the collaboration? Who should we be talking with? How do we convince funders that we're the best in the business? And that other group in some other part of the world is not quite as good. Social science humanities is a big aspect for us because UBC is very good at that. But again, because of the lack of indicators, it's difficult to show just how good UBC really is. So that's a big issue for us. And then there's all sorts of other things like growing internationalization is considered to be a good thing. We really do worry about THE times higher ed and ARWU. We pretend we don't, but at the university administration, you know, we worry about that a great deal. And of course, if you look at the KPIs involved in things like THE and ARWU, it's highly quantitative. There's a bit of reputational stuff, but there certainly isn't too much in the way of metrics. And those are the things that drive us. You know, people ask us, why are we going up or down in the various THE and ARWU? The other thing in Canada that really is driving us lately in terms of competitiveness, and I know this is true for many other parts of the world too, is low success rates in agencies. So we're looking at success rates as low as I think the latest CHR competition, project competition. We could be looking at less than 10%. So this really has become a very dire competitive issue being able to outmaneuver and out. Okay, so these are the big questions then. And these are the questions that we need tools to answer. And other uses are really not a compelling business case. Just because something's cool, it tells us something we don't already know. Those aren't the compelling business cases for institutions. So these are kind of some of the things that big questions that we're trying to ask. You know, how do we move up the global ranking scale? In the future, you know, how do we, who do we reward for great research? And how do we define research excellence, especially in the Shirk social science humanity areas? Who do we hire? We've got a new program that was just launched, I would say, very opportunistically, the Canada 150 Chair with Brexit and with Mr. Trump that we're trying to recruit some of the world's top researchers, bring them to Canada. And so that's the launch, the 150 anniversary Canada. And then we all belong to clubs, international clubs. UBC belongs to U21, amongst other. And so we want to know, you know, how do we align with our international university partners? And strategically, what are the best themes and best opportunities to work together? You know, what does this look like? Well, this is a traditional look at the world map for Syval. We do use Syval quite a bit. It's a pretty useful tool for telling us what we're good at, who else is good at that, these different competencies. And I guess one of the things I'll mention is that what attracts us to this particular tool is it does tend to predict new trends and new research areas. And that's of great value to us. So this is the map of UBC and it gives us a sense of who we're good at, who our collaborators are, who our competition is and who we should hire, et cetera, et cetera. Our metric, great tool, we've had a look at it. It does help us answer a couple of big questions. One of those questions is, you know, what is competition up to in strategic areas of interest to us? So for instance, we can do searches on things like Stanford and leukemia and gives us an idea of what the big stories are and what sort of impact our competition has in strategic areas of interest to us. And here's a kind of temporal look at that study. Who's looking at the articles, what sort of reach do the articles have is a big question for us as well. Now, looking at UBC, you know, who are big players, what sort of social media attention do they get? Mark Amara is one of our biggest. He's big in cancer and personalized medicine. So this, so our metrics gives us a sense of what our big guys are up to, how competitive they are with the other big guys and what sort of attention we're getting. I think we're running into lunchtime here, so I'm just going to speed things up a little bit. I think the other thing is when you delve into the metric reports, one of the things that's really important to us too is quality. So how do we define research excellence? We can look at papers and how they compare to other articles in things like nature, articles of a similar age. So there's some really important tools that tell us about quality and research excellence that we think are important alongside tools like SyVal. And again, you know, who's using our research and who should we be thinking about collaborating with are important aspects of our metric. So just a couple of some quick conclusions. I think evaluation of research in Canada is fairly spotty for some of the reasons I mentioned, the block funding and the lack of sort of drive from our funders. The current tools are good, but need to be better. I think we're using them at UBC some to, from a kind of an interest aspect, others because we really need them badly, but there's certainly improvements, more flexibility, better integration, more scope of metrics, better social science and humanities uptake. I think there's aspects of clean cleansing data. The data's not always all that clean. There's big issues of need for comprehensive data sets. What drives research as bonkers is when they look at data sets, when they look at metrics and their work doesn't appear in them. And we need more nuanced impact metrics. Snowball plus there's other sets of research metrics out there, but we need to do more work and develop scorecards of metrics. It really makes sense for all of us. I don't think anyone has any real good ideas on how to get that right. And certainly Times Higher Ed and ARWU does not have the answers. I don't think any of the systems lie in. There's a whole list of different global ranking systems. None of them seem to really provide a very nuanced ranking of how excellent we all are. And just finally before lunch, I would say what we really need to do is kind of transform institutional use of the tools from kind of interesting novel cool to mission critical. And that just hasn't happened. The tables I sit at, we just haven't got to that point at all. So that's a bit of a roundup on the Canadian context. Thank you, Martin. Very insightful. So we have time for questions or question? Anybody? Okay. Hi there. Just quickly, I find that within our institution, I'm still seeing a cultural perspective whereby people want the most basic metrics reported on a sort of a fiscal perspective. A quarterly, just to meet accountability. There's a lot of that there because as you pointed out, a lot of people are dabblers, right? They're almost weekend warriors when it comes to the use of metrics. People in leadership positions or individuals reporting their own data. What would you suggest or what's worked for you, particularly Martin, I think you use a lot of this data more strategically for planning to move those various agents, funders, like internal foundations, medical leadership, that sort of thing, to be able to use it more strategically and more for planning? So I think one of the biggest obstacles we have is integration. At the moment, you know, we've got pockets of data and pockets of metrics all over the place. And so at UBC, we literally have a senior committee that has representatives from all different parts of the university voting on literally on what metrics we're going to utilize. And it's so painful trying to pull it together and actually deliver an annual report, say, of metrics. And so that seems to be the biggest obstacle. And I think the other big obstacle, once you leave the university, is that we all know our researchers have a lot of administrative tasks and administrative burden, right? I saw, I know, FTP and CAGA in the US estimates that researchers spend 42% of their day doing administration. So for us, that's a big issue. And the funder realizes that. So the funder comes to UBC and says, we think you should fill in research fish profiles of all your researchers. And we say, well, we've already, you know, input that data, that data exists in different places, and they want us to duplicate the input. So until we get better integration of data sets, and we can minimize the administrative burden, I don't think anyone's going to sign up for it. I think those are the biggest key issues for us. Okay.
 Can you come and sit here please?  This works great. Yeah, go for it.  Okay, thank you. Kat, I'm just going to do this for you.  Okay, we have SuperWizClaire helping us out with slides and things so when I can't work the technology.  Okay, hope everyone managed to get coffee and is ready for an exciting day of talks.  Welcome to 4am, I'm Kat, I work at Altmetric and I'm one of the organisers of the conference so I hope you have a great time here.  So I'm pleased to introduce you all of the speakers of our Altmetrics Now and Next session.  And what we've really tried to do with the content in this session is to give you a bit of an overview of some of the latest thinking in Altmetrics.  Some of the new ideas and the new technologies that are starting to emerge.  And to really kind of set the scene for the next couple of days of discussion.  We have a slight program change because I'm afraid Katie from PLOS couldn't make it in the end.  But the good news is that that gives everyone else a little bit more time to talk.  So I won't be cutting them off too quickly.  I think we'll have everyone talk and then we'll do questions at the end.  So if you do think of anything please try and hold on to it till then.  And we'll get started.  So first up we have Ewan Addy.  Ewan is the founder of Altmetric.  He was a former researcher and he's going to share with you some of the thinking that we've been doing over the last year  and what we're thinking of moving forward with Altmetrics.  Hi everyone.  So what I was going to talk about is not so much anything to do with Altmetric.com.  If you're interested now you can talk to people here like I'm here and Gina and Kat.  It was more like things we've noticed as Altmetric.com.  So obviously we get a lot of people talking to us about Altmetrics.  And some things we can help with, some things we can't, but it's to give you an idea of the kind of queries we're getting and things we've noticed.  So it's this 10,000 feet view from Altmetric.com.  Is it showing up?  Oh, the first thing I should say is that following Antly's very clever idea, I've also put my slides online because I realized you can't necessarily see the bottom of them.  So they're there bitly Addy 4 a.m.  So without further ado, jumping straight into it, the first thing is that we're seeing academic Twitter still growing.  So this is a graph of tweets that mention articles, not books or data sets or software or anything like that, specifically articles.  And the interesting thing is it's still growing by quite a high percent.  So from 2014 to 15, it was 45% growth, 15 to 16, 32%.  It looks like this year it's going to grow again by between 30 and 40%.  And that by itself without context isn't necessarily the interesting part.  The interesting part of the Twitter itself is only growing 10% a year.  So for whatever reason, we're finding more articles being mentioned online.  Now, why is that? Is it because more scientists are using Twitter?  You know, they're overrepresented in that 10% of new users.  It's because journals are getting better at social media.  I don't know, but we've definitely seen the trend.  Second interesting thing, which I think is a very positive thing, is that people are talking about preprints more.  So this year has really been the year where we saw this kind of data take off.  People have always talked about archive online, but they're always talking about bioarchive.  So this is a very kind of loose analysis.  I'd be interested if anyone else wants to take it further.  But just quickly looking at plus one, I picked plus one just because it's, you know, a big journal has a lot of data.  And looking at how many articles it published in 2016,  and then comparing it to bioarchive, and then seeing how many mentions in total did it get,  for every 100 papers, how many tweets did it get, how many blogs, how many policy citations, this kind of thing.  So the interesting thing is with bioarchive, actually in general, and there could be all sorts of reasons for this,  but it actually gets more blog posts than tweets, a pair of 100 papers, than plus one.  And I could say that's because, you know, they're inviting people to submit preprints to bioarchive,  and you're kind of selecting a certain type of research to go in there, or maybe it's, you know, researchers think,  again, just like with open access at the beginning, it's a particularly controversial topic or something that's quite exciting,  so they deliberately make it open as soon as possible. There's all sorts of reasons.  But regardless, people are finding articles on preprint servers and talking about them.  So the opposite, as you might expect, maybe, is intuitive,  is you get fewer news stories, policy sites, Wikipedia sites from the preprints, from bioarchive, than you do plus one.  So there's still more news stories written about things in published journals than there are in preprint servers,  on bioarchive specifically, but they still exist. So you do still get news stories, and you do still get policy documents.  So there's an example here about, it was a preprint about Zika virus on bioarchive, and it was already picked up by the CDC.  So to be clear, I think this is a great thing that preprints is, you know, kind of taking off.  I don't think that's a controversial view in this room, probably.  What I think is maybe a problem is there's no difference in the way these things are cited.  If you go to Wikipedia page and it cites a preprint, it cites it in exactly the same way as it cites peer-reviewed research.  If you look at a news article, in the vast majority of cases, the way the news article talks about the research is exactly the same  as it would talk about peer-reviewed research, with a few notable exceptions.  So there's Spiegel there was one example I could find where the journalist actually goes, it's research that hasn't been peer-reviewed yet.  And I don't necessarily think it's an old metrics community issue or responsibility to fix,  but I think it's someone's responsibility to at least think about it and think, is this what we want?  You know, we talked, I remember four or five years ago, there was a lot of the crisis in scientific communications,  it was about, you know, people overhyping results and things in the mainstream media.  If that was peer-reviewed research, how much worse is it going to be if, you know, it's accepted, as long as it's on a preprint server,  it's now, you know, part of the scholarly record, I'm going to say whatever we want about it.  It's not all negative, some of these things seem negative, they're not necessarily.  But here is one that's unfortunate for us, especially.  We think publisher metadata get worse.  So Google Scholar, when it first came out, really kind of incentivized a lot of publishers to put meta tags on their page  with good metadata in it, because if they weren't there, they wouldn't get indexed in Google Scholar,  which is important for discovery and basically ties to your business as a publisher.  Nowadays, we don't see that so often.  If anything, we sometimes hear from publishers that they have meta tags,  but it's only activated when it's the Google bot that crawls their pages.  I cannot conceive of why this would be a good idea, but there you have it.  There's a move away from the kind of traditional scholarly platforms,  which for all of their strengths and weaknesses, you know, do have a record of putting good metadata on the pages,  and towards more custom platforms, where often the metadata is an afterthought.  So, you know, people say the metadata is the love letter to the future and things,  so people kind of forget that.  They don't include it in their first pass of their new publishing platform, no meta tags,  no way for us and other metric providers to kind of link a paper with,  or to rather connect an actual research output with a link.  Another thing, books versus articles.  So, I feel now pretty confident that we've got good data on article sharing on social media.  So, you know, if you, there's, you know, it's never 100%, there's always problems.  We talked about this a little bit in the workshop yesterday,  but in general, if you tweet a link to a paper,  I'm pretty confident that at least somebody in the all-media community is going to be picking it up properly.  But books, and I'm not going to talk a lot about this because I know Gene has got something coming up on it,  but books are very different.  And again, it's to do with not just technological issues, but also kind of social ones,  the way people cite books.  So, there are some example tweets there.  You can't really see them because they're down at the bottom.  But it's, for example, people don't link to a book.  If you think about it, you know, where is the canonical place for a book to live?  Is it Amazon?  Is it a publisher site?  You know, is it your institutional repository?  It's not a clear kind of accepted way to do it.  There's other connected problems around books,  not necessarily having the same level of quality metadata as articles.  It's certainly not with good unique identifiers like DOI, so that is changing.  So, it feels like it's a little bit behind.  So, you know, we talk about data sets and software being kind of second-class research out there, citizens.  But actually, from an all-metrics perspective, books are, you know, they're still out there as well.  They're not quite at the same level as articles.  So, moving more on to good things, I always, I think other people working in all-metrics  live experience as well, but I always really like it when people actually know what all-metrics are.  And occasionally, I meet a researcher through friends or whatever, and they'll be like,  oh, yeah, the donut guy, I know you.  And I always make it feel really good.  And most recently, someone said to me, oh, how much I finally know you do because you're in the economist.  That means that all-metrics has really made it.  Okay, maybe, I don't know what that standard is, but there you go, that did happen this year.  And I think it is a sign, amongst many other things, that awareness and update is still growing.  So, I really wish we had hard data here.  And I know it's Michael Habib around.  Like, I know Scopus used to do that, you know, what percentage of people have heard about all-metrics.  And I remember it was 0.5% the very first year.  I wish I knew what that percentage was now if we had the same kind of longitudinal study.  We don't, but we do know that, you know, increasingly, we have more researchers who know about all-metrics.  More publishers, more funders and governments, especially from an all-metric.com perspective.  We have many more publisher, sorry, many more funder and government customers than we have before.  And in general, their approach to all-metrics is also quite sensible.  So, the worry was always, you know, people are going to, you know,  have their tenure practice judged on how many tweets they have and this kind of thing.  And it never really seemed like a real worry.  And actually, it isn't.  Like, we've not worked with anyone who's really come to us and said,  I really want this one number to judge people on.  They're generally not like that.  That's very good.  One thing that we've realized as a company is that training and awareness and kind of pragmatic best practices  as important an activity as the actual development of the all-metrics,  which is easy to say is actually quite hard to put into practice.  But we spend a lot of time.  We have, I think, as a company, we have more people on the non-technical side than we do the development side.  It's still pretty close, but a lot of the work we do now is in training and awareness.  The flip side of this is, and I think maybe Rebecca, I don't know if this is the paper that Rebecca meant,  but things like this, so there was this blog post recently in the LSE Impact blog,  the click-bake and impact one, talking about a paper that had a high-up-metric score.  But the content was about the case for colonialism in a TNF journal.  And the authors say, well, you know, this article got published and had a lot of attention about it.  And the highlighted bit here is, by the rules of modern academia, this is a triumph.  And then following on from that, separately, earlier on, we've got two tweets from people saying,  in this conversation going, you know, we're talking about this colonialism paper.  And then you stop tweeting about it because it's adding to the metric score.  You're just giving the guy extra metric points.  And I don't know quite how to feel about this.  On the one hand, people assume that our metrics are accepted.  It's obviously not a niche thing.  If people are worrying about it at this extent, on the other hand, obviously it's completely wrong.  And the, you know, awareness and the training and stuff that we're trying to do is saying,  it's not about the counts.  You always have to look at the mentions in the context and things maybe it hasn't filtered all the way through.  It's a ongoing challenge.  Oh, okay.  I'm almost done.  So then because we're at an all metrics conference and, well, you know, we talk about indicators most of the time,  I thought what better place to make up a new metric, which I'll call the e-index.  It's precise to one decimal point.  And it's calculated by looking at the number of hits on Google News for any search term.  I thought, oh, let's do that size. Let's see if we can gauge how popular our metrics is in the public consciousness.  So for all metrics, 375.0 hits on Google News.  And for the impact factor, there's only 388.0 hits on Google News.  So, yeah, that's pretty good going.  And then just to bring everyone down to Earth, I look for CRISPR and it was 193,000 hits on Google News.  So, you know, not sure about the public consciousness, but at least maybe in the places that matter, you know, we're getting somewhere.  I'll leave it there. Thanks a lot. I'm looking forward to taking on the breaks.  Great. Thank you, Yvan.  We might have to have words about your new metric.  We'll see.  Okay. Next up, we have Heather and Jason, who are co-founders of Impact Story,  and they are going to talk to us about the design and implementation considerations for an open-autometric UI and API,  which sounds pretty exciting.  Let me just find the slides.  Do we need to make it full screen?  Can I make it full screen?  Yeah, I don't think I've done that right.  Sorry.  I see.  Let me make it full screen.  It is exciting. You're right.  Okay.  So, going to give you a sneak peek of a thing called Paper Buzz.  And it's brought to you by Impact Story and PKP.  I think Juan's in the audience.  Wave one. There we go. Right there.  So, he can answer questions later as well.  So, all of us, I think, are the vast majority of us use data from Altmetric,  and it's fantastic, and it's really good.  It's fantastic for lots of reasons.  It's comprehensive data.  It's easy to use.  It's free for APIs to call.  It just makes me happy.  It's so easy.  It's free for research use, which a lot of us really value and makes it really  different from lots of APIs in the scholarly space before now.  And on its detail page, on the page you get to from the book market,  many of those details are free for everybody.  There's a lot there that's really great.  But it's a company that has, as a business model,  keeping their code closed-source and closed-data.  And it also costs money for journals and startups and others to build things on  top of this.  So, because of that, Crossref has decided that they will build something with  open data and make that available openly for people to use, open-source,  open-data and free for everyone.  And Joe Woss is in the audience right here,  and he's going to be talking about that late on Thursday in more detail,  the Crossref Event Data API on Thursday in one of the last sessions.  So, stay tuned for that.  So, unfortunately, because they're building a system which philosophically,  they're not making judgments.  They're not rolling things up.  They're keeping it at a low level of abstraction,  which is really useful for the ground infrastructure for something.  But it means it's not that easy to use, actually, for lots of people,  for lots of purposes in some ways.  So, there's real value in having something that's a higher level of abstraction there.  So, and one of the reasons there's value in building this higher level of abstraction  is because open is so important, we need adoption here for this,  for all the transparency issues that we've been talking about.  So, early on, they were talking about the Almetrics providers.  Sorry, just a second.  So, all the Almetrics providers right now are closed source,  the main four that are used, PlumX, Almetric, and even ImpactStory,  because we're mostly getting Almetric data right now.  It's closed source, and the methods we use for getting data these days are so complicated,  you can't summarize them in brief sentences about how to call Twitter data.  You really need the source to be able to document how it's actually done,  which requires open source.  So, the CrossFit API is all open source.  You can see what's actually there.  The stuff we're building on top is open source.  It's the only way we're going to get to transparency.  So, it's a real game changer for Almetrics as a platform and really important.  The CrossFit Event Data API has 35 million events so far,  though 30 million of them are Wikipedia.  That's one of the things that makes it difficult to use.  They're putting out a lot of low-level events, specifically right now,  a lot of low-level events about Wikipedia.  It makes it hard to get to some of the stuff that a lot of people want about Twitter and so on.  And again, they're doing this for philosophical reasons, but it does make it harder to use.  So, it's a PKKP, an impact story.  We're building a higher level of abstraction on top of this,  which we're really excited to show you.  So, it's open source, open data, free for everyone, built on the CrossFit Event Data.  The audience includes everyone.  So, PKKP is interested because they want to include this data in OJS for philosophical reasons.  Again, to embrace the open.  So, it's going to be great for journals, researchers, tool builders,  and then others that use tools built on that.  The API simplifies events, it includes counts, it includes summary statistics by day,  adds that sort of value on top of the CrossFit API.  So, if you haven't seen the CrossFit Event Data API, here is a snapshot of what it looks like.  There's a lot of detail here, as you can tell.  Again, Joe will go into it in more detail.  Here's what we're proposing instead.  Excuse me, here's what we're proposing in addition.  So, this, for people who don't want that level of detail,  we just, we deduplicate events, summarize them,  include more detail there, Jason will talk about it in a few more seconds,  in a little bit more time.  We also include metadata there.  So, the journal title, the journal authors, things that you might want when you get the DOI.  You could go into CrossFit and get that in a separate call,  but lots of times people want that at the same time.  So, we're again enriching that data to make it an easy to use API.  We're also building a website, a bit like the details page you get to  when you click on the altmetric bookmarklet,  that's free for everyone, includes all the events, and is indexed by DOI.  So, one of those pages exists for every paper.  As a bonus, it includes a new source, which we're really excited about.  So, Unpaywall Views.  Let me tell you a little bit about Unpaywall real fast.  It is a free browser extension.  If you don't have it, you can go get it right now at Unpaywall.org.  It's a browser extension that pops up this little tab on the corner of your paper  when you're on a publisher site or somewhere with the DOI.  And that tab is gray when it is closed access, but green when we have been able to find  an open access copy of it anywhere on the web.  So, we look in institutional repositories in PubMed Central and Archive and so on,  turn that green and you can click on it and go straight to an open access copy.  More than 100,000 people have installed it so far in the six months it's been available  with more than 4.6 million uses.  So, it's a really rich data set, growing fast, really exciting.  The people come from 185 countries and we're able to see about 30% of them are from  university IP addresses.  We're releasing that view data with privacy issues taken into account.  So, no IP addresses obviously, but we're releasing the country a rounded time stamp  so it's not very identifiable and whether it's a college location.  So, this information is also in our new API with one more bonus feature in that website  about what's buzzing this week.  So, we're using the CrossRoof event data and this Unpaywall data to see what are people  talking about in reading this week.  And on that note, Jason's going to give a demo.  Thank you, Heather.  So, I actually want to show this in Google Chrome.  It's not super well tested.  This is a very early beta.  We wanted to show you guys because we're just really excited about it and really  interested in hearing your comments because it is still in very active development.  We're going to launch this officially at the end of this year.  So, as Heather said, we think one of the really exciting strengths about paper buzz is that  it's all open data and it's open code behind it.  So, we've been talking a lot yesterday at the kind of other workshop, the all metrics workshop  and also some today about the provenance chain behind the data.  How do we know exactly where every single one of these events has come from?  And so far that's been really difficult to do because we have this open platform now  it's not so difficult to do.  We can also, not only can we trace the provenance but for practical purposes,  we can use this data anywhere we want, any way we want.  We think that's a really powerful, and a lot of people think that's a really powerful attribute of open data.  So, lots of people can build tools in this and we thought, well, we'll build a tool line.  So, we built this thing called paper buzz hot buzzing this week.  So, I'll just show you this real quick.  You can go there, it's live on the web and you can play around with it a little bit.  But we said, what are the things that people are talking about most online using CrossFit.net data  and also using Unpaywall data so we're able to see what people are reading,  who have this, the 100,000 people who have this extension installed,  we're also able to see what people are talking about.  So, we combine these in, get a little algorithm, again, it's all an open source code  so you can track exactly how that algorithm is formed.  And so, we can say, okay, what are the top ones?  There's this thing about case for colonialism,  there's another thing about being type 2 diabetes, that's pretty fun.  Because we have open access data from the OADOI API,  we can click through to read some of these, some of them we can't,  but we're able to see which ones we're able to read.  And then we can also sort of slice and dice the data,  because again, we have these rich data sources behind it.  So, we can pick a certain topic and say, well, I just want to read what are the top, you know, environment stories.  And so then I can kind of see, these ones are obviously about the environment.  Another thing that's fun is because we do have data about the audience  because of the IP addresses that we have from Unpaywall,  we can use that to help us figure out, okay, are mostly scholars reading this  or are mostly people who aren't scholars reading this?  Where scholars are sort of proxied by, are you coming from university IP?  Obviously, it's not perfect, but by using this, we can, you can sort of see,  I'll just use, I'll do it on all topics.  If you look and see what are the mostly non-academic articles,  the first one is one about tattoos, which you maybe understand,  it's going to be a little more interesting to the wider population, maybe just as specifically,  that set of scientists who are studying tattoos, probably pretty small,  where it's quite a lot of the general public.  There's a lot of them, we found it was interesting, a lot about people's diseases, right?  Because a lot of times if someone has this disease, they're staying very up on the research for that.  So, there's one story about this therapy, there's a couple more about, right,  so type 1 diabetes.  So, people who have type 1 diabetes, of course, are very interested in following this research,  perhaps even more than the researchers who are doing the science about it.  And then of course, because open access is really important to us,  we kind of filter by open access, and then as you go through and read,  everything is something that a person, even without university library access, is able to read.  So, you can go through, as you see, there's always like blue read buttons,  means that you're able to just go ahead and read that, open access,  either open access on the publisher site, or like I said with the OADOI database,  which has 90 million articles in it, we can look at and see,  hey, is it somewhere else free to read online, so we can let people click through there.  So, I'll just click on one of the score details, and this is the part that I think is a little more relevant  for a lot of the journal users, is we can actually get this breakdown of the altmetrics  that we're getting from CrossFit Event Data by source, as well as the Unpaywall data.  So, as you can see, there's 138 event, 131 events for this,  I just kind of picked this one at random, so quite a lot of tweets,  and you can filter and just see the Unpaywall view, so you can see where they're from,  there's one from Romania, that's pretty cool.  Maybe they were at our recent last conference, and then we got a red link there,  and of course we have the other types of links as well, like Wikipedia and stuff like that.  Eventually, people will be able to subscribe to get alerts for this,  but quite prominently places the view on API, because again, we feel like the open data behind this  is a real big part of the advantage, so you can look through here,  and without any trouble, you can build your own apps on top of our data.  The Hotness API is also free for anybody to use.  So, in a lot of ways, this stuff that I'm showing you here is not actually as big a deal  as what Heather was talking about, because anyone can build this, right?  Like, we built it, it was fun, it took us maybe a week or something like that.  Anyone, a lot of people here in the room have the ability to build an app like this on this data,  and that's what we think is really powerful, and I've been interested in all metrics for a long time,  and this is definitely one of the things that I'm most excited about,  because now we have an open data set for altmetrics that anyone can build stuff on top of,  and anybody can do research.  Certainly one of the weaknesses, which needs to be acknowledged,  is there is not nearly as much data, and the quality of data is not nearly at the level that you would get  from altmetra.com or from Plum right now.  They've got a lot more years behind it, and they've been collecting it for a lot more years.  So, we even have a little warning on this, it's pre-2017,  where the data is not as complete.  So, we're looking forward to that improving, that's definitely something that I would kind of consider as a weakness  of what we're doing, but again, I would really consider the strength being this incredible source of open data.  We're super excited about it, really happy to talk to any of y'all at the break, and thanks for your attention.  I'll just go to that.  Thanks very much, Jason and Heather.  I think that was an amazing amount of information in 10 minutes,  so I'm sure there will be lots of good questions at the end for you.  So, next up we have Daniela Lohenberg from the California Digital Library.  Daniela is a research data specialist and product manager,  and she is going to talk about making data count.  Hi, so today I'm representing Make Data Count,  which is a project that should develop data level metrics and elevate data to a first class research output.  So, we're representing our team, which is a Sloan-funded grant collaboration between Data Site, Data One,  and the California Digital Library, and you may recognize some altmetric faces included on this project.  So, before I go into what we're working on right now,  I wanted to give a little bit of a history of how this project has come since the beginning in 2014.  NSF gave a grant to PLOS, the California Digital Library, and Data One,  to do a pilot into data level metrics,  and this was really done by John Kratz and Carly Strasser,  where they did a survey out to the research community,  both looking at what would it take to develop data level metrics  and what do researchers value in data level metrics.  And what we found from what will we need to do to actually make data level metrics  is that data are much different than articles, and it's a much more complex process.  Data have multiple granules, they're more complex than a PDF, they can be versioned,  and data can be derived from other data.  So, there was a lot more components to look into when trying to develop data level metrics,  even learning from when we built article level metrics.  And what they found from the research community is that people valued citations, downloads, and views,  similar to articles, and also touched on social media mentions also being important to them.  So in 2016, Moore funded a meeting grant for California Digital Library, Data Site, and Data One  to come together and look at this survey and what steps would need to be taken  to take this project to the next level of actually developing data level metrics.  And what was acknowledged is that journal articles already have an infrastructure for this,  but data really need an infrastructure to make, to elevate them to a first-class scholarly output  and give credit to researchers for data like they do for their articles.  Even though data play a larger role in the recent process,  there just hasn't been an infrastructure for this so far.  Which takes us to now.  In 2017, Data Site, Data One, and California Digital Library were funded by Sloan  for a two-year project that we're now working on to develop these data level metrics.  So what are we working on?  The first is a formal recommendation for measuring data usage.  We're working with Counter because we know that we need a standardized way of actually counting data metrics.  So right now we've come up with a draft, and that's on a Google Doc  that we're hoping to get community input and feedback on.  And then we'll be incorporating that, putting it up as a preprint  and iterating off of the feedback that we get from the community.  But we'll also be starting to build out the rest of this from this first initial draft  of the counter recommendations we've come up with.  Working from that, we're going to develop a hub for these data level metrics  similar to article level metrics using Legado.  We'll be using a similar framework for DLMs.  We want to make usage tracking easier.  We want to drive adoption and show how it can be done easily  by doing mass adoption across repositories and publishers.  And we want to engage researchers across all communities  and iterate as we learn from this project within the next two years.  What will this look like?  Well, it'll mostly look in the back end.  What we will be doing is taking with the hub.  We'll be taking citation usage information in,  and then repositories can be feeding information and log crunching  similar to article level metrics that we then can display out  on any repository that wants to use our consistent way of doing data metrics.  This will be all open source.  Everything will be documented on GitHub.  And anyone who wants to be involved in working on this  from the repositories, we'd love to get earlier adopters.  We'll be beginning with the data one and the California Digital Library  repositories first to show how log crunching can be done.  But we're looking also for other early adopters interested  in getting a start on displaying this.  And by the end, we hope to have a standardized widget  that also could be on repositories showing citations, downloads,  views, and mentions of social media for every dataset, just like articles.  So when we entered this space, though,  we knew that we couldn't be the only people working on data metrics.  So we wanted to do a survey to see what else is going on in this space.  So we talked with anyone who got in touch with us  when we released a survey about what we're working on.  A lot of publishers, societies, journals, repositories,  maybe a lot of you that are in this room are watching the live stream.  And what we learned when talking with them is that we need a better system  to monitor and track usage and impact metrics for data.  Right now, there's not a standardized way.  People want to know what's going on with their data,  and it led to more community input into what we can do to make data count  and data level metrics really useful.  So ways to get involved and where we're headed with this.  As I mentioned earlier, the countercutter practice for research data  is really where we're looking for community input.  I know a lot of you in the room now could give really valuable feedback on this,  and we're hoping that as many of you can contribute as possible.  We also have a link that I can send out for the survey.  We're looking for people who want to be early adopters in this.  We have a list of them now that we're trying to put together,  but the more the better, the more community input we can get into this.  And also help with community outreach.  Once we do have this up and running for data level metrics,  both researcher, publisher, all metric, all views,  to make sure that people are citing their data and talking about data  like they do of articles.  And also on our website, we have a roadmap for where we'll be headed  and where you can meet with us.  We really need the community to be a part of this, to make it a success,  and we're really excited about this.  So please get in touch with us, follow us,  and find me at the break to talk more about it.  Thank you.  Thanks very much, Danielle.  I have some really nice suggestions at the end there,  I think, for how people can get involved.  Next up, we have Polly Allen.  Polly is Director of Product Management at Plum Analytics,  and she is going to talk about all metrics in societal input.  Hi, everyone. How's everyone doing?  Okay, I'll stretch it out here.  So my name is Polly Allen,  and I was formerly Director of Product Management for Plum Analytics.  We were acquired by Elsevier in February 2017,  so I'm now Director of Product Management for Research Metrics at Elsevier.  And we're talking today about altmetrics and societal impact.  So thinking back to the early days of the altmetrics conference  and how the movement was born,  how one of the bright and shining hopes for altmetrics  has always been that it could provide some way to measure  social impact, or at least proxies for social impact,  some way to know in an area where we have not a lot of data,  can we drive better data-based decisions in the whole area as a whole?  And we've made a lot of really great strides in the area so far,  and there's, of course, so much more to do.  It's exciting to see that we're moving from an era of looking at  just looking for the signals in the online exhaust that exists out there  and things like social media and mass media,  and looking now more in specific places where people are saying,  can we measure social impact in a particular field  and find out where my research is being read and referenced  in areas like policy documents and clinical citations.  And it's some really exciting developments that I think provide a lot more usefulness,  while at the same time we're improving the robustness  of the underlying data, the different ways it can be gathered,  gathered and counted.  And I wanted to go through an example today of societal impact.  What are we talking about? What does that, what do we mean by that?  And how are we measuring it compared to how it goes on in the real world?  So let's consider the example of clinical guidelines and clinical citations.  So for a piece of research to gain clinical impact,  what has to happen?  So, of course, it has to be published,  and then it has to be read, consumed in some way by someone,  and largely, you know, those activities are moving online to a greater degree  than they were in the past.  The dissemination of that research is still largely at conferences  and people, you know, making sure other people in their field are aware of it.  Again, more on person, but there are ways to detect signals for this online  with platforms like SlideShare, et cetera.  So once things are begun to be disseminated,  there's the amplification of those messages,  and that's a really interesting place to see where people are choosing to amplify their message.  Does this differ, you know, from different fields?  And does the newsworthiness of an article really indicate this is an, you know,  amplification-worthy message, or is this something that just happens to be  fuzzy and something of interest to the lowest common denominator?  These signals are harder and harder to tease out,  but we started to, you know, look at the work of what do these signals mean.  And the goal, ultimately, for a lot of researchers is for their research  to become part of the consideration set for the people who make decisions  around clinical reviews and clinical guidelines to say, you know,  please read this work, become aware of it, and then actually reference it in your work  so that I can know and include in my funding application that my work actually had, you know, an impact  when it finally got to the stage of being included in a clinical review, a guideline,  a bedside assistance tracker, or part of physician education even,  and made its way to the clinic ultimately.  So what we do today is that we're able to put little, it's hard to see on the screen,  there's little gauges on a bunch of different spots, so we take our little gauge  thermometers and we can stick them in at various points in the pipeline to get these proxies of impact  because actually measuring impact quantitatively is very hard, and that's what people are asking for  in order to make decisions.  So we have this ability to gather events and count them, and what I'm excited to see now  is even though there's so many other areas, this is one example where we're looking at,  this is one pipeline we can stick things into, gather data from APIs and crawling,  there's a lot more, there's a whole world of, sometimes it feels like, to discover in terms of  what are the other areas, can we start looking at patents, can we start mining legal documents  for economists and social scientists, and really try to see where in all of this online chaos  we can actually try and make some order and pull some numbers out in a robust way  that people agree is counting something useful.  But when we talk about getting to the next stage, it really is getting beyond counting  and how do we count things.  And it's about designing systems that help people make decisions and provide feedback loops  the way your speedometer does on a car, so that you can make decisions in more real time,  and so that you can assess yourself, do all the things that people are eager to do  to improve their own performance in various ways.  This, of course, is a double-edged sword.  So there's power, great, great power in providing, you know, having systems that use data,  especially data at scale, underlying them to make decisions.  But there's also great, great danger.  And in Catherine O'Neill's book, Weapons of Math Destruction,  she describes a lot of really interesting scenarios in place today  where systems that have big data underlying them have been designed ostensibly for efficiency  and also to make sure that we're reducing the human bias and human error in a system.  And she shows how so many of them have actually done the reverse,  where they're actually encoding the errors that exist in the system today  and reproducing them in a massive scale, creating these rich-get-richer situations.  And I think it's important that we make sure as we're designing systems  and starting to move forward in that arena that we're making sure we recognize the underlying biases  in a lot of the data.  When we're looking at Wikipedia metrics, for example,  I think it's important to remember that less than 15% of Wikipedia editors are female.  What kind of biases are we introducing by counting these metrics and not counting others?  I don't think the U.S. News and World Report could have anticipated in 1983  when they first, as a struggling publication, decided that they would take on an ambitious project  to rank 1,800 U.S. universities in terms of excellence, their definition,  that they would have had such a huge impact on academia as a whole.  And so when they chose, originally they started doing the rankings,  they were doing qualitative surveys and the complaints arose, this is too biased.  Who are you phoning? Why are you phoning those people?  This is just opinion-based and it's actually affecting our bottom lines of universities.  You need to get more data, you need to get more quantitative.  And it's another example where how do you measure excellence?  How do you put that number on that and rank people based on that?  So from a system where people might have heard from word of mouth  that such and such a school has the following positives, negatives, find a good fit for them,  we moved more to a system where because of the attributes they chose to use  when they moved to a quantitative system in 1988,  it was very interesting to see that they chose, they wanted their ranking  for people to believe these are the schools that are excellent.  When people think of excellence, what schools do they think of?  They think of Harvard, they think of Stanford, and what attributes do they share?  They share very low acceptance rates, they share alumni go on to prestigious jobs  with high pay, what they didn't include or rank highly in those early days  were things like admission fees or the percentage of minorities that are going to the school.  And what we saw between 1985 and today is that tuition rates have risen 500%,  four times the rate of inflation, partially driven by schools need to improve their rankings  in a system where it was designed for winners continue to win and losers  unless they get a huge injection of cash, unless they start making sure they have more applicants  and people then that they can take, how do we drive that?  A lot of them did this by raising money to provide great sports teams  and get a lot of attention in the media.  So I think it's a cautionary tale about how the metrics we use today are being used  and how especially when you have a system that's designed using proxies,  they're more open than ever for gamification and we have to be wary,  but also think ahead so it's not a reactionary thing afterwards  and we've designed the system and it's in motion, but really thinking through  what are the implications of the underlying biases in a lot of the systems we're designing.  So it's not all doom and gloom.  I want to reassure you I really am excited about the area and I'm incredibly optimistic.  I think a lot of the problems we see in underlying big data systems that have these problems today,  they're born of laziness, they're born of the human tendency to use what's available  and if they have a quick number they can grab to use for whatever,  even if it's not fit for purpose, there's a danger that people will use it lazily  and not in appropriate ways.  So it's an ongoing effort for education, working closely with the people who are making decisions  based on our metrics and making sure that we're conscious going forward,  not just of what do we use to measure social impact,  but what is the social impact of using those measures.  Thanks very much.  APPLAUSE  Thanks very much, Polly.  Some really pertinent points in there I think that we all need to consider  as we go through the next couple of days.  Next up we have Jean Lu.  Jean is Product Development Manager Altmetric and she is going to talk about Altmetrics for Books.  Hello everybody.  My name is Jean and I lead all new product development at Altmetric.  So we've been talking a lot about metrics this morning and I think what we should also remember  is that the second aspect is the narrative.  So the mentions, for example, of research outputs make up those metrics that we are talking about.  So I'm going to tell you a story about Altmetrics for Books  and if you were in the workshop yesterday, that was one of the types of things  that might be perhaps coming up in next year's workshop.  How do we start to talk about Altmetrics data for more than just journal articles  and what does that even look like right now?  Well, bear with me.  So in The Donuts Quest, a book that I personally authored,  is a thrilling tale of Altmetrics for Books that begins with a little donut.  Once upon a time there was a little sentient donut  who wondered what people were saying about scholarly books online.  The little donut embarked on an epic quest across the scholarly landscape  hoping to map the online locations of all the world's books.  He crossed the Taylor and Francis Mountains  and ambled through the Springer Nature woodlands.  He traversed the Elsevier Rapids, canoeed around the venerable Wiley Islands  and surfed on the bustling Amazon River.  So I'm going to pause here for a minute because these are all obviously publisher names.  So when we start looking for Altmetrics for Books,  one of the things we need to do is know where to look.  So we need to know what kinds of records we actually want to track.  That is actually a surprisingly difficult task  because every single publisher will have lots and lots of different domains  that they host book records.  So when you think about a book, do you think about the physical version of the book?  Do you think about the e-book?  Do you think of the e-commerce page?  An institutional repository page?  Those are all valid places that books live.  And to actually find them and actually make sure we're tracking them properly  is actually quite difficult.  And for each publisher we've ever worked with,  some of them have at least two to three domains, if not up to eight.  And we have to make sure that our tracking software can actually recognize  every single one of those web page formats  so we can recognize that that's a book and not just a random page on their website.  So let's keep that in mind.  He even followed the mysterious nomadic books  who somehow managed to live in many places all at the same time.  So this is exactly what I mean when I say they live in different places  but they are actually the same book  because the concept of the book is what we're tracking.  We're not tracking necessarily the physical copy.  We could be, but it's not online.  We want to, for example, look at the e-book version  or just the e-commerce landing page.  But all of those represent the book.  And the author themselves may not use one particular version  to refer to their book when sharing the link.  So we need to be able to get that book wherever it lives  and you might have heard recently that AllMetric.com started tracking Amazon.  That was actually quite an undertaking because it turns out  there are tons of mentions of Amazon  and when we're working with data providers such as Twitter  that gets expensive very fast to actually track all that information  parse it down to the exact books that you want  because we're not necessarily wanting to go and track Harry Potter.  We want academic books.  So that separation is quite difficult.  And these books, these nomadic books, they live all over the place  so we need to identify where they are.  But we have a little secret weapon you see.  So the little doughnuts searched tirelessly for the magical ISBNs.  Special numbers that identified each unique book.  So ISBNs are really interesting because they come in two forms.  A 10-digit form and a 13-digit form and you can convert between the two.  However, 10 digits is also a phone number.  So recognizing an ISBN from a phone number  or any other random string of numbers is quite difficult as well.  You need to know the context.  You need to be able to find the ISBN out of a sea of numbers  from a policy document, for example.  You need to know that it is an ISBN in that case.  Also, sometimes, you had to try and avoid his foe,  the bad metadata snowman.  The bad metadata snowman like to mess with the little doughnuts  metadata reading goggles, making it hard to track books and chapters.  So Ewan already talked about this in his presentation,  that publisher metadata seems to be getting worse.  That's definitely true of journal articles.  For whatever reason, some publisher metadata seems to be disappearing.  With books, oftentimes, meta tags are not even available at all.  And so when you don't have meta tags to rely on,  what you have to do instead is scrape numbers like ISBNs  and titles and authors and everything out of the page directly.  Which sounds like perfectly valid.  It is fine. You get that information anyway.  You can build a book record.  But as soon as that web page layout changes, that breaks the scraper.  And you can imagine in a recently small team that we still have it all metric,  we have fewer developers and we have non-technical staff.  It's pretty unsustainable to go and fix every single page  that breaks to try and recapture that data.  So what we really want is for book publishers to add meta tags  to all of the different places that their books live.  And one of those meta tags has to include the ISBN.  So this is where metadata can really mess with things  because as soon as we don't know what the book is,  it's hard to match the attention to a specific record.  But the little donut remained determined,  knowing that ISBNs were the keys to identifying the books,  especially the ones that lived in many places.  So like I said, if you put a book record on a repository  and you put an ISBN there, that's great.  As long as we see the ISBN somewhere else,  we might be able to say, oh, that is the same book.  So we're building that book concept.  We know that all these different records all over the web  refer to one book.  ISBNs are a little bit different than DOIs.  They don't resolve to a specific URL.  So we don't have a canonical place to say,  this is the one place all the books actually came from.  So that makes our lives difficult.  Some books do get DOIs,  but that is not certainly the universal thing.  And ISBNs, you can have multiple for a specific book  because different ISBNs refer to different versions of a book.  So as you see already, this is very complicated  compared to tracking journal articles,  which has a very established infrastructure,  a very established way of putting meta tags on a page,  assigning DOIs, registering them,  and having them resolve to real URLs.  So it's quite a complicated mess that you end up in  when you're trying to juggle different ISBNs  and find them correctly.  But in this story,  we're not going to focus too much on the challenges, you see.  It is a nice story.  So finally, with a map of online book locations,  the little donut returned home and logged into the  fantastical Altmetrix database.  So obviously, if you've ever followed altmetrix.com,  you know that actually we don't go looking for the output records first.  The signal to start looking actually comes from our attention data.  So we see a tweet come in,  and it links to a domain that we recognize.  If we recognize that's a book publisher domain,  we go and get that book.  But we need to know that domain first.  We need to have that map of all the different domains  where content we care about lives  in order to make that connection.  And once we do, we have a huge database of great stuff.  But you see how very fragile it can be  if the books don't have the proper metadata.  So suddenly, he was flung into a rainbow of online attention,  flying past over one million mentions from mainstream news,  policy, social media, syllabi, and more.  And so, yeah, already in the short time  we've been tracking altmetrix for books.  I'd say about two years now.  We officially launched altmetrix for books last year.  But for about two years now, we've been collecting data.  We've at least been following a lot of the domains  that we know host book content.  And it's very astonishing, I think,  because we didn't even start by tracking attention sources  that were specific to books.  These are all sources that we know mentioned journal articles.  And even within those, they were already,  as you can see, over a million mentions  of these different books and occasionally chapters, too.  It was quite astonishing.  And, well, the donut agrees.  He says, this is wonderful stuff, exclaimed the little donut.  I must tell the authors.  Because one of the things that I think strikes me  when we were talking about altmetrix for books  is that there is a really strong need in the books area  compared to journal articles  to actually have some kind of meaningful data.  There isn't the same saturation of citation metrics  that you see with books and chapters.  If anything, the researchers are spending months, years,  on this content.  And then they get really no feedback afterwards.  Can you imagine publishing a blog post  and not checking Google Analytics?  It's kind of an odd feeling to not have any feedback  to know that the book's been read, utilized, et cetera.  So this is something that we think authors really must see.  And so what we're doing,  and this is our little donut's mission, too,  is to make that visible.  So he baked batches of rainbow donuts  and set off again, this time with a new mission,  to show hard-working authors all the online attention  paid to their scholarly books.  Thank you very much.  APPLAUSE  Thank you, Jean, for that very different presentation.  We're a little bit tight on time,  but are there any questions for any of the panelists?  Yeah, would you mind coming to the mic if there isn't one?  Can you hear me?  Are we good?  Just to make things more complicated and to add,  especially to Jean's task,  there are a few fields where people publish anthologies,  and so books by themselves are not enough.  We need to be able to track the chapter  in the same way that we track a journal article.  Can you talk to that?  Yes, so...  OK, so chapters are actually...  Whenever they do get their own webpage,  very often they are assigned a DOI.  So usually chapters are listed in a table of contents  on a book website, but for some publishers,  so Taylor and Francis was one publisher we piloted,  Altmetric for Books with,  and they had assigned DOIs to all of their chapters.  So in that case, it was possible for us to get chapter-level  attention data, and what we opted to do was roll that  into a big Altmetric details page  where you could actually toggle through the table of contents.  I won't speak too much to the way we scored it,  but we did actually make sure we...  When we gave an Altmetric attention score to the book,  we actually just pooled all the mentions for the chapters  and then didn't sum it up for...  So because that would be double counting,  and you could just, for example, mention loads of different chapters  to game your score.  So we actually didn't let people do that,  we just pooled it and then counted it as if it was one entity  to calculate the score that way.  So it was not as biased,  but for some publishers that have the DOIs,  it is definitely possible,  but that's where that publisher awareness of adding DOIs  to their different bits of content,  that really needs to start happening.  Because the technology is there,  it's almost like we're waiting for the publishers  to kind of pick up the pace and upgrade their systems.  I wanted to mention on the Plum Analytics side,  we really love the roll-up you guys are doing  on the Altmetric side, trying to, you know,  there's still open questions for a lot of the metrics of  if someone is just tweeting about a chapter  and you roll those up, that's not double counting  to count those between chapters.  So again, the devil's in the details  in a lot of these metrics of that roll-up task  and how do you properly attribute things to the right places.  Again, I do think it takes work from the publishers  to get on board to make sure that the metadata is there  and usable too.  Thanks, next one.  Thank you for your presentations.  My question involves, so we've heard about three or more platforms  and systems and all these types of things.  I wonder about the relationships between these platforms.  Are we partners?  Are we adversaries?  Are we working together?  And then the relationship, for me, for instance,  as a researcher, how should I choose between one or another  to do my research on?  And for instance, as a university,  how do I make a decision of which one to buy?  Do I need to buy all of them?  I mean, I'd like to see some answers to.  Who would like to speak first?  Jason?  I think coming from sort of the bibliometrics world,  I think that's a great opportunity for research as a researcher.  So quite a lot of bibliometrics research  as new bibliographic databases have come online  have been in looking at the properties of these various databases  against one another.  So originally there was this Web of Science,  and then Scopus, and then Google Scholar,  and then additional ones as well.  And there's quite a lot of literature now from the bibliometrics community  saying, well, what are the strengths and weaknesses  of these different databases for different kinds of research?  And I think that's a terrific job for the research community  to accept that me and Heather are occasional publishers.  That's something that I think we could be publishing on.  I think the different services can be publishing on it.  But I think it's a great research question  and one that is probably never going to be done,  because especially in the early days of this particular world,  there's a lot of changes going on really quickly.  And I think that's one of the...  I always try and contrast this between the early days of bibliometrics,  where it started quite similar to all metrics,  in that this was a set of metrics that a lot of people had a lot of concerns about.  We, when Eugene Garfield proposed this idea of mining citations  to try and track the progress of ideas through the literature,  he was met with a tremendous amount of skepticism.  Couldn't get anybody to listen to his ideas for years.  And then when he finally did, the research world was, in many cases,  extremely antagonistic this idea.  Like, how dare you reduce what we're doing to a number?  And that antagonism has certainly not gone away entirely,  but in many ways the research community has reached sort of a piece  with the idea of citation metrics.  That took decades.  And I think in many ways we in the all-metric community are way ahead of that  because we can at least look at that example and say,  okay, well, here are some of the questions people have.  Here's the kind of research needs to be done.  One thing I'm really excited about is there are multiple organizations  who are doing this work, so at least we can start doing that compared to contrast.  Again, Web of Science is the only game in town for quite a long time.  So I think trying to draw lessons from that,  my two-part answer is trying to draw lessons from early progress  of the bibliometrics world and then, like I said, relying on the research community  to look at the pros and cons.  As for the commercial benefits of one another system,  I think that's probably best left to others.  If we could keep comment short, but would anyone like to add anything?  I'll think you stabbed it.  I mean, I mentioned before about the awareness and the training and stuff  being equally important to the technical development.  So first of all, I'd say, is early enough, certainly, for all of us to benefit.  Frankly, the more people involved in the metrics, the more people learn about it,  and the more scope there is for when it comes to the commercial side,  where it would be incorrect to say we're definitely competing with plum commercially.  That doesn't mean we would never work together or we hate each other or anything like that.  It's not that kind of relationship.  But in general, on the non-commercial side, I'd say it's much friendlier,  certainly than other places, other industries.  I think the main point, like I said earlier,  it feels like there's this open world that is ready for exploration  that we've only started to explore.  And I was really excited to see Reddit highlighted in the keynote speeches  like this is an area that we've been gathering metrics for ages  and there's very little research on it yet.  So these platforms are available for research.  We're very happy to work with researchers who approach us on our side.  I know Altmetric has their API available publicly.  And I think it's going to take collaboration.  It's going to take open data systems.  It's going to take commercial partners as well to get where we need to go in the field.  Okay, I think for the sake of keeping me on time,  do we have one more question at all?  Yeah, in the back.  We'll try and get some more mics.  Hi, my question is in the,  it's in the debate on societal impacts and I think it's mostly directed at Polyengine.  And it's about country level metrics.  Have you tried and attempted to grasp the issue of country level metrics?  And if yes, could you please talk about some initiatives or some things that you've done in this work?  Regional, like Regional 2 a country or like?  Yeah, like any research that has been produced.  Like if you try to aggregate the work of researchers of a country and try to draw some comparisons, for example.  So I, we don't have specific research that we're doing yet,  but what we're trying to do is surface some of the country level information we do store in our database.  So for example, policy sources, news sources, Facebook, I think in Twitter are the four that we currently have available in Altmetric  and we actually see the country level data.  What we want to do a bit further.  So we started this project early in the summer to actually make maps so that you could look at,  for any given search query you do in Altmetric,  you'd be able to see which countries had mentioned you the most in those sources.  And it's quite interesting too, because it also reveals to us the coverage gaps.  So when we look at the whole database, for example, and then we look at the maps of news,  we can see there are some parts that we're not covering very well, some parts that are very well,  I mean, I'm Canadian, so we've added a lot of Canadian news sources.  Let's just say that.  But, you know, so there are definitely some regions of the world that we're not able to cover yet, but we can visualize it.  So the first step for us is to visualize the next step is to delve deeper,  but all that data is available in our API.  So we would want to surface that and get some really interesting research coming out about that.  So we're actually going to talk about this this afternoon actually and see a little bit about these maps and some.  Okay.  Now that we're part of Elsevier, Elsevier produces SciVal.  So that's been used in the bibliometric field for a long time.  And we're definitely, you know, speaking with them long term about how we can apply the same kinds of analyses to Altmetric data,  where it's appropriate and where it makes sense.  And then the second point is that what's really interesting to me at the country level and the region level is that  it becomes important to track completely different sources when you're looking at different regions of the world.  And specifically China, it came up in the workshop yesterday where a lot of these social media sources are blocked.  What can you use again for proxy?  So we've definitely been working with partners worldwide to try to find usage data, citation data from regions that typically are underserved  and don't have metrics in this area at all.  It's difficult actually because you need to aggregate the data from the article level or the research level.  And so you need a good coverage for that.  Of course.  Okay.  Okay.  Thank you very much.  A big thank you to all of our panelists.  I'm sure if anyone did have further points that they'd like to pick up, everyone will be around during the lunch break.  So please do go up and have a chat with them.  Okay.  I'm now going to hand over to our next chair for the Altmetrics and Research Evaluation Session.  Okay.  Hello, everybody.  Hello, everybody. My name is Dono Kubo and I am pinch-hitting for Katie Hinckling. She  ran into some travel issues. So, you know, as one of the organizing members, I know that  she really wished she would have been here. You know, I know she's done a lot of work  in this space, but so instead you have me for a little bit. So, this session is going  to be on altmetrics and research evaluation. And so, I'd like to introduce the first speaker  and this is Kate Williams from the University of Cambridge, who will lead us through an  understanding of emerging cultures of evaluation. Hello, everyone. My name is Kate. I'm a research  fellow at the University of Cambridge and also at Harvard. And I have an ESRC funded  grant on altmetric indicators of research impact in development research organizations.  And this project involves mixed methods, so it includes ethnography, interviews, and quantitative  data analysis. But I want to talk to you today about some preliminary development of a theoretical  framework which arising from this project. So, it's quite a departure from what we've  been talking about today. Although it links to Mike's observations about whether we need  a term for altmetritions and to Paulie's question about the social impact of using social  impact metrics. So, research is obviously crucial in addressing complex environmental,  social and economic challenges. As such, research institutions are eager to develop  efficient ways to measure the wider impact in order to inform strategies around research  focus, funding, communication and reporting. Yet, the measurement of impact is highly contested  and expertise in the area is under theorised. So, informed by a sociological perspective,  today I hope to intervene in this setting by reframing impact assessment as a space between  fields which shapes cultures of evaluation. And I hope this contribution will provide  new directions for the study of impact and altmetrics.  So I have five aims. The first, I begin very briefly by situating the recent focus on research  impact in its wider contracts. And then I outline three of the core ideas. The first  is that impact assessment occurs within a space between fields. The second is that altmetrics  are making available new types of capital or resources. And the third, that this framework  can be used to understand emerging cultures of evaluation in research organisations. And  I conclude by outlining an example from the case study of the World Bank. This work diverges  from previous work because of its field theoretic approach which is a distinctly sociological  perspective to elaborate this hybrid nature of the space between fields.  So recent years have seen a reformulation of science systems towards more applied socially  accountable enquiry produced by an increasingly diverse set of research institutions with  greater emphasis on the costs, output and quality of research. And recently this focus  has extended to attention on wider societal impact. And accordingly, new techniques of  impact assessment are emerging. Here amongst these, of course, are altmetrics. The available  academic literature on altmetrics largely focuses on the statistical comparison of altmetric  data to establish sources, the validation of measurements and the identification of  biases. Thus, there's a current need for greater understanding about how these metrics take  on meaning in practice. Given the significant recent uptake of altmetric technologies, research  is required that considers the specific cultural and institutional effects of these new tools,  set in the context of nuanced understanding of cultures of evaluation around research  impact. So using a sociological framework, this paper seeks to address this gap in two  specific ways. First, although there's a body of sociological research on traditional  metrics of research quality or scientific impact, work specifically on altmetrics occurs primarily  within the fields of scientific metrics and library or information science. Against this  backdrop, however, two recent relevant sociological publications argue that altmetrics are of  growing importance in contemporary research systems. And these studies indicate a growing  sociological interest in altmetrics and provide potential avenues for theoretically informed  research. Second, relatively few studies have focused on the sociological effects of altmetrics  rather than the application of altmetric methodologies to questions of sociological interest. Therefore,  there's a need for distinct but related work, including ethnographic studies, on the institutional  effects of research evaluation that pushes novel insights on impact assessment and altmetrics  out to different fields while also contributing new perspectives to information science and  research evaluation and policy. So onto my first core idea. In the study of expertise,  the production of knowledge has been dealt with as a function of relatively institutionalised  conditions. So there's an assumption that areas of expertise will grow into more defined  forms as they evolve, whereby they coalesce into formal structures such as academic disciplines  and professional organisations and thereby establish control over the production and  certification of expertise. So for example, in law. However, having only featured in Britain  and Australia since around 2004, the area of impact assessment does not conform to this  ideal typical notion of a bounded profession, disciplinary intellectual field of inquiry.  And one explanation is that impact assessment is an emergent field which will coalesce into  formal structures which perhaps underpins a lot of current altmetric research. The assumption  here is that it will mature into an institutionalised field, a discipline with strong, highly regulated  boundaries that offer autonomy from other fields and contain a specific logic and fixed  capitals or resources. However, an alternative explanation is that impact assessment occurs  in a space between fields because it comprises interactions of different arenas, not simply  different criteria within a self-contained field. The available resources or capitals  that make up this space are drawn from various fields with their own implicit and explicit  rules, activities and norms. Thus impact assessment can be conceived as a non-standard field of  expertise. And these types of fields are located within a multi-dimensional social space that  contains a number of actors, stakeholders and organisation types rather than tied to  a particular community. For example, in assessing impact, the borders are blurred between university  departments, research centres, publishers, libraries, academic networks, professional  societies, the media, government and the private sector. Each group can make a valid claim for  expertise in this area. The key feature here is shifting permeable borders that permit  people, ideas and techniques to travel between multiple fields. For the range of actors who  converge in these spaces, this poorly articulated hybrid space permits freedoms not available  within established fields. So actors are able to draw on the expertise, ideas and language  from outside their own field while remaining credible and authoritative within their distinct  arena. For example, impact assessment requires technical skills, so bibliometrics or metrics  or academic expertise, expert reviews, narrative case studies. And to be useful, it must be  communicated or implemented in particular ways. So performance management, institutional  evaluations, public accountability documents or national evaluations. And thus it requires  an ongoing balance of technical requirements, communications skills, orientation to value  for money and an orientation to policy, practice or management. Accordingly, this is also true  of the demonstration of impact by research organisations. To be impactful, research must  contain a definitive academic identity yet that produces a range of products for receptive  audiences and positive outcomes for diverse groups, which can be assessed by specialist  evaluators through a predefined set of practices. So as shown here on the slide, the measurement  of impact, the capitals or resources of several fields are at play. Specifically, this might  include the field of knowledge production, the rigor and methodologies of academia, politics,  the media and economics. So this idea offers an alternative perspective whereby to assess  research impact, evaluators and audiences draw upon various types of capital from different  fields and accordingly to demonstrate impact research organisations now must also do the  same. So the second idea here is that in addition to these fields that I showed before, we're  also an emerging, we're also observing an emerging field of metrics, which is the skills  to create and interpret rankings and indicators. In contemporary research systems, metrics  are imbued with significant power. In addition to influencing resource generation and allocation  and the recruitment, assessment and promotion of staff, they also change the way we do politics,  assess value, allocate guests, resources and justify worth. So this emergent field of metrics  links various actors to users or clients using the language of numbers. Yet the field of  metrics itself is quickly evolving. Scholarship and academic communication is rapidly changing  with many of the scholarly processes occurring online, which creates a wealth of data that  can be used to inform scholarly and impact metrics. Especially for applied policy and  practice focus fields of study, alternative metrics are attractive and are sensibly helping  scholars and institutions to find and observe what impact looks like. And as evidenced by  the fact that we're all here today, we're seeing large changes in the field of metrics. As  Halstein Bowman and Costas put it, what is considerably different, however, is that  altmetrics capture events on platforms that are in a constant state of change and whose  use and user communities are new, diverse and not entirely understood. Thus, the increasing  interest in and importance of altmetrics makes new types of capital available. For example,  on the research production side, it's at least theoretically possible for a researcher to  gain legitimacy by demonstrating a huge social media following or to bolster credentials by  combining citations and altmetrics on a CV or in a performance evaluation. And on the  evaluation side, funders, publishers and research institutions can also gain legitimacy by demonstrating  value for money and policy orientation by investing heavily in altmetric technologies.  So in this context, altmetrics are now a significant factor in the space between fields. Given  these recent challenges, recent changes, it's important to consider the specific cultural  and institutional effects of new metrics and technologies. This requires an examination  of how actors are involved in the process of gaining and maintaining legitimacy from  various fields. And highly relevant here is the work of Michel LeMont, who shows that  sociological research in particular can facilitate unveiling the evaluation criteria and bringing  to light the devices, institutions or cultural and social structures that support or enable  them. So the theoretical development that theoretical framework developed here could  potentially allow us to examine the intersection of cultures and logics of different fields.  So here I move to an example of research production and evaluation in an international development  institution, namely the World Bank. The bank was selected because of its dominance in  global policy research and because of the tension between its research and policy functions.  Over a three-month ethnographic period, I examined the context and structures of impact  at the institutional and individual level, examining the acquisition of various capitals  through existing and emerging cultures of evaluation. And I'll provide briefly some  early insights in order to demonstrate the potential for this theoretical framework for  understanding how cultures of evaluation are changing. So at the bank, as elsewhere, the  increasing focus on research impact is a structuring force in the construction of valued and legitimate  knowledge. The bank is an interesting case because it's engaged in a balancing act where  it seeks to dominate the language and resources of metrics, media, academic, political and  economic fields. And as a result, the cultures of evaluation reflect these diverse portfolio  of capitals held in constant tension with one another. As shown by some examples here,  what I've seen in particular is that altmetrics are starting to change contests for capital  or resources, but they're not in full force yet. There's great interest and sizable investment  from leaders, managers, communications and publishing teams in these new technologies,  and researchers are aware that they need to be developing the corresponding new skills.  As one researcher stated, in this day and age with this technology, it's no longer acceptable  for us just to be speaking to a small community, whether that's just an operational community  or just an academic community. There's always a sense that there needs to be some kind of  above and beyond. So my tentative impression is that we're seeing changing contests for  capital, whereby legitimacy can be gained in new ways, which will require research with  a strong theoretical framework to unpack. The implications of the framework I've outlined  for the altmetrics community might be around specific types of metrics, for example, that  policy and media citations are particularly valued in policy research contexts like at  the bank, because they speak specifically to political and media fields, where a great  deal of legitimacy is sought and amassed. In addition, the field of metrics is still  emerging, so we can expect to see shifts as the techniques and theories develop, which  will be reflected in the cultures of evaluation that rely on legitimacy gained from various  fields. Thank you. Thank you, Kate, for a very insightful presentation. The next speaker  I'd like to introduce is actually a former colleague of mine, Rebecca Kennison, and  she is from K&N Consultants, and she'll be discussing Humemetrics HSS Initiative, Rethinking  humane indicators of excellence in the humanities and social sciences.  Kate, I see your sociology and I raise you philosophy. We're clearly the theoretical  framework. I don't know what you're talking about. We are a melon-funded team that is  being led by the Dean of Arts and Letters at Michigan State, who is a philosopher, so  that's why I say that. You'll see a lot of that in what we're talking about here. Like  Kate's piece about theoretical frameworks, we're also proposing a framework, and we're  starting with this problem. It's all good to do all these great things, but you're hardly  making any impact on social media, and this is the world in which increasingly, particularly  administrators, and again, our team is led in part by a Dean, so we're really focused  on what we can do to help him. So the goal of the Humemetrics HSS Initiative, which is  humane metrics, is to create and support this framework for understanding and evaluating  all aspects of the scholarly life well-lived. There. You guess how your work could out for  you now? For promoting and nurturing these values in scholarly practice, and for empowering  scholars to tell more texture stories about the impact of their work. So we're taking  altmetrics and kind of flipping it around, and really taking a look at what values do  we want to look at? What do we want to encourage? And then how can we find a way to do that,  rather than what can we measure, and now we hope something's going to come out of that?  So we're flipping it on its head in a reverse engineer kind of way. So our preliminary values-based  framework is this, which is what we've thought a great deal about what we would like to see  in the academy, equity and openness and collegiality and quality and community. And we've started  to test that in conversation with people. And next week we'll be having our very first  workshop where we'll bring together 21 academics from Humanities and Social Science. We're  also just concentrating right now in Humanities and Social Science with the argument that  if we can get it right there, we can get it right in STEM. So we'll see. So what does  this mean in terms of what we're looking at? So equity, we're defining that as the willingness  to undertake study with social justice and access to research and the public good in  mind. There are a variety of different components of that that you can see here. Fairness, inclusivity,  diversity and so on. We're also interested in openness of all kinds. And by that we don't  just mean open access and open source, but also transparency and candor and accountability.  And again, willingness to change and adapt. These are all kinds of openness. So really  thinking about what the quality of openness might be. What does collegiality mean? We're  rapidly beginning to understand that without collegiality, none of these other values will  actually matter. There's no equity without collegiality. There's no openness without  collegiality. There's no collaboration without collegiality and so on. And collegiality, the  practice of kindness and generosity and empathy towards yourself and others, really hard in  a competitive environment like the academy. What about quality? So quality very important,  not just in replication and reproducibility and soundness, but also in intentionality,  really thinking about why you do what you do. These are all quality markers. And finally,  we're looking at community. So engagement with and leadership of communities of practice  and also with the public. And again, variety of elements that fall under that kind of value,  including solidarity. Okay. But what does that have to do with altmetrics? So that's  all very great, wonderful. We should all strive towards those values. Well, here's where we  think that we have shared goals with the altmetric community and with the work that many people  are doing and thinking about. Because what we're hoping to do is expose and highlight  or recognize and reward this scholarship that goes into all research activities and not  just publications in all the diversity of publications, but really looking at things  like peer review and mentoring and public scholarship of all kinds in order to be able  to reward all of those things. And so again, some things are easier to measure. And in  fact, this morning we found out not so easy to measure, right? So not so easy to measure.  And now what we're suggesting is we need to expand this beyond that to the work that scholars  do entirely. So I'm going to use an example of the syllabus as one of those scholarly  activities. Because creating a syllabus is a scholarly activity. Not just, it's not  just about teaching, but teaching is also a scholarly activity and we don't want to  lose sight of any of those things. So what might we look at in a syllabus? I know that  altmetrics is working with the open syllabus project. And one of our team members is Stacy  from Altmetrics. So we're very keen at looking in the first instance of what a syllabus might  what might we look at. So here's some elements that we think we can pull out in a machine  readable kind of way, but we'll find that out as we get into that. The course description  and the discipline and titles and authors and hopefully people are using DOIs and ISBNs,  but as we just heard, that's a challenge. And class scene, but also taking a look at  things like class engagement and student activities and assignments. How much time are you spending  within a course on a particular text, for example? That should tell us something. What  are you doing within your class about that? That should tell you something about engagement.  And tell us something different than when you're citing it in a published work that  you're doing. Because what are some of the things that we can learn by looking at how  a syllabus is created? Also tells us something about the class, the person, and so on. So  what can we discover about circulation of ideas? Is there a difference between what's  being cited in a syllabus and what's being cited in a published work? We have theories  about that, but we haven't really looked to see. I mean, our theory is that what you're  presenting within a syllabus is the best work possible out there. And that's not necessarily  the reason why you're citing something within an article or a book. Let's be honest about  that. So that's one thing that we want to take a look at. Are there differing ways to  take a look at that? What are the forms of scholarship that are included and by whom?  What roles do the students play in the scholarly conversation? And as I already mentioned earlier,  I'm really interested in engagement indicators and how much time is something is being spent  and whether we can learn something about context within that. So how might this then encourage  those values that I started out with? Just going to use some examples as you think about  syllabi and how they're created. A question like, am I including scholars and works from  all backgrounds? That would be equity. So really thinking about that and that would start to  encourage the value of equity. Have I made this syllabus open to other instructors? That  would be a kind of openness. Do I have a code of conduct that encourages kindness? Is this  the best word possible? And how can I encourage engagement? And there are many, many, many  questions that I'm sure all of you can think about as well that might fit into things that  we could look at that then once we can uncover them and unpack them and really think about  how you expose them, that might then encourage people to be more intentional for these kinds  of values as they're creating their syllabus. So this is the work that started all that  off. We were at the Triangle SCI last October. I cooked this all up. And we've gone on from  then to continue to think about this. So we'd love to hear from you. We'd love to engage  with you at all times. We're only just beginning, as I said, next week, as our very first workshop  to take a look at this. So please follow us and engage with us. We have a blog where we're  blogging as much as possible. Please come and comment. We welcome emails. And we really,  really want to engage with the community. Thank you. Thank you, Rebecca, for your philosophical  side of things. Our next speaker will be Martin Perk from the University of British Columbia.  And he'll be telling us about research evaluation in Canada. So I'll say I'm the chemist raising  from the sociology and the philosophy, or maybe raising lowering, I'm not sure. But I am from  a very different cohort from most of the rest of us. So I'm part of the University Administration  at a very large research university, number two in Canada, and have a very different perspective  on metrics and particularly altmetrics. So I thought I'd talk a little bit about what's  happening around us in Canada, a sense of what we're up against here. So it is all about  impact. We worry about that a lot at universities. Our stakeholders demand nowadays more than  ever to know, you know, what have we done lately and with all the taxpayers' funding  that we use in the process of our research. Somebody said earlier that we're also, even  in the competitive process of securing research funding, it's become a lot more important  to show impact and potential knowledge mobilization. So it's a competitive aspect as much as anything.  But there are obstacles to using altmetrics and metrics in general. And I'm part of that  administrative decision-making body that decides what metrics we worry about at UBC and what  systems and products that we implement to help us. One of our, I would say, main obstacles  is in Canada we do not have a REF or an ERA. So we don't have any source of block funding,  which in other jurisdictions forces universities to be very mindful of things like metrics  and altmetrics. So we don't have the same economic drivers towards using metrics and  systems as other jurisdictions. Now there are some funders. The good news is there are  some funders that are looking at specifically metrics. CFI, our Canada Foundation for Innovation,  is a big infrastructure funder. And they are quite big in metrics. But many of the other  funders, not so much, are health research funders, EHR, Canadian Institutes of Health  Research, are looking at things like research fish. But it's very early days for us. And  we have not really got a head around how the funder is going to hold us accountable in  terms of research metrics. So that's the basic context. Lack of block funding, it's all project  based. Everything is project based here. And the funders are not really pushing us too  hard yet on the metric side of things. The provincial governments do push us fairly  hard. And so we do produce reports on a yearly basis that involve quite a bit of bibliometrics  to this point. But I think we're starting to think about altmetrics. This is a look  at the funding landscape. Basically, a list of funders in this country. CFI is a research  infrastructure, NSERC, Science and Engineering, CHR, Health Research, SHERC, Arts, Social  Science Humanities. And we've got various other niche programs and chair programs. So  I think we used to be a very well funded jurisdiction. We've lost quite a bit of ground over the  last 10 years. We used to be the top funded per capita in university research. I think  we're sort of middle of the pack in OECD now. So instead of talking about cool metrics  and cool systems, I think what we as administrators look at more is, you know, what's the compelling  business case? Why do we need systems and tools? So I'll put my technology poll hat  on instead of technology posts. And these are the kind of things that we worry about  as university leaders. You know, we need to convince people, our stakeholders of the  research ROI, that's still very difficult to do. And I think the tools that help metrics  developing and plumb and other groups are really helping us get to that. But it's still  early days. Research as an economic development activity is huge for us, especially at the  provincial level. I'm sure that's true for most. Collaboration, what are the ideal impacts?  How do we build interdisciplinary teams? Who's missing from the collaboration? Who should  we be talking with? How do we convince funders that we're the best in the business? And that  other group in some other part of the world is not quite as good.  Social science humanities is a big aspect for us because UBC is very good at that. But  again, because of the lack of indicators, it's difficult to show just how good UBC really  is. So that's a big issue for us. And then there's all sorts of other things like growing  internationalization is considered to be a good thing. We really do worry about THE  times higher ed and ARWU. We pretend we don't, but at the university administration, you  know, we worry about that a great deal. And of course, if you look at the KPIs involved  in things like THE and ARWU, it's highly quantitative. There's a bit of reputational  stuff, but there certainly isn't too much in the way of metrics. And those are the things  that drive us. You know, people ask us, why are we going up or down in the various THE  and ARWU? The other thing in Canada that really is driving us lately in terms of competitiveness,  and I know this is true for many other parts of the world too, is low success rates in  agencies. So we're looking at success rates as low as I think the latest CHR competition,  project competition. We could be looking at less than 10%. So this really has become a  very dire competitive issue being able to outmaneuver and out. Okay, so these are the  big questions then. And these are the questions that we need tools to answer. And other uses  are really not a compelling business case. Just because something's cool, it tells us  something we don't already know. Those aren't the compelling business cases for institutions.  So these are kind of some of the things that big questions that we're trying to ask. You  know, how do we move up the global ranking scale? In the future, you know, how do we,  who do we reward for great research? And how do we define research excellence, especially  in the Shirk social science humanity areas? Who do we hire? We've got a new program that  was just launched, I would say, very opportunistically, the Canada 150 Chair with Brexit and with  Mr. Trump that we're trying to recruit some of the world's top researchers, bring them  to Canada. And so that's the launch, the 150 anniversary Canada. And then we all belong  to clubs, international clubs. UBC belongs to U21, amongst other. And so we want to  know, you know, how do we align with our international university partners? And strategically, what  are the best themes and best opportunities to work together? You know, what does this  look like? Well, this is a traditional look at the world map for Syval. We do use Syval  quite a bit. It's a pretty useful tool for telling us what we're good at, who else is  good at that, these different competencies. And I guess one of the things I'll mention  is that what attracts us to this particular tool is it does tend to predict new trends  and new research areas. And that's of great value to us. So this is the map of UBC and  it gives us a sense of who we're good at, who our collaborators are, who our competition  is and who we should hire, et cetera, et cetera.  Our metric, great tool, we've had a look at it. It does help us answer a couple of big  questions. One of those questions is, you know, what is competition up to in strategic areas  of interest to us? So for instance, we can do searches on things like Stanford and leukemia  and gives us an idea of what the big stories are and what sort of impact our competition  has in strategic areas of interest to us. And here's a kind of temporal look at that  study. Who's looking at the articles, what sort of reach do the articles have is a big  question for us as well. Now, looking at UBC, you know, who are big players, what sort of  social media attention do they get? Mark Amara is one of our biggest. He's big in cancer  and personalized medicine. So this, so our metrics gives us a sense of what our big guys  are up to, how competitive they are with the other big guys and what sort of attention  we're getting. I think we're running into lunchtime here, so I'm just going to speed  things up a little bit. I think the other thing is when you delve into the metric reports,  one of the things that's really important to us too is quality. So how do we define  research excellence? We can look at papers and how they compare to other articles in  things like nature, articles of a similar age. So there's some really important tools  that tell us about quality and research excellence that we think are important alongside tools  like SyVal. And again, you know, who's using our research and who should we be thinking  about collaborating with are important aspects of our metric. So just a couple of some quick  conclusions. I think evaluation of research in Canada is fairly spotty for some of the  reasons I mentioned, the block funding and the lack of sort of drive from our funders.  The current tools are good, but need to be better. I think we're using them at UBC some  to, from a kind of an interest aspect, others because we really need them badly, but there's  certainly improvements, more flexibility, better integration, more scope of metrics,  better social science and humanities uptake. I think there's aspects of clean cleansing  data. The data's not always all that clean. There's big issues of need for comprehensive  data sets. What drives research as bonkers is when they look at data sets, when they  look at metrics and their work doesn't appear in them. And we need more nuanced impact metrics.  Snowball plus there's other sets of research metrics out there, but we need to do more  work and develop scorecards of metrics. It really makes sense for all of us. I don't  think anyone has any real good ideas on how to get that right. And certainly Times Higher  Ed and ARWU does not have the answers. I don't think any of the systems lie in. There's a  whole list of different global ranking systems. None of them seem to really provide a very  nuanced ranking of how excellent we all are. And just finally before lunch, I would say  what we really need to do is kind of transform institutional use of the tools from kind of  interesting novel cool to mission critical. And that just hasn't happened. The tables I  sit at, we just haven't got to that point at all. So that's a bit of a roundup on the  Canadian context.  Thank you, Martin. Very insightful. So we have time for questions or question? Anybody?  Okay.  Hi there. Just quickly, I find that within our institution, I'm still seeing a cultural  perspective whereby people want the most basic metrics reported on a sort of a fiscal perspective.  A quarterly, just to meet accountability. There's a lot of that there because as you  pointed out, a lot of people are dabblers, right? They're almost weekend warriors when  it comes to the use of metrics. People in leadership positions or individuals reporting  their own data. What would you suggest or what's worked for you, particularly Martin,  I think you use a lot of this data more strategically for planning to move those various agents,  funders, like internal foundations, medical leadership, that sort of thing, to be able  to use it more strategically and more for planning?  So I think one of the biggest obstacles we have is integration. At the moment, you know,  we've got pockets of data and pockets of metrics all over the place. And so at UBC,  we literally have a senior committee that has representatives from all different parts  of the university voting on literally on what metrics we're going to utilize. And it's  so painful trying to pull it together and actually deliver an annual report, say, of  metrics. And so that seems to be the biggest obstacle. And I think the other big obstacle,  once you leave the university, is that we all know our researchers have a lot of administrative  tasks and administrative burden, right? I saw, I know, FTP and CAGA in the US estimates  that researchers spend 42% of their day doing administration. So for us, that's a big issue.  And the funder realizes that. So the funder comes to UBC and says, we think you should  fill in research fish profiles of all your researchers. And we say, well, we've already,  you know, input that data, that data exists in different places, and they want us to duplicate  the input. So until we get better integration of data sets, and we can minimize the administrative  burden, I don't think anyone's going to sign up for it. I think those are the biggest key  issues for us.  Okay.
4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_0_47.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_48_167.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_168_347.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_348_2327.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_2328_2915.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_2916_3263.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_3264_3503.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_3504_11699.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_11700_20831.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_20832_20879.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_20880_20927.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_20928_20999.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_21000_21911.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_21912_22703.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_22704_22799.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_22800_22859.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_22860_23207.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_23208_23339.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_23340_23471.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_23472_23855.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_23856_23915.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_23916_23999.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24000_24191.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24192_24275.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24276_24491.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24492_24563.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24564_24635.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24636_24803.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24804_24935.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_24936_25187.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_25188_25391.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_25392_25763.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_25764_25823.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_25824_26039.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_26040_26267.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_26268_26423.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_26424_26591.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_26592_26663.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_26664_26807.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_26808_27227.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_27228_27563.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_27564_27683.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_27684_27743.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_27744_27959.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_27960_28163.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_28164_28451.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_28452_28535.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_28536_28703.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_28704_28751.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_28752_29159.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_29160_29447.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_29448_29999.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30000_30203.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30204_30419.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30420_30539.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30540_30599.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30600_30923.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30924_30995.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_30996_31043.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_31044_31091.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_31092_31139.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_31140_31763.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_31764_32135.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_32136_32651.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_32652_32711.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_32712_32807.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_32808_32879.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_32880_32927.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_32928_33215.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_33216_35483.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_35484_35819.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_35820_36119.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_36120_36935.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_36936_37499.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_37500_37715.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_37716_37811.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_37812_41807.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_41808_41879.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_41880_41927.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_41928_42503.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_42504_43307.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_43308_43523.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_43524_43583.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_43584_46007.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_46008_46427.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_46428_46535.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_46536_46619.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_46620_46667.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_46668_46787.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_46788_50003.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_50004_50243.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_50244_51263.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_51264_52727.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_52728_52835.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_52836_53339.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_53340_68099.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_68100_68531.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_68532_80915.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_80916_83759.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_83760_83903.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_83904_83975.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_83976_85139.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_85140_100307.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_100308_100583.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_100584_101579.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_101580_102179.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_102180_102335.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_102336_102779.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_102780_103007.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_103008_103739.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_103740_122915.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_122916_123059.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_123060_128603.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_128604_138803.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_138804_139439.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139440_139511.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139512_139559.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139560_139631.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139632_139679.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139680_139751.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139752_139799.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139800_139883.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139884_139931.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_139932_140135.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_140136_140231.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_140232_140507.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_140508_142127.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_142128_145871.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_145872_153443.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_153444_155807.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_155808_159047.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159048_159119.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159120_159539.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159540_159599.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159600_159731.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159732_159911.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159912_159959.png 4_AM_Day_1__Altmetrics__Now_and_next_and_Altmetrics_in_Research_Evaluation_159960_163050.png