Keynote - Beyond Bibliometrics
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kirk, Martin
Chair: Cat Williams (Altmetric, UK). Beyond Bibliometrics: Martin Kirk (King’s College London, UK)
 Good? Yes, I can hear it now. Fantastic. Thank you. Thanks for that introduction. It's great to be here. I feel like we're friends and colleagues. I've been to quite a few of the AM conferences. I've always really very much enjoyed them. And my job at Kings really is to make sure that we've got the best tools and systems. So I mean, I know most of the vendors and Kings, we have pure. We have most recently out metrics. So we're a pretty happenin' kind of place. And it was interesting what John said about sterling transforming young people. Because 35 years ago, which is probably the last time I was at the University of Sterling, it transformed me. I didn't even study here. I was a serious chemistry student, an undergraduate in Edinburgh, where things were kind of serious. And sterling was where you came for a fun time. And to meet really interesting people. So that's what it's known for, is quirky, different, and transformational. That's for sure. Anyway, to serious things, I need to find my squidger. Let's see if the squidger works. All right. So here's what I do. I've got a kind of interesting job at Kings, but I really am where the rubber hits the road. And as the function lead, I provide all of the kind of central support for research at Kings College. So that includes the typical kind of research and bin, awards management, stuff, commercialization, clinical trials. We do enormous amount of clinical trials at Kings College. It's the post-grad scholarships, the compliance, all of the animal, human ethics, GDPR, research integrity, research governance. The core facility stuff, and all the research development stuff. And of course, research developments, the important competitive component that we're all very interested in. So there's about 600 people covering in my portfolio, covering every aspect of central support for research. So I know the research bit, but I'm not on the academic side. I really am trying to increase our performance at Kings as one of the top Russell University in the UK. So that's me. Cat asked me to talk about how do we use altmetrics at Kings. And I thought, before I do that, I think there's a bit of history that we have to look at. There's a transformation going on in the university world that I think is really important that we understand. Now, maybe many of you understand it, but it's a topic of much debate around the world. And of course, governments want to put money into a transforming university that does the right thing for society. So this is kind of what I wanted to talk a bit about today. The evolution of university through the ages won't talk too much about that. The changing role of the university, I think that's important. The evolution and diversity of the research provider, which isn't always a university. The transformation of metrics needed to describe these different pieces that we're trying to demonstrate our ROI to the government on. And then the bit that Cat asked for how and who uses altmetrics at universities. And I'll just do a few case studies. And I'll just say up front, thanks to Jurgen at Altmetrics who helped us and actually really did such a good job in persuading the administration at Kings. That was a good tool. And how he did that was he made a very sort of live, here's what Kings looks like through the altmetrics lenses. And it was very useful case studies that brought it really alive for us at Kings. So here's the history. I'm not sure if you know, I didn't really know. I knew about Bologna, but between Bologna and today, I didn't know the details. So this is what a history of university thousand year, actually even more than a thousand year looks like. So the first sort of indications of anything like a university were in the medieval ages. And these were studio where we're set up to educate clerks and monks. Ancient history. I suppose the first inkling of what today's university looked like was in Celerno in the ninth century. And it was a medical school that was created to teach physicians. Or in those days, I think they were called surgeon barbers or barber surgeons. There wasn't physicians. I think the first time there was a true university was in Bologna. I think we all agree that they were the first in the game in the 11th century. And they were very focused on canon law and civil law. And it was cool to be in Bologna for Iyama at that very university. I don't know how you felt. I thought that was a real exciting privilege to be there. Ben was there as well. And I recognize some of the rest of you were there too. Okay, moving forward a little bit to the 12th century, then a couple of key universities were set up. Paris and Oxford. Very focused on theology. And then theology and I was going to say dogma. That might be a bit insinuative. So I won't say that. And then between the 12th and 18th century, I think there was some development. And then it ended up with what's often called the Scottish enlightened version of the university. And that's the kind of liberal arts. Personified Rome, Heidelbergs and Andrews, maybe Glasgow as well. Offering a core curriculum. Liberal arts, grammar, logic, rhetoric, geometry, arithmetic, astronomy and music. It's quite a nice combination. And I think most recently we all recognized the sort of German model of university which 19th century typified University of Berlin. And that's where experimentation and the scientific method sort of replaced conjecture and sort of theological dogma. So that's kind of the beginning of the new age of history. So that's just kind of an evolution through the ages on where we've got to today roughly. All right, so where does that leave us looking at sort of more modern history? I think the old model was definitely a university as a womb, as a crucible of societal knowledge and new discoveries, right? That's pretty much where some of us still are, but where history got us to. But I think that role has really changed. We're now seen as the sort of fountain of high quality personnel and there's a very strong training agenda that the universities have been given. And that's a risky one because I think, and we'll talk about that a little bit more and I'm interested to hear from you. But I mean when I was a young lad there was a very small number of people got to go to university. It was a small percentage and that has changed completely. You know, I think in those days it was people that were just interested in learning and it was an opportunity to become a very rounded, educated member of society. That has changed. I think we present ourselves now as much more of a mass training kind of opportunity. Then there's a role in as part of the economic development machine and that's huge for us. So that's sort of messing around with the balance between pure and applied research and dabbling with industry and the metrics around spin-offs and licensing and all the rest of it. And then I think the most recent one, probably the last 10 years or so, I think the commercialization one and the economic development one started at the end of the Second World War, but really heated up about 20 years ago where there was a lot of money put into our industry liaison offices and there was a real ramp up of commercialization and spin-offs. The most recent one, probably the last 10 years, is kind of broader societal impact. That's going beyond kind of just economic return. That's how do we influence public policy and how do we improve public health and go beyond just making money. So that's the most recent one. But the question really is can we satisfy all those roles? Those are big roles in society. And do we all need to do that? I think another question is we have elite universities, we've got medium-sized universities, we've got small regionally relevant universities and everything in between. So that's the role. And I think most of us are some combination of all of these, the crucible of knowledge, the fountain of HQP, the engine of the economy and the societal impact agenda. And I think in some ways what you could do is sort of classify a university by weighting these four different aspects. But I will say one thing about impact that's really important that John kind of almost hinted at. So in the REF, and I think many of us think about impact as well that's citation impact, right? That's the definition of research excellence and impact. But interestingly, the Australians did a pilot where they tried to create a similar to REF type of funding scheme. But instead of having peer review as we do in the UK to judge how much impact our research is having, the Australians decided to go with non-peer review. And so they had industry, they had government, they had just taxpayers. And the one thing I'll tell you that is really interesting in that pilot that they did was that our version of impact is completely different from general societies. And so we can't even agree on what positive impact is when it comes to research. I think that's really important to understand as we think about the metrics that define success, right? Okay, so let me tell you a bit more history to give you a few numbers and give you an idea of the evolution. This is just the number of universities in the world and you can see there was very few in the 1400s, slowly ramping up 1625 to 73, the end of the Second World War there was 500, today there's 10,000. So we have grown like topsy all over the world. Some massive increase in number of universities. And of course we have our global ranking systems and we're all desperate to get up those global ranking systems, Shanghai and Times Higher Ed. And I think if you look at those recently and we worry about them a lot, some countries really are doing a great job of whipping up those ranking systems and we'll talk about that a little bit further. Here's another one that's really important is how much money your government spends on R&D in universities as opposed to in other places. So GERD is the total expenditure on R&D but that's made up of HIRD which is higher ed spending, BIRD which is business expenditure on R&D and then GERD, not GERD, GOVERD which is government spending on R&D. But it's important for us because the more money that obviously we spend on HIRD, that makes us busier. But what you can see here is that countries like China, Japan and US do a very small proportion of their research in universities of the total spend. That's very significant. On the other hand, the UK and Scandinavia do a third of their research in universities, third of their total research in universities. Now the sort of pros and cons, I suppose the Scandinavians would argue and the Brits would argue, well, you know, we're going to do all the disruptive research and make the big discoveries in universities not in industry. I don't think that's necessarily true but that's one argument. On the other hand, I think in the US they would say, well, because we're working so closely with industry, the discoveries are closer to the market and the more useful for the economy and for society. So it's kind of a dual argument there. Sometimes though, we don't have enough industry and we're kind of forced to do lots more HIRD almost as a proxy for missing industry R&D. So for candidates like that, we don't have a lot of nascent R&D capacity in industry. So we have no choice really. All right. So why have universities transformed? Well, we're educating more and more students. It used to be a sort of a 5% of the population and now it's 50 or somewhere between 25 and 50. I know Tony Blair said 50, right? I think 50 is way too much. But most Western countries are sort of between 25 and 50% of all young people are expected to get into university and have some sort of tertiary education. Fulfilling all the roles we've talked to, educator, creator of knowledge, worker trainer, engineer economy, provider of broader societal impact. But the most important thing is we're soaking up more and more of the GDP and we have to justify that. We're costing more, we're promising more, and we have to show up with what the ROI is and all these different roles. That's really where the pressure is. All right. So let's talk about the research providers because I think this is a really important factor as well. If you try and classify them into a taxonomy, there's four or five different versions. And I say providers because it's not always universities. So we got the comprehensive university. And most universities, I suppose, fall into this category where they teach a massive number of undergrads who pay the bills, really, especially in the UK. So we have a lower number of grad students, lots of researchers, but covering a huge research landscape needed to provide all of the courses to teach all this massive number of undergrads in many different disciplines. Right now, what that means is that there'll be a low number of areas of prominence. So if you use the SyVel lens and you look at those universities, what you'll find is a relatively low number of areas of prominence because there's such a spread of research expertise, right? A bit of a downside. But a high degree of curiosity-driven research. And then you've got the elite research-intensive universities. They have lower undergrad numbers, high number of grad students, but much higher areas of research prominence. And when you do the SyVel, look at this, you'll see that these elite universities have sort of 80% of the research is in areas where they are really world-class, whereas the comprehensive universities, it's somewhere usually between about 10 and 20%. So that's a major difference, is critical mass. And nowadays, research is a team sport, and this critical mass turns out to be really, really important if you're going to have world impact. I don't think the model of the one PI with the one grant to a world-class research still exists, but I think the team sport piece is taken over. Then the third one I want to mention is the elite research provider. And these are not universities, but they're sort of dominated by universities. Things like the Crick Institute, the newly formed Turing Institute, the Catapults in the UK. I'm not sure what these similar things look like in other countries, but they're definitely dominated by the university, they're created by governments, and they're supposed to supercharge our research activities, often with partnership, you know, the health authorities and industry, but they're special crucibles unencumbered by undergraduate education. And then the fourth one I want to mention is a very special one, and the Germans really invented and are very, very good at this. This is the Fraunhofer Institute, where they're not really dominated by universities at all. In fact, I would say they're more dominated by industry, and they're largely funded by industry, and they're largely contract research. So these are research institutes that really focus on what are the problems that industry and governments are trying to solve. The most pure form of solving the exact societal problems. Just a bit more detail on Fraunhofer, so this is about huge scale too. 72 themed institutes across the world, 26,000 people, 2.6 billion euro budget, and interestingly of that number, 2.2 is contract research, so 2.2 directed research, that's huge. To give you an example in the UK, I mean most universities are doing about 5% to 10% industry funding, and some of that will be contract research, so completely different scale of contract research. Alright, so that's what the models, and why is industry so important? Well, other than promoting universities as sort of engines of the economy, even from a very selfish PI perspective, when you have co-authored papers with industry, your impact goes way up, citation impact goes way up, and impact, and social media attention, everything goes way up, so you get huge bang for your buck. But again at Kings, that's about 5% of our publications are co-published with industry, it's a small number. And you can see the sole authorship is way at the bottom in terms of impact. So this is outputs in the top 1% citation percentile, and again the corporate ones at the top, sole authorship on the bottom as you might be able to predict. Alright, so let's talk about metrics now, so now we get into the metrics piece. I think most of us sort of have an idea that citation impact has done some good stuff, and has been a reasonable proxy for research excellence, but not for all of the research spectrum. So it doesn't work very well for a lot of social scientists or humanists, and in fact it doesn't even see them because they don't publish in traditional scholarly journals. The fountain of HQP, you know this should be really easy, but we don't do a very good job of it, and in fact when we think about our graduates, we don't really care that much about them until they become older and richer, and then we think of them as alumni and we phone them up and say, you know, how about giving us some cash? And it's big business, I mean at UBC we raised about $200 million a year from our friendly alumni and rich donors and stuff, so it's big stuff. The economic engine bit, I think we've got kind of, we know what the metrics are, but the societal impact is the one that really, really is difficult, and we're really going into this area hard. So at Kings, I mean we spent $30 million quid recently to buy this science gallery that's right beside Guy's, it's like a fabulous monument to, you know, it's kind of a gallery, and what we do is every three months we have a new sort of research theme, and it's free, there's a big sign saying come and visit for free, there's a cool coffee shop there, as I say it's thousands of people walk by it's right beside Guy's Hospital and right beside the shard. A different research theme every three months, the last one was dark matter, but it's not just about science, it's about art, so it's about the science and art of around dark matter at Kings. The one before that was called Hooked, and it was about addictions, the science of, you know, Kings is big into addiction research, and it was about, you know, art of people who are addicted to. So it's great, and we even have a VP called the VP London, who's in charge of public engagement, so we spend huge amounts of money on this, Baroness Deborah Bull is our VP London, who, you know, runs a science gallery and all of our kind of public engagement, outward facing stuff, so it really is very big for us, but how do we measure this? We absolutely fail to measure effectively to this point, and I think this is where the big opportunity for altmetrics really comes into play. So the bibliometrics works fine for some of the crucible and knowledge model, but absolutely not for the societal impact, and so, you know, this is where we look at the transformation of how we use metrics. Okay, so here's what Kat asked, you know, who's using altmetrics in universities? Well, as I say, I represent one piece of that, and that's kind of where the rubber hits the road at universities in the interface between researchers and the real world, researchers and funders, researchers and industry. And so the different bits of what we do are things like, you know, our Vice President Research, Reza Rezawe, worries about things like what's our research performance, what's our strategy, who do we partner with? Those are the big sort of executive research questions, and they are informed by, you know, our altmetric and our Syval view of the world. People like us, the research administrators, we worry about research excellent metrics. REF is absolutely huge, and I'm going to show a few slides further on that really show how valuable our metrics is in doing a better job of REF. So the way REF works for non-Britz is that there's a huge amount of block funding available, and we have to do a submission, and that submission means that we have to present the best papers for each academic at the institution, and it's all about impact, they get one, two, three, or four star. And then we have to write impact statements for a limited number of our researchers, and again, there's a sort of a peer review of all of this stuff, and at the end we get a big check, and we get a bigger check if we're more competitive than our peer universities. The last time I think we did really, really well, we're not quite sure why, but this year, this round we're going to do really well, and I think it will be largely because we really worked hard at identifying our impact and evidencing our impact, and we've gone beyond the usual kind of idea of field normalised citation impact and stuff like that. Librarians, I mean, you know better than I do how they use altmetrics, but I will say at Kings, our library system, it's very important because they are almost holders and protectors of the systems and tools and trainers of people, so when it comes to things like our effectiveness at using SyVal and altmetrics, the librarians play a huge role in making sure that that sticks and works well at Kings. And then there's the traditional kind of, you know, your department heads that worry about things like assessing their PIs and who gets promotion and who gets tenure and that type of stuff, you know, the basic kind of performance management. And then the big one I would say that really is a huge opportunity for us is the PR and communications, which is really big. We want to be noticed, we want to build our reputation, and we do that by finding our way into the news and into important public fora. So that's a bit of a, you know, that's a bit of a sort of a mishmash of where the opportunity lies. I don't think Kings has taken full advantage of that. We just started subscribing fairly recently. We're sort of still getting ahead around what the value is. If you ask me what's the biggest win for us, it's absolutely ref at the moment. That's where the value is. That's how I sold it to the decision makers that provide the big money. Actually, it wasn't, I had to convince myself it was a good idea. So this is what we do, identify research impact, evidencing research impact for ref. I think importantly, providing qualitative context to the bibliometric analysis of our outputs at Kings is important as well. Research strategy. We worry about partners, you know. We universities join clubs. We have big global clubs of universities in the UK. The clubs called the Russell Group, and it's just the biggest and best. But internationally, the clubs like the U21, UBC was a member of the U21. There's one called the Guild that we're a member of at Kings. It costs a lot of money to do this, and you want to make those decisions about which club to align with sensibly, so that there's maximum opportunity for collaboration as you invest this money in partnering with your club members. Assuring and just fine open access, talk about that a little bit more. And evaluating, obviously, the non-STAM research outputs is very important for us as well. So we know what the pros and cons are. You know, the bibliometrics, it's a great direct measure, or a good proxy for research excellence is very important for global rankings. But it only sees certain parts of the research outputs, the research spectrum. I think the alt-metrics is an opportunity for a more nuanced approach to measure and impact. It's a more direct measure of impact. It sees research activities in social sciences and humanities. They're really important. I mean, UBC, our social scientists were higher up the world ranking than our medical researchers. And so this is really important for us. But, you know, on the con side, it does measure negative attention as well as positive. And I think the biggest chink in the armor is that, you know, unlike citation impact, sometimes the datasets are not comprehensive, and so it misses stuff. And that's because things like mentions in policy documents, it's about the scrapability of those and accessibility of those documents. But you know what research is? If their paper isn't picked up by a particular system, the researcher immediately thinks, well, this is a crap system. It's missed me out. So that is a dangerous vulnerability of alt-metrics that I just mentioned as obvious. So this is what it's all about. We all know these different aspects of alt-metrics. I should mention that PlumEx does slightly different stuff. And, you know, I think these systems are very complementary. And I think it's not a question of either or. I think it's a question of just making sure that you've got access to all of them, or most of them, or as many as you can afford, and make the best possible use out of them. So here's a few case studies. And I wanted to convince myself, and this is the work of Jürgen, that our top researchers from a citation impact perspective were also top researchers of pure alt-metric look at the world. And so Seb is one of our top researchers. He's a medical imaging guy. I think this is a three-year snapshot, 400 outputs. 240 of them are picked up in the alt-metric view of the world. 1800 Twitter, 48 patent, 160 news mentions. You know, shows up loud and clear in both systems, and that's good to know. Here's some of his top spotted papers, real-time imaging of single neuronal cell apoptosis in patients with glaucoma. It's important stuff. But he does show up really well in both lenses. I think that was important for us. This is Reza. This is our vice president of research. Reza is a pediatric cardiologist who treats very young kids, and he's trying to improve the scanning capability so that he can do a better job of diagnosing and treating congenital heart diseases in very young kids. Interestingly, I asked him where his drive to do this came from, and he said, well, I have a daughter, Poppy, who has a congenital heart disease, and he thought this was worth going after. So that's his personal story, which I thought was very inspiring. And again, he shows up in the alt-metrics view of the world as well. Thanks, Jurgen. All right. Let me just sort of talk a bit about when you sort of compare and contrast, right? This is work that we did at King's, looking at 144,000 King's journal articles. And so on the x-axis, you've got the alt-metric view of the world, and on the y-axis, you've got the citation impact view of the world. Now, you look at that and go, it's a bit messy. It is. But, you know, there's a correlation. I think the most important thing, though, more important than correlation, is that if you want to pick up all of your impact and attention, right, the citation impact is going to give you this stuff. Both is this stuff, so you've got some correlation there, but what about all this stuff? You know, this is stuff that you're going to miss if you don't look through the alt-metric lens. And we can't afford to miss this. We can't afford to miss it for RAF. We can't afford to miss it for our outward-facing public engagement strategy. So this is really important. I think this is an important slide, and not to be forgotten. Okay, life and death is all about publishing for researchers and where they publish. And so everyone wants to publish in the highest impact journals. And of course, things like the Shanghai Global Ranking, one of their key metrics is how many papers do you have in Nature and Science. So, you know, that's how serious this stuff is. But I just wanted to give you a couple of things that we look at. So here's different publishers and different journals, and this is the five top publishers in psychology and cognitive science. And you can see that they vary very widely in terms of news mentions per article. Now, why is that important? Isn't it all about citation impact? Well, no. And there's lots of reasons that this is important. When it comes to news mentions, you know, things like donors, publish donors, and for performance reasons as well, it's really important to know what sort of attention you're going to get in different publications. You can then put that information beside your citation impact and make clever decisions on where you need to publish. If you're lucky enough to have that kind of choice. Same kind of thing. This is the five best journals in psychology. You can see the journal impact factors, which of course we're never supposed to even look at, but we all do. And what you can see there, though, interestingly, is the Almetric Attention Score per article is almost 108 degrees apart. The top journal in terms of impact factor is the lowest on the Almetric Attention Score. So interesting what you can drive out of this kind of analysis. It's not all obvious that the top GIF is the top place to get attention outside of the scholarly community. And as I started my conversation today, you know, scholarly impact isn't the same as real impact, or when I say real impact, what society thinks is real impact. Again, here's more of the same. This is the top journals in psychology and cognitive sciences, average news mentions. And so again, this gives you the massive scale of difference between the different journals and how likely they are to get picked up and picked up by the media. This is a super important one as well. So when I think about public engagement, I think this is the scopus topic at King's receiving the most news over the last three months. So this is kind of what's hot at King's. Why is this important? This is what people are talking about and want to know about and need to know about. And so that informs a lot of different things at King's. Like, you know, what's our externally facing PR strategy for this month and what sort of externally engagement activities, what kind of lectures are we going to be doing? You know, if we want to be relevant and we want to be upfront, this informs that exact strategy. Baroness Bull wants to know what this looks like and what to react to. And interestingly, some of these, like for instance, tobacco, you know, that doesn't show up in the citation impact because it's not stuff that we publish on, but the public are sure very interested in it. So there are areas that really, you know, diverge when we look at our traditional scholarly assessment of quality. This is policy mentions. So again, you know, why is this important? Well, we've got a policy institute that needs to know what areas of research that we're doing at King's are the most important. I'll mention another thing, and I shouldn't really say this, but we do worry about it, is that there's quite a bit of research recently on funding patterns. And it turns out that your success rates are way higher if you are asking for money in sort of trendy areas of research. Now, that's probably no surprise to anyone, but you know, it's important to know what those trendy areas are. So you might ask, well, how important is it really to be in the news? Why don't we just ignore it? I don't think we can for many different reasons. If you think about Times Higher Ed ranking as an example, a huge part of that is the subjective global survey. They send out 300,000 surveys to researchers all over the world who publish in their journals and ask questions about, you know, what you're ranking the best research in the best teaching universities in the world. Now, I've talked to some of those researchers who fill in that survey and I've said, you know, what do you think about? How do you make your decision on the best universities in the world? Well, there's three things. Usually they've had a personal experience. You know, I was a postdoc at Oxford, wonderful university, and of course I'm biased. I think that's a great place. There's history and prestige. Universities like King's who've been around for a while, and we have. I think our medical faculty began, you know, 900 years, a long, long time ago. So we get some free, free prestige, free points for being old and prestigious. And then the third one that's really important always is said is you are in the news. You know, we hear about you. You're obviously doing relevant research. Society cares about it and that's important. So it's kind of those three things. So I think absolutely it's important. I'm getting close to the end of my time here, but I'm almost finished. Open access. This is really important for us. It's coming down as regulated. Plan S has been delayed by Europe, but it is coming. The reason it's important is that the publishers charge us twice for a lot of stuff. You know, they're charges for a subscription and then charges for an APC to bring a piece of research outside the paywall. And so we have to make different difficult decisions about where we publish, whether we can afford to publish in certain journals. But this is just so we have to convince ourselves and convince our funders that they should fund APCs so that we can do this open access stuff. And you can see the difference between closed access and open access is huge. It's a factor of three for alt metric attention. It's a factor of five for news mentions, factor of four for blog mentions, you know. So open access, we can demonstrate the value of that if we're not doing open access, we're not being noticed. Similarly here, this is the field normalised citation impact. This is the funder. So this is the welcome trust funding. And what you can see here is that the research funded by welcome trust gets a lot more attention, gets a lot higher citation impact, attracts a lot better science compared to non welcome trust funders. Again, I can use this and say researchers, you should be applying for welcome. I know it's hard to get, but you know, it's going to be worth it. And by the way, I'll help you get it with various research development activities. This is a side by side comparison of field weighted citation impact and alt metric attention scores on the same issue, welcome trust funding. And you can see two things, big difference between welcome trust and non welcome trust. But you can see nice sort of lineup of both systems, alt metrics and citation impact. I think this is the last one. When we look at ref, like I say, an important thing for us is to be able to predict how good our papers are in terms of one star, two star, three star, four star. We want everything we submit to be four star. We use citation impact. We use a bit of internal peer review and we also look at the scores here. Now you can say, wow, that's, you know, there's a lot of overlap. Yes, there is. But as one of a few different filters to put our outputs that we're considering for ref submission, you know, this makes sense to look at. All right. So what have we learned? And I'm a minute early, I think the university models evolving, absolutely evolving. And it really is changing the way we use KPIs and the KPIs that we need to use. I think I've shown a little bit that our metrics is highly complimentary to a traditional. It does the same kind of job and it does a different, very valuable job for us in universities. I think it's the impact agenda, which is very important in the UK, certainly in North America as well, absolutely demands these new ways of measuring impact and the R&D. Just can't be done with the traditional bibliometrics. And the last thing, you know, Outmetrics is used by and can be used by many parts of the institution for a very effective outcome. I think those are the main things I want to say. Thanks for listening. I hope you learned a little bit and certainly I know this is a diverse crowd of people. I don't know, are there research administration people? I know it's libraries and publishers, but yeah, okay, good. Well, it's good to see some colleagues in the same business. I hope the rest of you have learned a little bit about, you know, research administration and how university works from our perspective. Thanks very much for the opportunity. Thank you. Before we let you escape, any questions that anyone would like to ask? No, everyone wants coffee? Yeah. Martin, you refer to, I made a note of it, PR and communications as becoming a more important aspect of university life as well. How does that work out in King's College regarding bibliometrics feeding back into PR and communications? Normally, those are two different departments. Are they teaming up more than before? How's that going? So I'll tell you, interestingly, King's has made a big bet on a completely different model of how we run the university. It's called functional alignment. What that means is that there's a function lead that manages every aspect of a particular part of the institution. I'm research. Now, what that really means, though, and so communications is a function. What it means, though, is that, I mean, I have communication people in my area, but all of them have a strong direct relationship to the function lead in communications. So you could think of them as moles or spies or whatever in research. But we have a level of communication and cross-disciplinary kind of communication that allows that kind of thing to happen. So the communication in my group feed back into the central group with things like these metrics, all of this stuff I'm talking about. So we have sort of an extra layer of channeling built in through this functional alignment at King's. That's probably the best way to answer it. I think it's a big advantage. It's not easy to do, and it's costing us lots of money and time. There's a dip in, you know, the satisfaction surveys of our researchers because it takes time to get right. But that's kind of what we're aspiring to, and that's where we've seen that level of communication happening. Okay, I think that's very good to hear. You were also talking about this negative attention that may be something that may work out as negative towards altmetrics as well. How do you deal with that in King's College? You know, I actually remember a couple of years ago asking you about that. And I have to admit that although it's sort of the witch in the cupboard, I don't think it's that big a deal. But I seem to remember you and saying that what the industry was looking at was there are clever potential AI applications where you can screen out and filter sort of quite easily. It's not that difficult to do. So I think that's one of the traditional sort of here's why we don't like altmetrics, but I actually think it's almost a non-excuse nowadays. And you know, the way AI is going. So I've got AI applications that can sort of do research development, do very clever things. So I actually, I think that's almost not worth mentioning anymore to be honest with you. Hi, Joe from Crossref. Do you care about fake news articles? There's a lot of news articles out there. Is that not exactly the same question? I think it is. And I don't know if I know the right answer to that. And you know, how sophisticated is King's at spotting that? I don't think we are very sophisticated, but we do worry about a lot. And you know, if you expand the fake news to fake science, you know, one of the things we worry more, more, most about is research integrity. And you know, you've probably heard about some of the breaches that have happened in the recent past that really have tarnished university reputations and brought down presidents and vice president researchers. Karolinska, Paolo Macarena is probably a fairly recent one. So fake news and fake science, absolutely deadly for us. And I, you know, we worry about that a lot. Again, we've got good process in place to handle some of this stuff. But I don't think we've got a track record of successfully protecting our reputation in these areas.
 Good?  Yes, I can hear it now.  Fantastic.  Thank you.  Thanks for that introduction.  It's great to be here.  I feel like we're friends and colleagues.  I've been to quite a few of the AM conferences.  I've always really very much enjoyed them.  And my job at Kings really is to make sure that we've got the best tools and systems.  So I mean, I know most of the vendors and Kings, we have pure.  We have most recently out metrics.  So we're a pretty happenin' kind of place.  And it was interesting what John said about sterling transforming young people.  Because 35 years ago, which is probably the last time I was at the University of Sterling,  it transformed me.  I didn't even study here.  I was a serious chemistry student, an undergraduate in Edinburgh,  where things were kind of serious.  And sterling was where you came for a fun time.  And to meet really interesting people.  So that's what it's known for, is quirky, different, and transformational.  That's for sure.  Anyway, to serious things, I need to find my squidger.  Let's see if the squidger works.  All right.  So here's what I do.  I've got a kind of interesting job at Kings, but I really am where the rubber hits the road.  And as the function lead, I provide all of the kind of central support for research at Kings College.  So that includes the typical kind of research and bin, awards management, stuff, commercialization, clinical trials.  We do enormous amount of clinical trials at Kings College.  It's the post-grad scholarships, the compliance, all of the animal, human ethics, GDPR, research integrity, research governance.  The core facility stuff, and all the research development stuff.  And of course, research developments, the important competitive component that we're all very interested in.  So there's about 600 people covering in my portfolio, covering every aspect of central support for research.  So I know the research bit, but I'm not on the academic side.  I really am trying to increase our performance at Kings as one of the top Russell University in the UK.  So that's me.  Cat asked me to talk about how do we use altmetrics at Kings.  And I thought, before I do that, I think there's a bit of history that we have to look at.  There's a transformation going on in the university world that I think is really important that we understand.  Now, maybe many of you understand it, but it's a topic of much debate around the world.  And of course, governments want to put money into a transforming university that does the right thing for society.  So this is kind of what I wanted to talk a bit about today.  The evolution of university through the ages won't talk too much about that.  The changing role of the university, I think that's important.  The evolution and diversity of the research provider, which isn't always a university.  The transformation of metrics needed to describe these different pieces that we're trying to demonstrate our ROI to the government on.  And then the bit that Cat asked for how and who uses altmetrics at universities.  And I'll just do a few case studies.  And I'll just say up front, thanks to Jurgen at Altmetrics who helped us and actually really did such a good job in persuading the administration at Kings.  That was a good tool.  And how he did that was he made a very sort of live, here's what Kings looks like through the altmetrics lenses.  And it was very useful case studies that brought it really alive for us at Kings.  So here's the history.  I'm not sure if you know, I didn't really know.  I knew about Bologna, but between Bologna and today, I didn't know the details.  So this is what a history of university thousand year, actually even more than a thousand year looks like.  So the first sort of indications of anything like a university were in the medieval ages.  And these were studio where we're set up to educate clerks and monks.  Ancient history.  I suppose the first inkling of what today's university looked like was in Celerno in the ninth century.  And it was a medical school that was created to teach physicians.  Or in those days, I think they were called surgeon barbers or barber surgeons.  There wasn't physicians.  I think the first time there was a true university was in Bologna.  I think we all agree that they were the first in the game in the 11th century.  And they were very focused on canon law and civil law.  And it was cool to be in Bologna for Iyama at that very university.  I don't know how you felt.  I thought that was a real exciting privilege to be there.  Ben was there as well.  And I recognize some of the rest of you were there too.  Okay, moving forward a little bit to the 12th century, then a couple of key universities were set up.  Paris and Oxford.  Very focused on theology.  And then theology and I was going to say dogma.  That might be a bit insinuative.  So I won't say that.  And then between the 12th and 18th century, I think there was some development.  And then it ended up with what's often called the Scottish enlightened version of the university.  And that's the kind of liberal arts.  Personified Rome, Heidelbergs and Andrews, maybe Glasgow as well.  Offering a core curriculum.  Liberal arts, grammar, logic, rhetoric, geometry, arithmetic, astronomy and music.  It's quite a nice combination.  And I think most recently we all recognized the sort of German model of university which 19th century typified University of Berlin.  And that's where experimentation and the scientific method sort of replaced conjecture and sort of theological dogma.  So that's kind of the beginning of the new age of history.  So that's just kind of an evolution through the ages on where we've got to today roughly.  All right, so where does that leave us looking at sort of more modern history?  I think the old model was definitely a university as a womb, as a crucible of societal knowledge and new discoveries, right?  That's pretty much where some of us still are, but where history got us to.  But I think that role has really changed.  We're now seen as the sort of fountain of high quality personnel and there's a very strong training agenda that the universities have been given.  And that's a risky one because I think, and we'll talk about that a little bit more and I'm interested to hear from you.  But I mean when I was a young lad there was a very small number of people got to go to university.  It was a small percentage and that has changed completely.  You know, I think in those days it was people that were just interested in learning and it was an opportunity to become a very rounded, educated member of society.  That has changed. I think we present ourselves now as much more of a mass training kind of opportunity.  Then there's a role in as part of the economic development machine and that's huge for us.  So that's sort of messing around with the balance between pure and applied research and dabbling with industry and the metrics around spin-offs and licensing and all the rest of it.  And then I think the most recent one, probably the last 10 years or so, I think the commercialization one and the economic development one started at the end of the Second World War,  but really heated up about 20 years ago where there was a lot of money put into our industry liaison offices and there was a real ramp up of commercialization and spin-offs.  The most recent one, probably the last 10 years, is kind of broader societal impact.  That's going beyond kind of just economic return.  That's how do we influence public policy and how do we improve public health and go beyond just making money.  So that's the most recent one.  But the question really is can we satisfy all those roles?  Those are big roles in society.  And do we all need to do that?  I think another question is we have elite universities, we've got medium-sized universities, we've got small regionally relevant universities and everything in between.  So that's the role.  And I think most of us are some combination of all of these, the crucible of knowledge, the fountain of HQP, the engine of the economy and the societal impact agenda.  And I think in some ways what you could do is sort of classify a university by weighting these four different aspects.  But I will say one thing about impact that's really important that John kind of almost hinted at.  So in the REF, and I think many of us think about impact as well that's citation impact, right?  That's the definition of research excellence and impact.  But interestingly, the Australians did a pilot where they tried to create a similar to REF type of funding scheme.  But instead of having peer review as we do in the UK to judge how much impact our research is having,  the Australians decided to go with non-peer review.  And so they had industry, they had government, they had just taxpayers.  And the one thing I'll tell you that is really interesting in that pilot that they did was that our version of impact is completely different from general societies.  And so we can't even agree on what positive impact is when it comes to research.  I think that's really important to understand as we think about the metrics that define success, right?  Okay, so let me tell you a bit more history to give you a few numbers and give you an idea of the evolution.  This is just the number of universities in the world and you can see there was very few in the 1400s,  slowly ramping up 1625 to 73, the end of the Second World War there was 500, today there's 10,000.  So we have grown like topsy all over the world.  Some massive increase in number of universities.  And of course we have our global ranking systems and we're all desperate to get up those global ranking systems,  Shanghai and Times Higher Ed.  And I think if you look at those recently and we worry about them a lot,  some countries really are doing a great job of whipping up those ranking systems and we'll talk about that a little bit further.  Here's another one that's really important is how much money your government spends on R&D in universities as opposed to in other places.  So GERD is the total expenditure on R&D but that's made up of HIRD which is higher ed spending,  BIRD which is business expenditure on R&D and then GERD, not GERD,  GOVERD which is government spending on R&D.  But it's important for us because the more money that obviously we spend on HIRD, that makes us busier.  But what you can see here is that countries like China, Japan and US do a very small proportion of their research in universities of the total spend.  That's very significant.  On the other hand, the UK and Scandinavia do a third of their research in universities,  third of their total research in universities.  Now the sort of pros and cons, I suppose the Scandinavians would argue and the Brits would argue,  well, you know, we're going to do all the disruptive research and make the big discoveries in universities not in industry.  I don't think that's necessarily true but that's one argument.  On the other hand, I think in the US they would say, well, because we're working so closely with industry,  the discoveries are closer to the market and the more useful for the economy and for society.  So it's kind of a dual argument there.  Sometimes though, we don't have enough industry and we're kind of forced to do lots more HIRD almost as a proxy for missing industry R&D.  So for candidates like that, we don't have a lot of nascent R&D capacity in industry.  So we have no choice really.  All right.  So why have universities transformed?  Well, we're educating more and more students.  It used to be a sort of a 5% of the population and now it's 50 or somewhere between 25 and 50.  I know Tony Blair said 50, right?  I think 50 is way too much.  But most Western countries are sort of between 25 and 50% of all young people are expected to get into university  and have some sort of tertiary education.  Fulfilling all the roles we've talked to, educator, creator of knowledge, worker trainer,  engineer economy, provider of broader societal impact.  But the most important thing is we're soaking up more and more of the GDP and we have to justify that.  We're costing more, we're promising more, and we have to show up with what the ROI is and all these different roles.  That's really where the pressure is.  All right.  So let's talk about the research providers because I think this is a really important factor as well.  If you try and classify them into a taxonomy, there's four or five different versions.  And I say providers because it's not always universities.  So we got the comprehensive university.  And most universities, I suppose, fall into this category where they teach a massive number of undergrads  who pay the bills, really, especially in the UK.  So we have a lower number of grad students, lots of researchers,  but covering a huge research landscape needed to provide all of the courses to teach all this massive number of undergrads in many different disciplines.  Right now, what that means is that there'll be a low number of areas of prominence.  So if you use the SyVel lens and you look at those universities,  what you'll find is a relatively low number of areas of prominence  because there's such a spread of research expertise, right?  A bit of a downside.  But a high degree of curiosity-driven research.  And then you've got the elite research-intensive universities.  They have lower undergrad numbers, high number of grad students,  but much higher areas of research prominence.  And when you do the SyVel, look at this,  you'll see that these elite universities have sort of 80% of the research is in areas where they are really world-class,  whereas the comprehensive universities, it's somewhere usually between about 10 and 20%.  So that's a major difference, is critical mass.  And nowadays, research is a team sport,  and this critical mass turns out to be really, really important if you're going to have world impact.  I don't think the model of the one PI with the one grant to a world-class research still exists,  but I think the team sport piece is taken over.  Then the third one I want to mention is the elite research provider.  And these are not universities, but they're sort of dominated by universities.  Things like the Crick Institute, the newly formed Turing Institute, the Catapults in the UK.  I'm not sure what these similar things look like in other countries,  but they're definitely dominated by the university, they're created by governments,  and they're supposed to supercharge our research activities,  often with partnership, you know, the health authorities and industry,  but they're special crucibles unencumbered by undergraduate education.  And then the fourth one I want to mention is a very special one,  and the Germans really invented and are very, very good at this.  This is the Fraunhofer Institute, where they're not really dominated by universities at all.  In fact, I would say they're more dominated by industry, and they're largely funded by industry,  and they're largely contract research.  So these are research institutes that really focus on what are the problems that industry and governments are trying to solve.  The most pure form of solving the exact societal problems.  Just a bit more detail on Fraunhofer, so this is about huge scale too.  72 themed institutes across the world, 26,000 people, 2.6 billion euro budget,  and interestingly of that number, 2.2 is contract research, so 2.2 directed research, that's huge.  To give you an example in the UK, I mean most universities are doing about 5% to 10% industry funding,  and some of that will be contract research, so completely different scale of contract research.  Alright, so that's what the models, and why is industry so important?  Well, other than promoting universities as sort of engines of the economy,  even from a very selfish PI perspective, when you have co-authored papers with industry,  your impact goes way up, citation impact goes way up, and impact, and social media attention,  everything goes way up, so you get huge bang for your buck.  But again at Kings, that's about 5% of our publications are co-published with industry, it's a small number.  And you can see the sole authorship is way at the bottom in terms of impact.  So this is outputs in the top 1% citation percentile,  and again the corporate ones at the top, sole authorship on the bottom as you might be able to predict.  Alright, so let's talk about metrics now, so now we get into the metrics piece.  I think most of us sort of have an idea that citation impact has done some good stuff,  and has been a reasonable proxy for research excellence, but not for all of the research spectrum.  So it doesn't work very well for a lot of social scientists or humanists,  and in fact it doesn't even see them because they don't publish in traditional scholarly journals.  The fountain of HQP, you know this should be really easy, but we don't do a very good job of it,  and in fact when we think about our graduates, we don't really care that much about them until they become older and richer,  and then we think of them as alumni and we phone them up and say, you know, how about giving us some cash?  And it's big business, I mean at UBC we raised about $200 million a year from our friendly alumni and rich donors and stuff,  so it's big stuff.  The economic engine bit, I think we've got kind of, we know what the metrics are,  but the societal impact is the one that really, really is difficult, and we're really going into this area hard.  So at Kings, I mean we spent $30 million quid recently to buy this science gallery that's right beside Guy's,  it's like a fabulous monument to, you know, it's kind of a gallery,  and what we do is every three months we have a new sort of research theme,  and it's free, there's a big sign saying come and visit for free, there's a cool coffee shop there,  as I say it's thousands of people walk by it's right beside Guy's Hospital and right beside the shard.  A different research theme every three months, the last one was dark matter,  but it's not just about science, it's about art, so it's about the science and art of around dark matter at Kings.  The one before that was called Hooked, and it was about addictions,  the science of, you know, Kings is big into addiction research,  and it was about, you know, art of people who are addicted to.  So it's great, and we even have a VP called the VP London, who's in charge of public engagement,  so we spend huge amounts of money on this, Baroness Deborah Bull is our VP London,  who, you know, runs a science gallery and all of our kind of public engagement, outward facing stuff,  so it really is very big for us, but how do we measure this?  We absolutely fail to measure effectively to this point,  and I think this is where the big opportunity for altmetrics really comes into play.  So the bibliometrics works fine for some of the crucible and knowledge model, but absolutely not for the societal impact,  and so, you know, this is where we look at the transformation of how we use metrics.  Okay, so here's what Kat asked, you know, who's using altmetrics in universities?  Well, as I say, I represent one piece of that, and that's kind of where the rubber hits the road at universities  in the interface between researchers and the real world, researchers and funders, researchers and industry.  And so the different bits of what we do are things like, you know, our Vice President Research, Reza Rezawe,  worries about things like what's our research performance, what's our strategy, who do we partner with?  Those are the big sort of executive research questions, and they are informed by, you know, our altmetric and our Syval view of the world.  People like us, the research administrators, we worry about research excellent metrics.  REF is absolutely huge, and I'm going to show a few slides further on that really show how valuable our metrics is in doing a better job of REF.  So the way REF works for non-Britz is that there's a huge amount of block funding available, and we have to do a submission,  and that submission means that we have to present the best papers for each academic at the institution,  and it's all about impact, they get one, two, three, or four star.  And then we have to write impact statements for a limited number of our researchers, and again, there's a sort of a peer review of all of this stuff,  and at the end we get a big check, and we get a bigger check if we're more competitive than our peer universities.  The last time I think we did really, really well, we're not quite sure why, but this year, this round we're going to do really well,  and I think it will be largely because we really worked hard at identifying our impact and evidencing our impact,  and we've gone beyond the usual kind of idea of field normalised citation impact and stuff like that.  Librarians, I mean, you know better than I do how they use altmetrics, but I will say at Kings, our library system,  it's very important because they are almost holders and protectors of the systems and tools and trainers of people,  so when it comes to things like our effectiveness at using SyVal and altmetrics, the librarians play a huge role in making sure that that sticks and works well at Kings.  And then there's the traditional kind of, you know, your department heads that worry about things like assessing their PIs  and who gets promotion and who gets tenure and that type of stuff, you know, the basic kind of performance management.  And then the big one I would say that really is a huge opportunity for us is the PR and communications, which is really big.  We want to be noticed, we want to build our reputation, and we do that by finding our way into the news and into important public fora.  So that's a bit of a, you know, that's a bit of a sort of a mishmash of where the opportunity lies.  I don't think Kings has taken full advantage of that. We just started subscribing fairly recently.  We're sort of still getting ahead around what the value is.  If you ask me what's the biggest win for us, it's absolutely ref at the moment.  That's where the value is. That's how I sold it to the decision makers that provide the big money.  Actually, it wasn't, I had to convince myself it was a good idea.  So this is what we do, identify research impact, evidencing research impact for ref.  I think importantly, providing qualitative context to the bibliometric analysis of our outputs at Kings is important as well.  Research strategy. We worry about partners, you know.  We universities join clubs. We have big global clubs of universities in the UK.  The clubs called the Russell Group, and it's just the biggest and best.  But internationally, the clubs like the U21, UBC was a member of the U21.  There's one called the Guild that we're a member of at Kings.  It costs a lot of money to do this, and you want to make those decisions about which club to align with sensibly,  so that there's maximum opportunity for collaboration as you invest this money in partnering with your club members.  Assuring and just fine open access, talk about that a little bit more.  And evaluating, obviously, the non-STAM research outputs is very important for us as well.  So we know what the pros and cons are. You know, the bibliometrics, it's a great direct measure,  or a good proxy for research excellence is very important for global rankings.  But it only sees certain parts of the research outputs, the research spectrum.  I think the alt-metrics is an opportunity for a more nuanced approach to measure and impact.  It's a more direct measure of impact. It sees research activities in social sciences and humanities.  They're really important. I mean, UBC, our social scientists were higher up the world ranking than our medical researchers.  And so this is really important for us.  But, you know, on the con side, it does measure negative attention as well as positive.  And I think the biggest chink in the armor is that, you know, unlike citation impact,  sometimes the datasets are not comprehensive, and so it misses stuff.  And that's because things like mentions in policy documents, it's about the scrapability of those  and accessibility of those documents.  But you know what research is? If their paper isn't picked up by a particular system,  the researcher immediately thinks, well, this is a crap system. It's missed me out.  So that is a dangerous vulnerability of alt-metrics that I just mentioned as obvious.  So this is what it's all about. We all know these different aspects of alt-metrics.  I should mention that PlumEx does slightly different stuff.  And, you know, I think these systems are very complementary.  And I think it's not a question of either or. I think it's a question of just making sure  that you've got access to all of them, or most of them, or as many as you can afford,  and make the best possible use out of them.  So here's a few case studies. And I wanted to convince myself, and this is the work of Jürgen,  that our top researchers from a citation impact perspective were also top researchers  of pure alt-metric look at the world. And so Seb is one of our top researchers.  He's a medical imaging guy. I think this is a three-year snapshot, 400 outputs.  240 of them are picked up in the alt-metric view of the world.  1800 Twitter, 48 patent, 160 news mentions.  You know, shows up loud and clear in both systems, and that's good to know.  Here's some of his top spotted papers, real-time imaging of single neuronal cell apoptosis  in patients with glaucoma. It's important stuff.  But he does show up really well in both lenses.  I think that was important for us.  This is Reza. This is our vice president of research.  Reza is a pediatric cardiologist who treats very young kids,  and he's trying to improve the scanning capability so that he can do a better job of diagnosing  and treating congenital heart diseases in very young kids.  Interestingly, I asked him where his drive to do this came from,  and he said, well, I have a daughter, Poppy, who has a congenital heart disease,  and he thought this was worth going after.  So that's his personal story, which I thought was very inspiring.  And again, he shows up in the alt-metrics view of the world as well.  Thanks, Jurgen.  All right. Let me just sort of talk a bit about when you sort of compare and contrast, right?  This is work that we did at King's, looking at 144,000 King's journal articles.  And so on the x-axis, you've got the alt-metric view of the world,  and on the y-axis, you've got the citation impact view of the world.  Now, you look at that and go, it's a bit messy. It is.  But, you know, there's a correlation.  I think the most important thing, though, more important than correlation, is that  if you want to pick up all of your impact and attention, right,  the citation impact is going to give you this stuff.  Both is this stuff, so you've got some correlation there, but what about all this stuff?  You know, this is stuff that you're going to miss if you don't look through the alt-metric lens.  And we can't afford to miss this.  We can't afford to miss it for RAF.  We can't afford to miss it for our outward-facing public engagement strategy.  So this is really important. I think this is an important slide, and not to be forgotten.  Okay, life and death is all about publishing for researchers and where they publish.  And so everyone wants to publish in the highest impact journals.  And of course, things like the Shanghai Global Ranking, one of their key metrics is how many papers do you have in Nature and Science.  So, you know, that's how serious this stuff is.  But I just wanted to give you a couple of things that we look at.  So here's different publishers and different journals, and this is the five top publishers in psychology and cognitive science.  And you can see that they vary very widely in terms of news mentions per article.  Now, why is that important? Isn't it all about citation impact?  Well, no. And there's lots of reasons that this is important.  When it comes to news mentions, you know, things like donors, publish donors, and for performance reasons as well,  it's really important to know what sort of attention you're going to get in different publications.  You can then put that information beside your citation impact and make clever decisions on where you need to publish.  If you're lucky enough to have that kind of choice.  Same kind of thing. This is the five best journals in psychology.  You can see the journal impact factors, which of course we're never supposed to even look at, but we all do.  And what you can see there, though, interestingly, is the Almetric Attention Score per article is almost 108 degrees apart.  The top journal in terms of impact factor is the lowest on the Almetric Attention Score.  So interesting what you can drive out of this kind of analysis.  It's not all obvious that the top GIF is the top place to get attention outside of the scholarly community.  And as I started my conversation today, you know, scholarly impact isn't the same as real impact, or when I say real impact,  what society thinks is real impact.  Again, here's more of the same. This is the top journals in psychology and cognitive sciences, average news mentions.  And so again, this gives you the massive scale of difference between the different journals and how likely they are to get picked up  and picked up by the media.  This is a super important one as well.  So when I think about public engagement, I think this is the scopus topic at King's receiving the most news over the last three months.  So this is kind of what's hot at King's.  Why is this important?  This is what people are talking about and want to know about and need to know about.  And so that informs a lot of different things at King's.  Like, you know, what's our externally facing PR strategy for this month and what sort of externally engagement activities,  what kind of lectures are we going to be doing?  You know, if we want to be relevant and we want to be upfront, this informs that exact strategy.  Baroness Bull wants to know what this looks like and what to react to.  And interestingly, some of these, like for instance, tobacco, you know, that doesn't show up in the citation impact  because it's not stuff that we publish on, but the public are sure very interested in it.  So there are areas that really, you know, diverge when we look at our traditional scholarly assessment of quality.  This is policy mentions.  So again, you know, why is this important?  Well, we've got a policy institute that needs to know what areas of research that we're doing at King's are the most important.  I'll mention another thing, and I shouldn't really say this, but we do worry about it, is that there's quite a bit of research recently on funding patterns.  And it turns out that your success rates are way higher if you are asking for money in sort of trendy areas of research.  Now, that's probably no surprise to anyone, but you know, it's important to know what those trendy areas are.  So you might ask, well, how important is it really to be in the news?  Why don't we just ignore it?  I don't think we can for many different reasons.  If you think about Times Higher Ed ranking as an example, a huge part of that is the subjective global survey.  They send out 300,000 surveys to researchers all over the world who publish in their journals and ask questions about, you know, what you're ranking the best research in the best teaching universities in the world.  Now, I've talked to some of those researchers who fill in that survey and I've said, you know, what do you think about?  How do you make your decision on the best universities in the world?  Well, there's three things.  Usually they've had a personal experience.  You know, I was a postdoc at Oxford, wonderful university, and of course I'm biased.  I think that's a great place.  There's history and prestige.  Universities like King's who've been around for a while, and we have.  I think our medical faculty began, you know, 900 years, a long, long time ago.  So we get some free, free prestige, free points for being old and prestigious.  And then the third one that's really important always is said is you are in the news.  You know, we hear about you.  You're obviously doing relevant research.  Society cares about it and that's important.  So it's kind of those three things.  So I think absolutely it's important.  I'm getting close to the end of my time here, but I'm almost finished.  Open access.  This is really important for us.  It's coming down as regulated.  Plan S has been delayed by Europe, but it is coming.  The reason it's important is that the publishers charge us twice for a lot of stuff.  You know, they're charges for a subscription and then charges for an APC to bring a piece of research outside the paywall.  And so we have to make different difficult decisions about where we publish,  whether we can afford to publish in certain journals.  But this is just so we have to convince ourselves and convince our funders that they should fund APCs so that we can do this open access stuff.  And you can see the difference between closed access and open access is huge.  It's a factor of three for alt metric attention.  It's a factor of five for news mentions, factor of four for blog mentions, you know.  So open access, we can demonstrate the value of that if we're not doing open access, we're not being noticed.  Similarly here, this is the field normalised citation impact.  This is the funder.  So this is the welcome trust funding.  And what you can see here is that the research funded by welcome trust gets a lot more attention, gets a lot higher citation impact,  attracts a lot better science compared to non welcome trust funders.  Again, I can use this and say researchers, you should be applying for welcome.  I know it's hard to get, but you know, it's going to be worth it.  And by the way, I'll help you get it with various research development activities.  This is a side by side comparison of field weighted citation impact and alt metric attention scores on the same issue, welcome trust funding.  And you can see two things, big difference between welcome trust and non welcome trust.  But you can see nice sort of lineup of both systems, alt metrics and citation impact.  I think this is the last one.  When we look at ref, like I say, an important thing for us is to be able to predict how good our papers are in terms of one star, two star, three star, four star.  We want everything we submit to be four star.  We use citation impact.  We use a bit of internal peer review and we also look at the scores here.  Now you can say, wow, that's, you know, there's a lot of overlap.  Yes, there is.  But as one of a few different filters to put our outputs that we're considering for ref submission, you know, this makes sense to look at.  All right.  So what have we learned?  And I'm a minute early, I think the university models evolving, absolutely evolving.  And it really is changing the way we use KPIs and the KPIs that we need to use.  I think I've shown a little bit that our metrics is highly complimentary to a traditional.  It does the same kind of job and it does a different, very valuable job for us in universities.  I think it's the impact agenda, which is very important in the UK, certainly in North America as well, absolutely demands these new ways of measuring impact and the R&D.  Just can't be done with the traditional bibliometrics.  And the last thing, you know, Outmetrics is used by and can be used by many parts of the institution for a very effective outcome.  I think those are the main things I want to say.  Thanks for listening.  I hope you learned a little bit and certainly I know this is a diverse crowd of people.  I don't know, are there research administration people?  I know it's libraries and publishers, but yeah, okay, good.  Well, it's good to see some colleagues in the same business.  I hope the rest of you have learned a little bit about, you know, research administration and how university works from our perspective.  Thanks very much for the opportunity.  Thank you.  Before we let you escape, any questions that anyone would like to ask?  No, everyone wants coffee?  Yeah.  Martin, you refer to, I made a note of it, PR and communications as becoming a more important aspect of university life as well.  How does that work out in King's College regarding bibliometrics feeding back into PR and communications?  Normally, those are two different departments.  Are they teaming up more than before?  How's that going?  So I'll tell you, interestingly, King's has made a big bet on a completely different model of how we run the university.  It's called functional alignment.  What that means is that there's a function lead that manages every aspect of a particular part of the institution.  I'm research.  Now, what that really means, though, and so communications is a function.  What it means, though, is that, I mean, I have communication people in my area, but all of them have a strong direct relationship to the function lead in communications.  So you could think of them as moles or spies or whatever in research.  But we have a level of communication and cross-disciplinary kind of communication that allows that kind of thing to happen.  So the communication in my group feed back into the central group with things like these metrics, all of this stuff I'm talking about.  So we have sort of an extra layer of channeling built in through this functional alignment at King's.  That's probably the best way to answer it.  I think it's a big advantage.  It's not easy to do, and it's costing us lots of money and time.  There's a dip in, you know, the satisfaction surveys of our researchers because it takes time to get right.  But that's kind of what we're aspiring to, and that's where we've seen that level of communication happening.  Okay, I think that's very good to hear.  You were also talking about this negative attention that may be something that may work out as negative towards altmetrics as well.  How do you deal with that in King's College?  You know, I actually remember a couple of years ago asking you about that.  And I have to admit that although it's sort of the witch in the cupboard, I don't think it's that big a deal.  But I seem to remember you and saying that what the industry was looking at was there are clever potential AI applications where you can screen out and filter sort of quite easily.  It's not that difficult to do.  So I think that's one of the traditional sort of here's why we don't like altmetrics, but I actually think it's almost a non-excuse nowadays.  And you know, the way AI is going.  So I've got AI applications that can sort of do research development, do very clever things.  So I actually, I think that's almost not worth mentioning anymore to be honest with you.  Hi, Joe from Crossref.  Do you care about fake news articles?  There's a lot of news articles out there.  Is that not exactly the same question?  I think it is. And I don't know if I know the right answer to that.  And you know, how sophisticated is King's at spotting that?  I don't think we are very sophisticated, but we do worry about a lot.  And you know, if you expand the fake news to fake science, you know, one of the things we worry more, more, most about is research integrity.  And you know, you've probably heard about some of the breaches that have happened in the recent past that really have tarnished university reputations and brought down presidents and vice president researchers.  Karolinska, Paolo Macarena is probably a fairly recent one.  So fake news and fake science, absolutely deadly for us.  And I, you know, we worry about that a lot.  Again, we've got good process in place to handle some of this stuff.  But I don't think we've got a track record of successfully protecting our reputation in these areas.
Keynote_-_Beyond_Bibliometrics_0_47.png Keynote_-_Beyond_Bibliometrics_48_95.png Keynote_-_Beyond_Bibliometrics_96_527.png Keynote_-_Beyond_Bibliometrics_528_1751.png Keynote_-_Beyond_Bibliometrics_1752_3467.png Keynote_-_Beyond_Bibliometrics_3468_5891.png Keynote_-_Beyond_Bibliometrics_5892_9779.png Keynote_-_Beyond_Bibliometrics_9780_11075.png Keynote_-_Beyond_Bibliometrics_11076_13415.png Keynote_-_Beyond_Bibliometrics_13416_15599.png Keynote_-_Beyond_Bibliometrics_15600_16835.png Keynote_-_Beyond_Bibliometrics_16836_19487.png Keynote_-_Beyond_Bibliometrics_19488_19859.png Keynote_-_Beyond_Bibliometrics_19860_20975.png Keynote_-_Beyond_Bibliometrics_20976_25751.png Keynote_-_Beyond_Bibliometrics_25752_26615.png Keynote_-_Beyond_Bibliometrics_26616_27779.png Keynote_-_Beyond_Bibliometrics_27780_31619.png Keynote_-_Beyond_Bibliometrics_31620_31991.png Keynote_-_Beyond_Bibliometrics_31992_37379.png Keynote_-_Beyond_Bibliometrics_37380_39035.png Keynote_-_Beyond_Bibliometrics_39036_41207.png Keynote_-_Beyond_Bibliometrics_41208_42035.png Keynote_-_Beyond_Bibliometrics_42036_43079.png Keynote_-_Beyond_Bibliometrics_43080_43559.png Keynote_-_Beyond_Bibliometrics_43560_44483.png Keynote_-_Beyond_Bibliometrics_44484_44675.png Keynote_-_Beyond_Bibliometrics_44676_46487.png Keynote_-_Beyond_Bibliometrics_46488_48491.png Keynote_-_Beyond_Bibliometrics_48492_49823.png Keynote_-_Beyond_Bibliometrics_49824_50399.png Keynote_-_Beyond_Bibliometrics_50400_52307.png Keynote_-_Beyond_Bibliometrics_52308_53303.png Keynote_-_Beyond_Bibliometrics_53304_55427.png Keynote_-_Beyond_Bibliometrics_55428_56963.png Keynote_-_Beyond_Bibliometrics_56964_57815.png Keynote_-_Beyond_Bibliometrics_57816_58391.png Keynote_-_Beyond_Bibliometrics_58392_59327.png Keynote_-_Beyond_Bibliometrics_59328_61511.png Keynote_-_Beyond_Bibliometrics_61512_61571.png Keynote_-_Beyond_Bibliometrics_61572_61763.png Keynote_-_Beyond_Bibliometrics_61764_61811.png Keynote_-_Beyond_Bibliometrics_61812_61859.png Keynote_-_Beyond_Bibliometrics_61860_62039.png Keynote_-_Beyond_Bibliometrics_62040_62159.png Keynote_-_Beyond_Bibliometrics_62160_62279.png Keynote_-_Beyond_Bibliometrics_62280_62471.png Keynote_-_Beyond_Bibliometrics_62472_62663.png Keynote_-_Beyond_Bibliometrics_62664_63335.png Keynote_-_Beyond_Bibliometrics_63336_64019.png Keynote_-_Beyond_Bibliometrics_64020_64175.png Keynote_-_Beyond_Bibliometrics_64176_64223.png Keynote_-_Beyond_Bibliometrics_64224_64715.png Keynote_-_Beyond_Bibliometrics_64716_64835.png Keynote_-_Beyond_Bibliometrics_64836_64907.png Keynote_-_Beyond_Bibliometrics_64908_65039.png Keynote_-_Beyond_Bibliometrics_65040_66347.png Keynote_-_Beyond_Bibliometrics_66348_66395.png Keynote_-_Beyond_Bibliometrics_66396_66467.png Keynote_-_Beyond_Bibliometrics_66468_66527.png Keynote_-_Beyond_Bibliometrics_66528_67571.png Keynote_-_Beyond_Bibliometrics_67572_68171.png Keynote_-_Beyond_Bibliometrics_68172_68199.png