Evaluation (2)
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Schlögl, Christian
This lecture offers examples of Evaluation of Information Retrieval Systems. This lecture was held during ESSIS 2018 under the european project EINFOSE funded by Erasmus+
 So I think we are not complete yet, we should continue. I will shortly repeat what I have said before. First of all, I try to give you a short introduction into evaluation in general. Then I mentioned a little bit about evaluation of information services and here you have experience that it's already critical to define what an information service actually is. And as you have seen, there's much going on and I think this has also big consequences for university libraries and it will stay the same also the next year. When you try to evaluate information retrieval systems, you can evaluate single information services, one possibility or you can for instance evaluate a library as a whole. What concerns the later, I have shortly introduced you to LibQual, which is a special set of surfcrawl and the big advantage of LibQual for instance is that you don't have to make your own questionnaire, you can participate in a survey in which also other university libraries for instance participate, you have a comparison with other universities, you have a comparison with yourself over time. So I think quite interesting if you are maybe head of a university library and if you want to get regular feedback. Next, I started with the third topic of my presentation, it's about evaluation of information retrieval systems. And to some degree you can compare it with evaluation of information services in general. So usually you can evaluate an information retrieval system with regards to three dimensions. The first one is the perceived quality of an information retrieval system, so you for instance could use surfcrawl or a similar method. But the problem is, as also mentioned before, you only get subjective evaluations of the users. Therefore it makes sense also to do a little bit more objective. And with regards to information retrieval systems, one additional important dimension is the information, the knowledge covered. And I tried to explain it for university library, for researchers, it's very, very important to have all top journals of that field which are covered by the university. So for me, I need full text of all journals in Scientometrics and to the top journals in Library Information Science. And since I am also involved in management business administration, also e-chernels in debt disabling. So that's very important. This refers to knowledge quality and this must be covered by an information retrieval system. That's one aspect. It refers here to documentary reference units. And with regards to journals, I told you already, and with regards to databases, of course, I only use databases which cover all that information, that knowledge. Also important is the second aspect, not only the covered literature, but also how it is documented in the information retrieval system. And one important aspect is if the particular information retrieval system, for instance, uses a classification system, if the information retrieval system, for instance, uses a Tessaurus and so on, because it makes it much easier to find the relevant literature. And we discussed it before. Google Scholar doesn't offer anything in this regard, but good databases usually have either a classification system or a Tessaurus or nomenclature, or even both. In case of web of science, you only have a classification system, but that's not too bad. And with regards to indexing, it's not too detailed, as I told you, but at least also the other key birds are included, so you really manage to find the most relevant literature. And just one remark. Why do you think it is not necessary to do more detailed indexing or to do consistent indexing in web of science or in other citation databases? Which other possibility exists to find similar literature, which is even better to do it with classifications and indexing? With citations. For instance, in web of science, you have great, if you found maybe one document, you can then ask, please, I want to have similar literature, and on which basis do you think can this similar literature be found? Not to some degree on other key birds, but what do you think is more effective to find similar literature in a citation database? From the text to the sound. Yes, right, right, right, right, right. Yes, that's it. So in web of science, you can say related records, and then you get the ranking. Most related record has, for instance, 20 references in common is this publication. Second most similar publication is this one with 18 identical references and so on. And usually, to my experience, this is much better to search for similar literature than to use keyword or even indexing. And that might have been the main reason why this is not so good in web of science, for instance. Okay, but in principle, it exists and what's very good in web of science are abstracts. And usually, this is obligatory in web of science that each article has an abstract. In Google Scholar, it depends if you're lucky and if Google Scholar retrieved this information, then you have an abstract. But in many cases, you don't have an abstract. And on the other hand, it's of course free. Okay, then another important point which is good in both information retrieval systems is update speed. So usually, as soon as there are new general issues appearing, then you can retrieve them even before. That's one aspect. And the third aspect then is which functions do the information retrieval system provide? In other words, how easy is it to perform the tasks you have to perform with this information retrieval system? And it starts very basic. For instance, several information retrieval systems consist of more databases. In the case of web of science, these are six, I think. These are three journal certification indexes. That's a science certification index for sciences, social sciences certification index for social sciences. And the third one is arts and humanities certification index. In total, 12,000 journals are covered in web of science, which is quite a lot. And all articles appearing in them, so I guess 40 million articles, maybe even 50 million articles in the meantime. In Google Scholar, you don't know it. I read some figures, approximately 150 million articles. So much more comprehensive, but also several articles are publicated. Then Google Scholar also covers books. Google Scholar also covers e-prints and so on. Web of science, it's much more regulated. Okay, these are three databases, and then there are two proceedings databases. Web of science, not only journal articles are included, but also articles who appear in proceedings. And there are two databases. One is the social sciences proceedings index, and the other one, the sciences certification index. Then there's the books certification index, and you said there might be even several others. And what's very nice, you can say, since we mainly work, for instance, in social sciences, I only want to use social sciences certification index and, for instance, conference social sciences certification index. But you can also search on all databases. And two of these in parallel are only one, so this is no problem. It's the same also in EPSCO, for instance. Okay, that's the general point. If you really want to work with the database, it goes into more detail. And one important point is looking for search arguments. Does the database assist you in finding search terms? Some databases allow that, for instance, if you're searching the title, if you type in a name, for instance, information lit, and then it suggests information literacy. This would be helpful, actually. It's a kind of browsing a dictionary, but could be even better. But this is actually meant. Only a few databases also allow you to find synonyms very easily. But this is not the case in the Depth of Science, and definitely not the case in Google Scholar, for instance. Sometimes it would be very helpful, for instance, if you want to make a search in a certain category, that you can have a look which categories exist. This is not so easy possible in the Depth of Science. In the Depth of Science, there are the so-called journal citation reports where you can have a look to which field, to which discipline a journal is assigned to. But that's it, actually. It would be nicer if it's much better supported when you do a search. In Google Scholar, definitely, you have no assistance at all. This is, yeah, noted. For instance, in Epsco, you have a Desaurus. You even can search in the Desaurus, and you can discover it. For instance, there's one descriptor called Information Literacy. You can search or see narrower terms, broader terms, related terms, synonyms, and so on. In Epsco, this is partly very well assisted. And why do you think this is important? That Epsco allows you to search in the Desaurus and to discover the terms which are related to your search. Yes, it helps you to find the relevant search terms, actually. For instance, you make a subject search for information literacy and you find, let's say, 100,000 hits. What could you do? You could go back to the Desaurus and have a look at their narrower terms. And then, for instance, one narrower term is Information Search Literacy and so on. So you could narrow your search using this Desaurus in this case. Usually, as far as I know, but Thomas Mandel might know it better, this is not very well supported in most, or not at all supported in most information retrieval systems. So it would be very helpful. What I do, for instance, is, before I start a search, I often consult the Desaurus in another database, actually, only to find the relevant terms. And then I go back to Web of Science and then here I start with my performance search. But a very critical phase finding the right words for your search, actually, and also a very important one. Okay, what are other important functions that which assist you in retrieving the right information? For instance, output functions. One function which is very important according to which criteria you can search the results. Google Scholar, how are the results sorted? But there are two criteria according to which you can sort the results in Google Scholar. Yes, one is up to date-ness. And the second one is, you have heard it several times before. The other one is relevant. So you will see it tomorrow. The group who works with Google Scholar, one is relevant, one is chronological. The question is what means relevant. In Google Scholar, you also can see the situations. And my impression is that the number of situations play a very important role. But actually, it's not the only criterion. So my assumption is that it's also very important in Google Scholar if the search term is in the title or if it's in the document. And another criterion might be how many times the search term is in the document. But would be an interesting research finding it out. Okay, but sorting is very important. In the depth of science, I think you have 10 possibilities to sort the results. One result is citations. Another one is relevance. Another one is alphabetic by author. Another one is alphabetic by journal, etc., etc. So much more possibilities. And that's very important, in particular, if you have found many hits. Okay, sorting possibilities. Marking filtering. What does marking mean? To mark one document in the research list. So you make your ticket. And then in the depth of science, you add on the button add to mark list. And this way you can collect all documents which are relevant for you. And what's very nice, then you can print them, you can export them to other files. So very nice in depth of science. Can you do marking in Google Scholar? Actually, I don't know, but the group tomorrow will find out. What's about filtering? Filter results. What could this mean? Just copy the filter to a document. Yes, very good. For a user or... Yes. For citations. Yes, very good. In the web of science, you have very good filtering functions. For instance, you have found, let's say, 1000 documents. And then you can filter, for instance, by document type. For instance, you only want to consider articles, document type articles. Or for instance, you have to... You search for... You want to buy a book. You can... There's another document type, book review. And you only can filter for book reviews. So maybe 10, something like that. That's one possibility. By document type, by year, by citation, by journal. I think you might have probably 10 different kinds of filtering possibilities in the web of science. In Google Scholar, I guess none filtering possibilities. So very important. And usually then you can work with the filter document, search them, output them, print them, etc., etc. And usually, yes. For instance, if you have your mark list and all together you have collected, let's say, 20 documents. Afterwards, you can export them both in Google Scholar and in the web of science. And important here, which file formats are possible. For instance, CSV is for spreadsheet programs. But there are also diverse formats for bibliographic databases and so on. And in this regard, as far as I know, both information retrieval systems are quite good. The next one... Linked to digital version. Yes. Google Scholar is, and also the web of science, are bibliographical information retrieval systems. And so therefore it would be, of course, important to get the full text. In Google Scholar, if you're lucky, yes, there exists maybe a version of a document where you have also the full text. For instance, the preprint. In the web of science, I think there should be a link to a document delivery service where you can buy for one article. How much is ton average? Then 20, 30 euro is not seldom, actually. But usually there are also so-called, I think, link resolver to our university library. So if the journal in which the article appeared is available, and if you click on the link, you get the full text. Usually it works in, I would say, 90% of all cases. But that's an issue which our university library has installed. Okay, this relates to display and output. And this is one category where you can compare information retrieval systems. The next one are so-called push services. What could this be? Do you know what the push service is? Or another name. I forgot it. But it's not so important. But the idea behind is if you work, for instance, on a master thesis, it takes maybe one year or half a year. And usually you would make the same search once a month, actually. And usually it's very boring to go into the database and type in the same commands. So therefore you can save your search command and tell the system if a new document appears in the database, then you get automatically an email. So it's called push service. So there's all the... Or selective dissemination of information, SDI, is also one term how it's called. I don't know if Google's call support could be. Google News supports it, I know, because I regularly get newspapers for a topic which interests me. Google Scholar, I believe that it's also available in Google Scholar, but you have to have an account with Google and then you can define it. Okay, this is a push services. And then the next one are so-called informatric analysis in Web of Science. For those of you who know it, for instance, I can do simple ranking commands. For instance, I search for information literacy. I found, let's say, 1,000 documents. Then I can say, analyze, buy author, and then I would get a ranking which author has published most articles about information literacy, second most, and so on. And it's very easy. I want to find out the journals in which most literatories publish. Then I say analyze, journal or subject, or source, I think. And then I get a ranking of the journals, or if I want to know if the topic is very up to date, then I say analyze, and then I select publication year, and then I get a ranking by publication year and so on. So very helpful. And Web of Science is one of the only databases which offer this feature, actually. And I can do the same with citation. It's called citation report, then I get the ranking of the most cited paper, second most cited paper, the distribution of citations over the years, and so on. So also very helpful. In this regard, many other information retrieval systems are relatively poor. And in Google Scroll, of course, you don't have any feature at all. Okay, and there are many other functions which you will discover tomorrow. For instance, another important feature could what be to help you searching documents. What would be else important? But we do it tomorrow in more detail. For instance, how many fields you can search for in Google Scroll, for which fields do you search usually? In which field do you search in Google Scroll? If you type in, for instance, information literacy, where is it searched in Google Scroll? You may in the search engine. Yes, in the whole document. So information literacy can appear in the title, can appear on page 19 of the article. So usually the standard assumption is Google Scroll does not distinguish among search fields. In the app of science, I think you have maybe 20 different search fields. One is title search, order search, you can search by publication year, by journal, abstract, etc. So you have many possibilities and the more possibilities you have to refine your search debate actually. Is it possible to make a title search in Google Scroll? Yes, how you can search only for documents who have information literacy in the title, what do you have to do? Yes, and how you can apply it? Yes, right, right, right, right. Yes, very good. You only can do it in the expanded search, in the extended search, not in the simple search. You must select expanded search, which is a little bit hidden, so Google Scroll is not so much interested to assist you in searching. And here you can limit your search to, for instance, the title, to the else, what else is possible. And that's very important. Also, yes, very good, and I think also publication year. So you have only three or four fields for which you can search in Google Scholar, in Web of Science you have 20 or 30. So also a very important criterion. What else could be very important if you use a search engine? So number of fields for which you can search. What else would you... Yes, topic or subject, but we had it already when I mentioned the thing about Tesaros. What's about Boolean operators? Are Boolean operators important? Yes. Definitely. Does Google Scholar support Boolean operators? Yes. Yes, you have end, you have or, you have not, but of course you have to know how to use it. As far as I know, Google Scholar does not support parent. That gets a little bit complicated, so often you don't get what you want, and this is much better in Web of Science. You can also set parent, and this is... You can also use phrase search in both databases. Are there proximity operators? So not all of you know it, but... What does it mean that you can use any Google Scholar? You can use? Yeah. And Google Scholar not, no? In Google Scholar, I think there's no proximity operator. You can only use truncation, of course. Oh, I forgot it. But you know what I mean, and that's all. In Web of Science, to some degree, there are proximity operators. Is it important to use truncation? Yes. Does Google Scholar support truncation? No, not at all. In Web of Science, you can use the star, which stands for one, two, three characters, which... And that can be very helpful, etc., etc. So these are many other search functionalities, and tomorrow you will get the criteria catalog in which these criteria are all listed, and you really have to check if the relevant database supports them. But I think for today this should be enough, because the next very important thing which we should deal with today is the effectiveness of an information retrieval system. What is this? Effectiveness means... there are actually two indicators which are very important. The first one is recall. Recall means if you have performed a search, then recall tells you how many percent of all relevant documents you really have found. That's recall. And precision, what could this mean? Precision means if you have, for instance, found 20 documents, and then you usually check it, and then you find out, let's say, 10 of the documents are relevant and 10 of the documents are not relevant. Precision means that 50 percent of the found documents are relevant. That's it. So precision means how many of the found documents are relevant, and recall means how many of all relevant documents you have found. What you think is difficult when you calculate the recall. Is it easy to calculate the recall, you think? Yes. That's the big problem with recall. Usually you don't know how much relevant documents are included in the depth of science database. You only can make a rough estimate, and that's it. With precision, it's much easier, because you have research results, and then you really can identify if it's relevant. Yes, no, and then you calculate it. And fallout is the third one, which is not so important. It's the percentage of non-relevant documents which you have found. And here an example, which makes it a little bit easier. Let's say we searched in depth of science, and let's say the topic is information literacy. In total we found 12 documents, and when checking the results, we found that 8 of them are relevant, and 4 are not relevant. So with this information, we can calculate the precision, which is 8 were relevant, out of 12 documents which were found, so 8 divided by 12, let me see, by 12 is 67%, not too bad actually. Actually we have some problems to calculate the recall, but let's guess that in total 20 documents would have been relevant. We have found 8 of them, so you have to divide 8 by 20, that's 40%. And fallout, 4 of our documents were not relevant. If we assume that in total the database consists of 136 documents, then we know that non-relevant 116, so 4 divided by 116, is 3.4% wastage, but that's actually not so important. So since I talk so much already, it's highly time that you make a small exercise, it's also on the platform, but that we don't over stress internet, I have prepared it for you. You just have to read it, it deals with an OPAC of a departmental library on information studies, total of 10,000 books, and you perform two searches, a keyword search and then a subject search, using a Tessaro for instance, and your duty is to calculate recall and precision for both cases. One question, this was very good, is I would say that recall and precision there is some relation between the two indicators. And here I have indicated this, and we have also just said the example before, if the precision is very high, what is this to recall? Is it to recall then also high or is it to recall lower? So you think if the precision is high, then also lower? Yes, very good. So usually there's a negative precision, if I have 100% precision, it is very likely that the recall is very low. And this was just example before, if I search for the exact type of a publication, the precision might be 100%, but I have only found one relevant document out of 1000 maybe. So high precision, low recall. If the recall is very high, how will the precision be? If I try to get each document which is relevant, will it be low? Will be low, yes, because it's in version actually, I have my, the query must be very wide, and as a consequence the precision will suffer. For example we just saw it's in the dynamon, I mean it's not always the case. Yes, yes, very true. Must not be... It's not a double implication, so it's a one-way implication. Yes, of course, but... If that also then... Right, right, right. But actually if you look at the other way around, it's also a reverse relation. Okay, another question is... I would say, not in all cases, both precision must be high and recall must be high. For, let's for instance, took the example, you work on your PhD thesis, and if you work your PhD thesis you should really read most or nearly all of the relevant literature. So what would you prefer in this case, a high precision in the search or a high recall in the search? High recall. High recall, because you want to... For all of them. Yes, yes. For a PhD thesis it's very important that... And yes, you have to scan, and if you scan the search maybe 90% is not relevant, but you get more relevant. And for the PhD it's important that you really have a good knowledge of the whole literature. So if you write on your PhD thesis, a high recall is very important. Another example, you want to get... you participate in the course, and you want to get some basic knowledge about the course. Is a high recall very important, or is a high precision very important? If high precision, it's enough if maybe even if Wikipedia is already enough, and if not then maybe a search in the lab of science and you scan through a few documents, and that's fine actually. So in this case, high precision is more important. What's about the pattern search? Is a high recall very important or a high precision? For instance, a company develops a new product, and it's very critical because it costs, let's say, 100,000 euro. Developing costs, the problem is if at the end you find out that there is already a patent on that topic, then you can throw away 100,000 euro at developing costs. So what is important here? High precision or a high recall? We still want to scan through all the documents. Yes, yes, high recall, yes. So you see... I think that we often prefer higher precision because I wouldn't... In my own reason, that's why I also said high precision because I don't want to search like 200 documents. I mean, I have like a medium, I guess, 260, you know? I don't want extremes. You're right, but you have to make compromises in reality. For PhDs, we would make compromises. For patents, not because so much money is involved. But in that case, you would pay 20 guys. Yes, yes, that's it. And that was actually the sense of why I noticed this. There is no clear response. You really have to decide what you strive for, actually. But I think you have understood it. So in particular, the concept of recall is a little bit theoretical because you never know how many documents there are. It gives you some understanding about search of literature. Okay, another question is, can you use recall and precision to compare also to information retrieval systems? In this case, Web of Science and Google Scholar. Is this possible? Yes, because if, for instance, Web of Science, and you have seen it before, offers many possibilities to search for literature, not Google. And therefore, usually recall, but also precision would be higher in Web of Science than in Google Scholar. So you have two curves like this. And as you have seen, they are high precision, low recall, high recall, low precision. But in both cases, Web of Science should be much better than Google Scholar, for instance. So you could even compare information retrieval systems using these two indicators. Clear? Yes, and problems, I have already indicated this. Recall is usually not known. If you are very well experienced on patents, for instance, you know approximately. If you consult a librarian, for instance, the guy who made the guided tour, he may know how many documents approximately there are. That's one thing. The next question is, what do you do with search engines? Google, for instance. Can you calculate recall with Google? Impossible. Can you calculate precision with Google? You could, you say, for instance, you only consider the first 50 documents, and you calculate the precision for the first 50 documents. And if you compare, for instance, Google with Bing with Yahoo, then you get actually an approximation. And there's even one possibility to calculate the so-called relative recall. If you, for instance, compare three search engines, and if you say you only consider the first 500 documents, and you look for them in Google, and then you look for them in Yahoo, and you find in Yahoo documents under the first 500, which were not considered in Google, you can add this to the total number of relevant documents, and you do the same with Bing. And that's the way you could calculate the kind of relative recall. And I think it would not be uninteresting, because you see that there are quite big differences also between different search engines. I think the big problem is monopoly of Google. I told you already 90% use only Google and only rely on Google, but you would see that the differences are more or less big, in particular with regards to recall, I guess. So it would be interesting, and I guess there are also several studies in literature about this topic. What is another problem with recall and precision? Actually, you say a document is either relevant or not relevant. What is your experience with all of you who have written the seminar papers? If you write a seminar paper, is each document the same relevant or are there differences? There are differences. There's one document which is most relevant, another document second most. It's relatively easy to make a ranking, and this is not at all considered here, of course. And actually, therefore, it's possible, they developed a method how to calculate relative precision, and I will show it to you approximately how it works. You don't have to understand it. It was mainly important for me to show you that it's a shortcoming. But the main idea is that you only consider, for instance, the first ten hits, and you order them according to relevance. And here you searched, for instance, for information management, and the first hit was relevant, the second not. The second most relevant was number three, it was relevant. The third relevant document was ranked under the sixth position, and then these two. And then, next, you calculate the relative precision. So this one is relevant, one divided by one is one. And here, if you got the first two hits, out of the two hits, one is relevant, out of two, so one divided by two is 0.5. Here, if you consider only the first three, two are relevant, divide by three is 0.67. Two out of four are relevant, two divided by four is 0.5. And you do it, you continue that way, and you get these numbers here, and then you add those values, there you have a yes, one plus 0.6 plus this one plus this one plus this one, divided by five gives a relative precision of 0.62. And this is another query, and you do the same, so the first one is not relevant, one divided by, zero divided by one is zero. Here, one out of two is relevant, one divided by two is 0.2, and so on and so on. And so, you add the precision values, where you have a yes, 0.5 plus this plus this, divided by three is 0.44. And then, if you want, you can add these two divided by two gives an average precision of 0.5. But it's not so important. But the main difference to the other method is that the very relevant documents get the higher weight, so one, for instance, or 0.5, and the documents which are further behind get lower weight, and somehow you can consider the fact that not all documents have the same value. You can, difficult? But the main idea is that the most relevant documents get higher numbers, and the most relevant document gets one. And this is the fifth, so there's one, two, three, four. The fifth relevant document only gets the weight 0.5. So you give not the same weight for all relevant documents, you really try to distinguish. That's the main idea. And then you do some calculation, and of course you get other values. And usually, for instance, if you have two searches, and you have five relevant hits under the first five positions, you have much higher values than if you have your five relevant hits here, and you really could say that this precision will be much better. But forget it, it's not so important. It should only illustrate that in the normal calculation, there's no distinction between more and less relevant hits. You only say relevant, not relevant. And I don't know who invented this, and I cannot imagine that it's used in practice, but it helps you to make clear that there is an order of relevance, of course. Okay, other questions? No. And that's it. And yes, I mainly referred to the book chapter by Professor Stoch from Bisseldorf, which gives a very good overview on information of the trivial system. It's also uploaded on the platform, and if you absolutely couldn't understand me, you read the book chapter and everything will be totally clear to you.
 So I think we are not complete yet, we should continue.  I will shortly repeat what I have said before.  First of all, I try to give you a short introduction into evaluation in general.  Then I mentioned a little bit about evaluation of information services  and here you have experience that it's already critical to define what an information service actually is.  And as you have seen, there's much going on and I think this has also big consequences for university libraries  and it will stay the same also the next year.  When you try to evaluate information retrieval systems, you can evaluate single information services,  one possibility or you can for instance evaluate a library as a whole.  What concerns the later, I have shortly introduced you to LibQual,  which is a special set of surfcrawl  and the big advantage of LibQual for instance is that you don't have to make your own questionnaire,  you can participate in a survey in which also other university libraries for instance participate,  you have a comparison with other universities, you have a comparison with yourself over time.  So I think quite interesting if you are maybe head of a university library and if you want to get regular feedback.  Next, I started with the third topic of my presentation, it's about evaluation of information retrieval systems.  And to some degree you can compare it with evaluation of information services in general.  So usually you can evaluate an information retrieval system with regards to three dimensions.  The first one is the perceived quality of an information retrieval system,  so you for instance could use surfcrawl or a similar method.  But the problem is, as also mentioned before, you only get subjective evaluations of the users.  Therefore it makes sense also to do a little bit more objective.  And with regards to information retrieval systems, one additional important dimension is the information, the knowledge covered.  And I tried to explain it for university library, for researchers,  it's very, very important to have all top journals of that field which are covered by the university.  So for me, I need full text of all journals in Scientometrics and to the top journals in Library Information Science.  And since I am also involved in management business administration, also e-chernels in debt disabling.  So that's very important. This refers to knowledge quality and this must be covered by an information retrieval system.  That's one aspect. It refers here to documentary reference units.  And with regards to journals, I told you already, and with regards to databases, of course,  I only use databases which cover all that information, that knowledge.  Also important is the second aspect, not only the covered literature, but also how it is documented in the information retrieval system.  And one important aspect is if the particular information retrieval system, for instance, uses a classification system,  if the information retrieval system, for instance, uses a Tessaurus and so on, because it makes it much easier to find the relevant literature.  And we discussed it before. Google Scholar doesn't offer anything in this regard,  but good databases usually have either a classification system or a Tessaurus or nomenclature,  or even both. In case of web of science, you only have a classification system, but that's not too bad.  And with regards to indexing, it's not too detailed, as I told you, but at least also the other key birds are included,  so you really manage to find the most relevant literature.  And just one remark. Why do you think it is not necessary to do more detailed indexing  or to do consistent indexing in web of science or in other citation databases?  Which other possibility exists to find similar literature, which is even better to do it with classifications and indexing?  With citations. For instance, in web of science, you have great, if you found maybe one document,  you can then ask, please, I want to have similar literature, and on which basis do you think can this similar literature be found?  Not to some degree on other key birds, but what do you think is more effective to find similar literature in a citation database?  From the text to the sound.  Yes, right, right, right, right, right. Yes, that's it.  So in web of science, you can say related records, and then you get the ranking.  Most related record has, for instance, 20 references in common is this publication.  Second most similar publication is this one with 18 identical references and so on.  And usually, to my experience, this is much better to search for similar literature than to use keyword or even indexing.  And that might have been the main reason why this is not so good in web of science, for instance.  Okay, but in principle, it exists and what's very good in web of science are abstracts.  And usually, this is obligatory in web of science that each article has an abstract.  In Google Scholar, it depends if you're lucky and if Google Scholar retrieved this information, then you have an abstract.  But in many cases, you don't have an abstract.  And on the other hand, it's of course free.  Okay, then another important point which is good in both information retrieval systems is update speed.  So usually, as soon as there are new general issues appearing, then you can retrieve them even before.  That's one aspect.  And the third aspect then is which functions do the information retrieval system provide?  In other words, how easy is it to perform the tasks you have to perform with this information retrieval system?  And it starts very basic.  For instance, several information retrieval systems consist of more databases.  In the case of web of science, these are six, I think.  These are three journal certification indexes.  That's a science certification index for sciences, social sciences certification index for social sciences.  And the third one is arts and humanities certification index.  In total, 12,000 journals are covered in web of science, which is quite a lot.  And all articles appearing in them, so I guess 40 million articles, maybe even 50 million articles in the meantime.  In Google Scholar, you don't know it.  I read some figures, approximately 150 million articles.  So much more comprehensive, but also several articles are publicated.  Then Google Scholar also covers books.  Google Scholar also covers e-prints and so on.  Web of science, it's much more regulated.  Okay, these are three databases, and then there are two proceedings databases.  Web of science, not only journal articles are included, but also articles who appear in proceedings.  And there are two databases.  One is the social sciences proceedings index, and the other one, the sciences certification index.  Then there's the books certification index, and you said there might be even several others.  And what's very nice, you can say, since we mainly work, for instance, in social sciences,  I only want to use social sciences certification index and, for instance, conference social sciences certification index.  But you can also search on all databases.  And two of these in parallel are only one, so this is no problem.  It's the same also in EPSCO, for instance.  Okay, that's the general point.  If you really want to work with the database, it goes into more detail.  And one important point is looking for search arguments.  Does the database assist you in finding search terms?  Some databases allow that, for instance, if you're searching the title,  if you type in a name, for instance, information lit, and then it suggests information literacy.  This would be helpful, actually.  It's a kind of browsing a dictionary, but could be even better.  But this is actually meant.  Only a few databases also allow you to find synonyms very easily.  But this is not the case in the Depth of Science, and definitely not the case in Google Scholar, for instance.  Sometimes it would be very helpful, for instance, if you want to make a search in a certain category,  that you can have a look which categories exist.  This is not so easy possible in the Depth of Science.  In the Depth of Science, there are the so-called journal citation reports where you can have a look  to which field, to which discipline a journal is assigned to.  But that's it, actually.  It would be nicer if it's much better supported when you do a search.  In Google Scholar, definitely, you have no assistance at all.  This is, yeah, noted.  For instance, in Epsco, you have a Desaurus.  You even can search in the Desaurus, and you can discover it.  For instance, there's one descriptor called Information Literacy.  You can search or see narrower terms, broader terms, related terms, synonyms, and so on.  In Epsco, this is partly very well assisted.  And why do you think this is important?  That Epsco allows you to search in the Desaurus and to discover the terms which are related to your search.  Yes, it helps you to find the relevant search terms, actually.  For instance, you make a subject search for information literacy and you find, let's say, 100,000 hits.  What could you do?  You could go back to the Desaurus and have a look at their narrower terms.  And then, for instance, one narrower term is Information Search Literacy and so on.  So you could narrow your search using this Desaurus in this case.  Usually, as far as I know, but Thomas Mandel might know it better,  this is not very well supported in most, or not at all supported in most information retrieval systems.  So it would be very helpful.  What I do, for instance, is, before I start a search, I often consult the Desaurus in another database, actually,  only to find the relevant terms.  And then I go back to Web of Science and then here I start with my performance search.  But a very critical phase finding the right words for your search, actually, and also a very important one.  Okay, what are other important functions that which assist you in retrieving the right information?  For instance, output functions.  One function which is very important according to which criteria you can search the results.  Google Scholar, how are the results sorted?  But there are two criteria according to which you can sort the results in Google Scholar.  Yes, one is up to date-ness.  And the second one is, you have heard it several times before.  The other one is relevant.  So you will see it tomorrow.  The group who works with Google Scholar, one is relevant, one is chronological.  The question is what means relevant.  In Google Scholar, you also can see the situations.  And my impression is that the number of situations play a very important role.  But actually, it's not the only criterion.  So my assumption is that it's also very important in Google Scholar if the search term is in the title or if it's in the document.  And another criterion might be how many times the search term is in the document.  But would be an interesting research finding it out.  Okay, but sorting is very important.  In the depth of science, I think you have 10 possibilities to sort the results.  One result is citations.  Another one is relevance.  Another one is alphabetic by author.  Another one is alphabetic by journal, etc., etc.  So much more possibilities.  And that's very important, in particular, if you have found many hits.  Okay, sorting possibilities.  Marking filtering.  What does marking mean?  To mark one document in the research list.  So you make your ticket.  And then in the depth of science, you add on the button add to mark list.  And this way you can collect all documents which are relevant for you.  And what's very nice, then you can print them, you can export them to other files.  So very nice in depth of science.  Can you do marking in Google Scholar?  Actually, I don't know, but the group tomorrow will find out.  What's about filtering?  Filter results.  What could this mean?  Just copy the filter to a document.  Yes, very good.  For a user or...  Yes.  For citations.  Yes, very good.  In the web of science, you have very good filtering functions.  For instance, you have found, let's say, 1000 documents.  And then you can filter, for instance, by document type.  For instance, you only want to consider articles, document type articles.  Or for instance, you have to...  You search for...  You want to buy a book.  You can...  There's another document type, book review.  And you only can filter for book reviews.  So maybe 10, something like that.  That's one possibility.  By document type, by year, by citation, by journal.  I think you might have probably 10 different kinds of filtering possibilities in the web of science.  In Google Scholar, I guess none filtering possibilities.  So very important.  And usually then you can work with the filter document,  search them, output them, print them, etc., etc.  And usually, yes.  For instance, if you have your mark list and all together you have collected, let's say, 20 documents.  Afterwards, you can export them both in Google Scholar and in the web of science.  And important here, which file formats are possible.  For instance, CSV is for spreadsheet programs.  But there are also diverse formats for bibliographic databases and so on.  And in this regard, as far as I know, both information retrieval systems are quite good.  The next one...  Linked to digital version.  Yes.  Google Scholar is, and also the web of science, are bibliographical information retrieval systems.  And so therefore it would be, of course, important to get the full text.  In Google Scholar, if you're lucky, yes, there exists maybe a version of a document where you have also the full text.  For instance, the preprint.  In the web of science, I think there should be a link to a document delivery service where you can buy for one article.  How much is ton average?  Then 20, 30 euro is not seldom, actually.  But usually there are also so-called, I think, link resolver to our university library.  So if the journal in which the article appeared is available, and if you click on the link, you get the full text.  Usually it works in, I would say, 90% of all cases.  But that's an issue which our university library has installed.  Okay, this relates to display and output.  And this is one category where you can compare information retrieval systems.  The next one are so-called push services.  What could this be?  Do you know what the push service is?  Or another name.  I forgot it.  But it's not so important.  But the idea behind is if you work, for instance, on a master thesis, it takes maybe one year or half a year.  And usually you would make the same search once a month, actually.  And usually it's very boring to go into the database and type in the same commands.  So therefore you can save your search command and tell the system if a new document appears in the database,  then you get automatically an email.  So it's called push service.  So there's all the...  Or selective dissemination of information, SDI, is also one term how it's called.  I don't know if Google's call support could be.  Google News supports it, I know, because I regularly get newspapers for a topic which interests me.  Google Scholar, I believe that it's also available in Google Scholar,  but you have to have an account with Google and then you can define it.  Okay, this is a push services.  And then the next one are so-called informatric analysis in Web of Science.  For those of you who know it, for instance, I can do simple ranking commands.  For instance, I search for information literacy.  I found, let's say, 1,000 documents.  Then I can say, analyze, buy author, and then I would get a ranking which author has published most articles about information literacy,  second most, and so on.  And it's very easy.  I want to find out the journals in which most literatories publish.  Then I say analyze, journal or subject, or source, I think.  And then I get a ranking of the journals, or if I want to know if the topic is very up to date, then I say analyze,  and then I select publication year, and then I get a ranking by publication year and so on.  So very helpful.  And Web of Science is one of the only databases which offer this feature, actually.  And I can do the same with citation.  It's called citation report, then I get the ranking of the most cited paper, second most cited paper,  the distribution of citations over the years, and so on.  So also very helpful.  In this regard, many other information retrieval systems are relatively poor.  And in Google Scroll, of course, you don't have any feature at all.  Okay, and there are many other functions which you will discover tomorrow.  For instance, another important feature could what be to help you searching documents.  What would be else important?  But we do it tomorrow in more detail.  For instance, how many fields you can search for in Google Scroll,  for which fields do you search usually?  In which field do you search in Google Scroll?  If you type in, for instance, information literacy, where is it searched in Google Scroll?  You may in the search engine.  Yes, in the whole document.  So information literacy can appear in the title, can appear on page 19 of the article.  So usually the standard assumption is Google Scroll does not distinguish among search fields.  In the app of science, I think you have maybe 20 different search fields.  One is title search, order search, you can search by publication year, by journal, abstract, etc.  So you have many possibilities and the more possibilities you have to refine your search debate actually.  Is it possible to make a title search in Google Scroll?  Yes, how you can search only for documents who have information literacy in the title, what do you have to do?  Yes, and how you can apply it?  Yes, right, right, right, right.  Yes, very good.  You only can do it in the expanded search, in the extended search, not in the simple search.  You must select expanded search, which is a little bit hidden, so Google Scroll is not so much interested to assist you in searching.  And here you can limit your search to, for instance, the title, to the else, what else is possible.  And that's very important.  Also, yes, very good, and I think also publication year.  So you have only three or four fields for which you can search in Google Scholar, in Web of Science you have 20 or 30.  So also a very important criterion.  What else could be very important if you use a search engine?  So number of fields for which you can search.  What else would you...  Yes, topic or subject, but we had it already when I mentioned the thing about Tesaros.  What's about Boolean operators?  Are Boolean operators important?  Yes.  Definitely.  Does Google Scholar support Boolean operators?  Yes.  Yes, you have end, you have or, you have not, but of course you have to know how to use it.  As far as I know, Google Scholar does not support parent.  That gets a little bit complicated, so often you don't get what you want, and this is much better in Web of Science.  You can also set parent, and this is...  You can also use phrase search in both databases.  Are there proximity operators?  So not all of you know it, but...  What does it mean that you can use any Google Scholar?  You can use?  Yeah.  And Google Scholar not, no?  In Google Scholar, I think there's no proximity operator.  You can only use truncation, of course.  Oh, I forgot it.  But you know what I mean, and that's all.  In Web of Science, to some degree, there are proximity operators.  Is it important to use truncation?  Yes.  Does Google Scholar support truncation?  No, not at all.  In Web of Science, you can use the star, which stands for one, two, three characters, which...  And that can be very helpful, etc., etc.  So these are many other search functionalities,  and tomorrow you will get the criteria catalog in which these criteria are all listed,  and you really have to check if the relevant database supports them.  But I think for today this should be enough,  because the next very important thing which we should deal with today  is the effectiveness of an information retrieval system.  What is this?  Effectiveness means... there are actually two indicators which are very important.  The first one is recall.  Recall means if you have performed a search,  then recall tells you how many percent of all relevant documents you really have found.  That's recall.  And precision, what could this mean?  Precision means if you have, for instance, found 20 documents,  and then you usually check it,  and then you find out, let's say, 10 of the documents are relevant  and 10 of the documents are not relevant.  Precision means that 50 percent of the found documents are relevant.  That's it.  So precision means how many of the found documents are relevant,  and recall means how many of all relevant documents you have found.  What you think is difficult when you calculate the recall.  Is it easy to calculate the recall, you think?  Yes.  That's the big problem with recall.  Usually you don't know how much relevant documents are included in the depth of science database.  You only can make a rough estimate, and that's it.  With precision, it's much easier,  because you have research results,  and then you really can identify if it's relevant.  Yes, no, and then you calculate it.  And fallout is the third one, which is not so important.  It's the percentage of non-relevant documents which you have found.  And here an example, which makes it a little bit easier.  Let's say we searched in depth of science,  and let's say the topic is information literacy.  In total we found 12 documents,  and when checking the results,  we found that 8 of them are relevant, and 4 are not relevant.  So with this information, we can calculate the precision,  which is 8 were relevant,  out of 12 documents which were found, so 8 divided by 12,  let me see, by 12 is 67%, not too bad actually.  Actually we have some problems to calculate the recall,  but let's guess that in total 20 documents would have been relevant.  We have found 8 of them, so you have to divide 8 by 20, that's 40%.  And fallout, 4 of our documents were not relevant.  If we assume that in total the database consists of 136 documents,  then we know that non-relevant 116, so 4 divided by 116,  is 3.4% wastage, but that's actually not so important.  So since I talk so much already,  it's highly time that you make a small exercise,  it's also on the platform, but that we don't over stress internet,  I have prepared it for you.  You just have to read it, it deals with an OPAC  of a departmental library on information studies,  total of 10,000 books, and you perform two searches,  a keyword search and then a subject search,  using a Tessaro for instance,  and your duty is to calculate recall and precision for both cases.  One question, this was very good, is I would say that recall  and precision there is some relation between the two indicators.  And here I have indicated this, and we have also just said the example before,  if the precision is very high, what is this to recall?  Is it to recall then also high or is it to recall lower?  So you think if the precision is high, then also lower?  Yes, very good.  So usually there's a negative precision, if I have 100% precision,  it is very likely that the recall is very low.  And this was just example before, if I search for the exact type of a publication,  the precision might be 100%, but I have only found one relevant document out of 1000 maybe.  So high precision, low recall.  If the recall is very high, how will the precision be?  If I try to get each document which is relevant,  will it be low?  Will be low, yes, because it's in version actually,  I have my, the query must be very wide,  and as a consequence the precision will suffer.  For example we just saw it's in the dynamon, I mean it's not always the case.  Yes, yes, very true.  Must not be...  It's not a double implication, so it's a one-way implication.  Yes, of course, but...  If that also then...  Right, right, right.  But actually if you look at the other way around, it's also a reverse relation.  Okay, another question is...  I would say, not in all cases, both precision must be high and recall must be high.  For, let's for instance, took the example, you work on your PhD thesis,  and if you work your PhD thesis you should really read most or nearly all of the relevant literature.  So what would you prefer in this case, a high precision in the search or a high recall in the search?  High recall.  High recall, because you want to...  For all of them.  Yes, yes.  For a PhD thesis it's very important that...  And yes, you have to scan, and if you scan the search maybe 90% is not relevant, but you get more relevant.  And for the PhD it's important that you really have a good knowledge of the whole literature.  So if you write on your PhD thesis, a high recall is very important.  Another example, you want to get... you participate in the course,  and you want to get some basic knowledge about the course.  Is a high recall very important, or is a high precision very important?  If high precision, it's enough if maybe even if Wikipedia is already enough,  and if not then maybe a search in the lab of science and you scan through a few documents, and that's fine actually.  So in this case, high precision is more important.  What's about the pattern search?  Is a high recall very important or a high precision?  For instance, a company develops a new product, and it's very critical because it costs, let's say, 100,000 euro.  Developing costs, the problem is if at the end you find out that there is already a patent on that topic,  then you can throw away 100,000 euro at developing costs.  So what is important here? High precision or a high recall?  We still want to scan through all the documents.  Yes, yes, high recall, yes.  So you see...  I think that we often prefer higher precision because I wouldn't...  In my own reason, that's why I also said high precision because I don't want to search like 200 documents.  I mean, I have like a medium, I guess, 260, you know?  I don't want extremes.  You're right, but you have to make compromises in reality.  For PhDs, we would make compromises.  For patents, not because so much money is involved.  But in that case, you would pay 20 guys.  Yes, yes, that's it.  And that was actually the sense of why I noticed this.  There is no clear response.  You really have to decide what you strive for, actually.  But I think you have understood it.  So in particular, the concept of recall is a little bit theoretical  because you never know how many documents there are.  It gives you some understanding about search of literature.  Okay, another question is, can you use recall and precision to compare also to information retrieval systems?  In this case, Web of Science and Google Scholar.  Is this possible?  Yes, because if, for instance, Web of Science, and you have seen it before,  offers many possibilities to search for literature, not Google.  And therefore, usually recall, but also precision would be higher in Web of Science than in Google Scholar.  So you have two curves like this.  And as you have seen, they are high precision, low recall, high recall, low precision.  But in both cases, Web of Science should be much better than Google Scholar, for instance.  So you could even compare information retrieval systems using these two indicators.  Clear?  Yes, and problems, I have already indicated this.  Recall is usually not known.  If you are very well experienced on patents, for instance, you know approximately.  If you consult a librarian, for instance, the guy who made the guided tour,  he may know how many documents approximately there are.  That's one thing.  The next question is, what do you do with search engines?  Google, for instance.  Can you calculate recall with Google?  Impossible.  Can you calculate precision with Google?  You could, you say, for instance, you only consider the first 50 documents,  and you calculate the precision for the first 50 documents.  And if you compare, for instance, Google with Bing with Yahoo,  then you get actually an approximation.  And there's even one possibility to calculate the so-called relative recall.  If you, for instance, compare three search engines,  and if you say you only consider the first 500 documents,  and you look for them in Google,  and then you look for them in Yahoo,  and you find in Yahoo documents under the first 500,  which were not considered in Google,  you can add this to the total number of relevant documents,  and you do the same with Bing.  And that's the way you could calculate the kind of relative recall.  And I think it would not be uninteresting,  because you see that there are quite big differences also between different search engines.  I think the big problem is monopoly of Google.  I told you already 90% use only Google and only rely on Google,  but you would see that the differences are more or less big,  in particular with regards to recall, I guess.  So it would be interesting,  and I guess there are also several studies in literature about this topic.  What is another problem with recall and precision?  Actually, you say a document is either relevant or not relevant.  What is your experience with all of you who have written the seminar papers?  If you write a seminar paper,  is each document the same relevant or are there differences?  There are differences.  There's one document which is most relevant,  another document second most.  It's relatively easy to make a ranking,  and this is not at all considered here, of course.  And actually, therefore, it's possible,  they developed a method how to calculate relative precision,  and I will show it to you approximately how it works.  You don't have to understand it.  It was mainly important for me to show you that it's a shortcoming.  But the main idea is that you only consider, for instance,  the first ten hits, and you order them according to relevance.  And here you searched, for instance, for information management,  and the first hit was relevant, the second not.  The second most relevant was number three, it was relevant.  The third relevant document was ranked under the sixth position,  and then these two.  And then, next, you calculate the relative precision.  So this one is relevant, one divided by one is one.  And here, if you got the first two hits,  out of the two hits, one is relevant, out of two,  so one divided by two is 0.5.  Here, if you consider only the first three, two are relevant,  divide by three is 0.67.  Two out of four are relevant, two divided by four is 0.5.  And you do it, you continue that way, and you get these numbers here,  and then you add those values, there you have a yes,  one plus 0.6 plus this one plus this one plus this one,  divided by five gives a relative precision of 0.62.  And this is another query, and you do the same,  so the first one is not relevant,  one divided by, zero divided by one is zero.  Here, one out of two is relevant, one divided by two is 0.2, and so on and so on.  And so, you add the precision values, where you have a yes, 0.5 plus this plus this,  divided by three is 0.44.  And then, if you want, you can add these two  divided by two gives an average precision of 0.5.  But it's not so important.  But the main difference to the other method is that the very relevant documents  get the higher weight, so one, for instance, or 0.5,  and the documents which are further behind get lower weight,  and somehow you can consider the fact that not all documents have the same value.  You can, difficult?  But the main idea is that the most relevant documents get higher numbers,  and the most relevant document gets one.  And this is the fifth, so there's one, two, three, four.  The fifth relevant document only gets the weight 0.5.  So you give not the same weight for all relevant documents,  you really try to distinguish.  That's the main idea.  And then you do some calculation, and of course you get other values.  And usually, for instance, if you have two searches,  and you have five relevant hits under the first five positions,  you have much higher values than if you have your five relevant hits here,  and you really could say that this precision will be much better.  But forget it, it's not so important.  It should only illustrate that in the normal calculation,  there's no distinction between more and less relevant hits.  You only say relevant, not relevant.  And I don't know who invented this,  and I cannot imagine that it's used in practice,  but it helps you to make clear that there is an order of relevance, of course.  Okay, other questions?  No.  And that's it.  And yes, I mainly referred to the book chapter by Professor Stoch from Bisseldorf,  which gives a very good overview on information of the trivial system.  It's also uploaded on the platform,  and if you absolutely couldn't understand me,  you read the book chapter and everything will be totally clear to you.
Evaluation-2_0_5975.png Evaluation-2_5976_11279.png Evaluation-2_11280_11339.png Evaluation-2_11340_11399.png Evaluation-2_11400_11519.png Evaluation-2_11520_11711.png Evaluation-2_11712_12179.png Evaluation-2_12180_12299.png Evaluation-2_12300_12491.png Evaluation-2_12492_12923.png Evaluation-2_12924_13115.png Evaluation-2_13116_14987.png Evaluation-2_14988_16235.png Evaluation-2_16236_16523.png Evaluation-2_16524_16859.png Evaluation-2_16860_17795.png Evaluation-2_17796_18887.png Evaluation-2_18888_19163.png Evaluation-2_19164_19799.png Evaluation-2_19800_20759.png Evaluation-2_20760_21059.png Evaluation-2_21060_35459.png Evaluation-2_35460_46295.png Evaluation-2_46296_46379.png Evaluation-2_46380_49823.png Evaluation-2_49824_57839.png Evaluation-2_57840_59387.png Evaluation-2_59388_63959.png Evaluation-2_63960_64235.png Evaluation-2_64236_64283.png Evaluation-2_64284_71411.png Evaluation-2_71412_72085.png