Towards searching as a learning process : A review of current perspectives and future directions <s> Inferring the presence and nature of learning-related tasks from search logs <s> Previous work on understanding user web search behavior has focused on how people search and what they are searching for, but not why they are searching. In this paper, we describe a framework for understanding the underlying goals of user searches, and our experience in using the framework to manually classify queries from a web search engine. Our analysis suggests that so-called navigational" searches are less prevalent than generally believed while a previously unexplored "resource-seeking" goal may account for a large fraction of web searches. We also illustrate how this knowledge of user search goals might be used to improve future web search engines. <s> BIB001 </s> Towards searching as a learning process : A review of current perspectives and future directions <s> Inferring the presence and nature of learning-related tasks from search logs <s> The Internet is the largest source of information in the world. Search engines help people navigate the huge space of available data in order to acquire new skills and knowledge. In this paper, we present an in-depth analysis of sessions in which people explicitly search for new knowledge on the Web based on the log files of a popular search engine. We investigate within-session and cross-session developments of expertise, focusing on how the language and search behavior of a user on a topic evolves over time. In this way, we identify those sessions and page visits that appear to significantly boost the learning process. Our experiments demonstrate a strong connection between clicks and several metrics related to expertise. Based on models of the user and their specific context, we present a method capable of automatically predicting, with good accuracy, which clicks will lead to enhanced learning. Our findings provide insight into how search engines might better help users learn as they search. <s> BIB002 </s> Towards searching as a learning process : A review of current perspectives and future directions <s> Inferring the presence and nature of learning-related tasks from search logs <s> Current research on Web search has focused on optimizing and evaluating single queries. However, a significant fraction of user queries are part of more complex tasks [Jones and Klinkner 2008] which span multiple queries across one or more search sessions [Liu and Belkin 2010; Kotov et al. 2011]. An ideal search engine would not only retrieve relevant results for a user's particular query but also be able to identify when the user is engaged in a more complex task and aid the user in completing that task [Morris et al. 2008; Agichtein et al. 2012]. Toward optimizing whole-session or task relevance, we characterize and address the problem of intrinsic diversity (ID) in retrieval [Radlinski et al. 2009], a type of complex task that requires multiple interactions with current search engines. Unlike existing work on extrinsic diversity [Carbonell and Goldstein 1998; Zhai et al. 2003; Chen and Karger 2006] that deals with ambiguity in intent across multiple users, ID queries often have little ambiguity in intent but seek content covering a variety of aspects on a shared theme. In such scenarios, the underlying needs are typically exploratory, comparative, or breadth-oriented in nature. We identify and address three key problems for ID retrieval: identifying authentic examples of ID tasks from post-hoc analysis of behavioral signals in search logs; learning to identify initiator queries that mark the start of an ID search task; and given an initiator query, predicting which content to prefetch and rank. <s> BIB003 </s> Towards searching as a learning process : A review of current perspectives and future directions <s> Inferring the presence and nature of learning-related tasks from search logs <s> Online services rely on unique identifiers of machines to tailor offerings to their users. An implicit assumption is made that each machine identifier maps to an individual. However, shared ma-chines are common, leading to interwoven search histories and noisy signals for applications such as personalized search and ad-vertising. We present methods for attributing search activity to individual searchers. Using ground truth data for a sample of almost four million U.S. Web searchers-containing both machine identifiers and person identifiers-we show that over half of the machine identifiers comprise the queries of multiple people. We characterize variations in features of topic, time, and other aspects such as the complexity of the information sought per the number of searchers on a machine, and show significant differences in all measures. Based on these insights, we develop models to accurately estimate when multiple people contribute to the logs ascribed to a single machine identifier. We also develop models to cluster search behavior on a machine, allowing us to attribute historical data accurately and automatically assign new search activity to the correct searcher. The findings have implications for the design of applications such as personalized search and advertising that rely heavily on machine identifiers to custom-tailor their services. <s> BIB004 </s> Towards searching as a learning process : A review of current perspectives and future directions <s> Inferring the presence and nature of learning-related tasks from search logs <s> Web searchers sometimes struggle to find relevant information. Struggling leads to frustrating and dissatisfying search experiences, even if searchers ultimately meet their search objectives. Better understanding of search tasks where people struggle is important in improving search systems. We address this important issue using a mixed methods study using large-scale logs, crowd-sourced labeling, and predictive modeling. We analyze anonymized search logs from the Microsoft Bing Web search engine to characterize aspects of struggling searches and better explain the relationship between struggling and search success. To broaden our understanding of the struggling process beyond the behavioral signals in log data, we develop and utilize a crowd-sourced labeling methodology. We collect third-party judgments about why searchers appear to struggle and, if appropriate, where in the search task it became clear to the judges that searches would succeed (i.e., the pivotal query). We use our findings to propose ways in which systems can help searchers reduce struggling. Key components of such support are algorithms that accurately predict the nature of future actions and their anticipated impact on search outcomes. Our findings have implications for the design of search systems that help searchers struggle less and succeed more. <s> BIB005
How do people use web search for learning tasks? As described earlier, learning-related search tasks can be complex, requiring multiple queries and significant time spent searching and browsing. There have been few published studies of how users engage in such tasks based on commercial search engine data. However, one study by Bailey et al.  described a task taxonomy of web searches -based on analysis by human judges of 4 months of search log data from Google, Yahoo and Bing -that included some learning-related tasks, including topic exploration, fact-finding and procedural learning. A second study by Eickhoff et al. BIB002 included an automated log analysis that focused specifically on the fraction of sessions that potentially involved a procedural or declarative knowledge intent. The relevant results from both studies are summarized in Table 2 . Both studies found that learning-related tasks accounted for a non-trivial proportion of all search sessions, as shown in the 'percentage of sessions' column in Table 2 . In line with the task study by Bailey et al.  , Eickhoff et al. BIB002 found that about 3% of sessions had an explicit knowledge-acquisition intent that was either procedural or declarative. Moreover, reflecting the complex, multi-step nature of many learning-related tasks, while the absolute percentage of total sessions for the learning tasks in Table 2 is around 20%, these tasks accounted for a disproportionally larger fraction of time spent searching. Raman et al. BIB003 had a similar finding for intrinsically diverse (ID) search tasks, which are exploratory web searches whose intent is to learn about multiple aspects of a specific topic. Given that ID sessions require multiple queries, Raman et al. hypothesized that ID sessions accounted for a disproportionately larger fraction of time spent searching by all users. To test this, they estimated the time a user spent in a session by the elapsed time from the first query to the last action (i.e. query or click). Sessions with a single query and no clicks were assigned a constant duration of 5 s. (The time in session includes the whole session once an ID task was identified in that session.) While ID sessions with at least two distinct exploratory aspects represented only 1.0% of all sessions, they accounted for more than 4.3% of total time spent searching, showing the significant role that ID sessions play in overall search activity. The study by Raman et al. did not focus on more complex ID information seeking, such as tasks that span multiple sessions, and so the true percentage associated with ID tasks is likely to be larger. These studies reflect the two basic approaches -manual vs automated -that are used to infer user intent from search logs. Manual approaches rely on either self-reported intent indicators generated by the users themselves (e.g. via an affordance in the search user interface), or intent descriptions generated by human judges who try to deduce intent from a post-hoc analysis of search log behaviour data. The latter was the approach used by Bailey et al.  , following an earlier method developed by Rose and Levinson BIB001 , to determine intent categories (e.g. 'Discover more information about a specific topic'). The log behaviour data that is used as the basis for these judgments includes (a) the queries issued by a user in a session, (b) the results returned by the search engine, (c) the results clicked by the user, and (d) further queries or other actions taken by the user. This data also potentially includes additional resources such as the actual content of the pages returned in the search results, along with meta-data computed from this content, for example, the page's reading level or topic category. On the other hand, automated approaches BIB002 BIB003 typically rely on a representative sample of user behaviour data for which some information about intent has been identified by human judges, but then apply this initial set of labelled data to train a statistical model that attempts to automatically classify intent for future instances of user behaviour. In the case of learning-related search, one approach that automated methods have used to operationalize indicators of learning intent is to use the type of knowledge offered by a website's educational content as a proxy for a learning intent label. For example, in Eickhoff et al. BIB002 a user's visitation of a site specializing in procedural 'how-to' descriptions (e.g. ehow.com) was assumed to indicate a procedural learning intent, while Wikipedia visits were assumed to be indicative of intent to acquire declarative 'fact-based' knowledge. These implicit intent labels were then used as the basis for characterizing features of queries and patterns of search behaviour that were associated with each type of learning intent. Similarly, in Raman et al. the nature and breadth of topic categories for the content covered during a search is used as a proxy indicator of the user's desire to explore different aspects of a topic. Other examples include using reading-level features associated with motivation (Section 4.2) and features of site topic focus and difficulty to measure change in expertise over time (Section 4.3). While these operational choices for learning indicators lead to richer representations of user behaviour while learning, they are not without their limitations. First, the use of proxy indicators of learning, such as representing a user's content intent by the content type they read, assumes that what the user is finding is what they intended to find, and that what they intend to find represents what they intend to learn. These assumptions often do not hold in practice: users may have a variety of goals when looking at all or part of a particular page. Users may also be restricted by the results of an ineffective search engine or choice of search strategies for a given learning goal, leading to less relevant content being Rows 1-5 from Bailey et al.  and rows 6-7 from Eickhoff et al. BIB002 . A blank (-) cell indicates specific numbers were not reported for that intent/task. perused, creating an inaccurate set of proxy learning indicators. We discuss future directions researchers are pursuing for overcoming some of these issues in Section 5.4. Second, search log data may provide a limited window on what the user is seeing, paying attention to and expressing about their need. In many cases, for example, the search log behaviour of a user who is struggling can appear to be very similar to that of a user who is successful in their efforts BIB005 . In such scenarios, researchers have successfully developed and applied classifiers that detect characteristic query reformulation patterns of struggling users in order to distinguish success from frustration BIB005 . Finally, an additional set of limitations stems from the nature of search log data itself. Even assuming a perfect ability to infer intent from data, obtaining complete and correct user behaviour data for individuals can be problematic. Attribution of search activity to a specific user is usually performed using an auto-generated unique identifier based on Web cookies or a browser instance. However, multiple people searching on the same machine is common: a recent study by White et al. estimated that 56% of unique identifiers in a search log actually comprised activity from multiple searchers BIB004 . Conversely, a single user's log activity may be split across machines or devices, making a complete picture of that individual's behaviour more difficult to obtain. This problem is being addressed in part by the development of automatic methods to attribute activity to the correct user BIB004 . In short, there are many sources of potential noise and uncertainty in attempting to infer the latent intent or mental state of the user from a limited set of online observations. While this is indeed a difficult challenge, the examples cited above, along with future work discussed below, show that researchers are making progress in using machine learning and deeper natural language processing approaches to at least identify high-precision subsets of cases where algorithms can achieve an effective level of confidence in their predictions of learning intent and progress.