We propose the PeerRank method for peer assessment. This constructs a grade for an agent based on the grades proposed by the agents evaluating the agent. Since the grade of an agent is a measure of their ability to grade correctly, the PeerRank method weights grades by the grades of the grading agent. The PeerRank method also provides an incentive for agents to grade correctly. As the grades of an agent depend on the grades of the grading agents, and as these grades themselves depend on the grades of other agents, we define the PeerRank method by a fixed point equation similar to the PageRank method for ranking web-pages. We identify some formal properties of the PeerRank method (for example, it satisfies axioms of unanimity, no dummy, no discrimination and symmetry), discuss some examples, compare with related work and evaluate the performance on some synthetic data. Our results show considerable promise, reducing the error in grade predictions by a factor of 2 or more in many cases over the natural baseline of averaging peer grades.
There are a number of closely related problems to ours in the social choice literature. For example, Holzman and Moulin have studied a related problem in which a set of agents wish to select one amongst them to receive a prize @cite_2 . A fundamental assumption of this work is that nominations are impartial: your message never influences whether you win the prize. In our setting, such an assumption is less appropriate. We want your evaluation of the work of another agent to influence your evaluation. There are several reasons behind this change. First, your ability to evaluate the work of other agents measures in part your command of the subject being examined. Second, you will be incentivized to grade accurately by a better final grade. If your message cannot influence your evaluation, then you have no incentive to provide good evaluations. For this reason, even if we extend the sort of methods proposed by Holzman and Moulin to the task of ranking, the starting assumptions are very different.
@cite_2
1558752712
A group of peers must choose one of them to receive a prize; everyone cares only about winning, not about who gets the prize if someone else. An award rule is impartial if one's message never influences whether or not one wins the prize. We explore the consequences of impartiality when each agent nominates a single (other) agent for the prize. @PARASPLIT On the positive side, we construct impartial nomination rules where both the influence of individual messages and the requirements to win the prize are not very different across agents. Partition the agents in two or more districts, each of size at least 3, and call an agent a local winner if he is nominated by a majority of members of his own district; the rule selects a local winner with the largest support from nonlocal winners, or a fixed default agent in case there is no local winner. @PARASPLIT On the negative side, impartiality implies that ballots cannot be processed anonymously as in plurality voting. Moreover, we cannot simultaneously guarantee that the winner always gets at least one nomination, and that an agent nominated by everyone else always wins. We study the problem of selecting a member of a set of agents based on impartial nominations by agents from that set. The problem was studied previously by and by Holzman and Moulin and has important applications in situations where representatives are selected from within a group or where publishing or funding decisions are made based on a process of peer review. Our main result concerns a randomized mechanism that in expectation selects an agent with at least half the maximum number of nominations. Subject to impartiality, this is best possible. We examine strategy-proof elections to select a winner amongst a set of agents, each of whom cares only about winning. This impartial selection problem was introduced independently by Holzman and Moulin [5] and [1]. Fischer and Klimm [4] showed that the permutation mechanism is impartial and ( )-optimal, that is, it selects an agent who gains, in expectation, at least half the number of votes of the most popular agent. Furthermore, they showed the mechanism is ( 7 12 )-optimal if agents cannot abstain in the election. We show that a better guarantee is possible, provided the most popular agent receives at least a large enough, but constant, number of votes. Specifically, we prove that, for any e > 0, there is a constant N e (independent of the number n of voters) such that, if the maximum number of votes of the most popular agent is at least N e then the permutation mechanism is (( 3 4 - ) )-optimal. This result is tight. We propose the PeerRank method for peer assessment. This constructs a grade for an agent based on the grades proposed by the agents evaluating the agent. Since the grade of an agent is a measure of their ability to grade correctly, the PeerRank method weights grades by the grades of the grading agent. The PeerRank method also provides an incentive for agents to grade correctly. As the grades of an agent depend on the grades of the grading agents, and as these grades themselves depend on the grades of other agents, we define the PeerRank method by a fixed point equation similar to the PageRank method for ranking web-pages. We identify some formal properties of the PeerRank method (for example, it satisfies axioms of unanimity, no dummy, no discrimination and symmetry), discuss some examples, compare with related work and evaluate the performance on some synthetic data. Our results show considerable promise, reducing the error in grade predictions by a factor of 2 or more in many cases over the natural baseline of averaging peer grades. In many settings, an effective way of evaluating objects of interest is to collect evaluations from dispersed individuals and to aggregate these evaluations together. Some examples are categorizing online content and evaluating student assignments via peer grading. For this data science problem, one challenge is to motivate participants to conduct such evaluations carefully and to report them honestly, particularly when doing so is costly. Existing approaches, notably peer-prediction mechanisms, can incentivize truth telling in equilibrium. However, they also give rise to equilibria in which agents do not pay the costs required to evaluate accurately, and hence fail to elicit useful information. We show that this problem is unavoidable whenever agents are able to coordinate using low-cost signals about the items being evaluated (e.g., text labels or pictures). We then consider ways of circumventing this problem by comparing agents' reports to ground truth, which is available in practice when there exist trusted evaluators---such as teaching assistants in the peer grading scenario---who can perform a limited number of unbiased (but noisy) evaluations. Of course, when such ground truth is available, a simpler approach is also possible: rewarding each agent based on agreement with ground truth with some probability, and unconditionally rewarding the agent otherwise. Surprisingly, we show that the simpler mechanism achieves stronger incentive guarantees given less access to ground truth than a large set of peer-prediction mechanisms. Emotion is a cognitive mechanism that directs an agent's thoughts and attention to what is relevant, important, and significant. Such a mechanism is crucial for the design of resource-bounded agents that must operate in highly-dynamic, semi-predictable environments and which need mechanisms for allocating their computational resources efficiently. The aim of this work is to propose a logical analysis of emotions and their influences on an agent's behavior. We focus on four emotion types (viz., hope, fear, joy, and distress) and provide their logical characterizations in a modal logic frame-work. As the intensity of emotion is essential for its influence on an agent's behavior, the logic is devised to represent and reason about graded beliefs, graded goals and intentions. The belief strength and the goal strength determine the intensity of emotions. Emotions trigger different types of coping strategy which are aimed at dealing with emotions either by forming or revising an intention to act in the world, or by changing the agent's interpretation of the situation (by changing beliefs or goals). We develop a formal model of opinion polls in elections and study how they influence the voting behaviour of the participating agents, and thereby election outcomes. This approach is particularly relevant to the study of collective decision making by means of voting in multiagent systems, where it is reasonable to assume that we can precisely model the amount of information available to agents and where agents can be expected to follow relatively simple rules when adjusting their behaviour in response to polls. We analyse two settings, one where a single agent strategises in view of a single poll, and one where multiple agents repeatedly update their voting intentions in view of a sequence of polls. In the single-poll setting we vary the amount of information a poll provides and examine, for different voting rules, when an agent starts and stops having an incentive to manipulate the election. In the repeated-poll setting, using both analytical and experimental methods, we study how the properties of different voting rules are affected under different sets of assumptions on how agents will respond to poll information. Together, our results clarify under which circumstances sharing information via opinion polls can improve the quality of election outcomes and under which circumstances it may have negative effects, due to the increased opportunities for manipulation it provides. Negotiation is essential in settings where autonomous agents have conflicting interests and a desire to cooperate. For this reason, mechanisms in which agents exchange potential agreements according to various rules of interaction have become very popular in recent years as evident, for example, in the auction and mechanism design community. However, a growing body of research is now emerging which points out limitations in such mechanisms and advocates the idea that agents can increase the likelihood and quality of an agreement by exchanging arguments which influence each others' states. This community further argues that argument exchange is sometimes essential when various assumptions about agent rationality cannot be satisfied. To this end, in this article, we identify the main research motivations and ambitions behind work in the field. We then provide a conceptual framework through which we outline the core elements and features required by agents engaged in argumentation-based negotiation, as well as the environment that hosts these agents. For each of these elements, we survey and evaluate existing proposed techniques in the literature and highlight the major challenges that need to be addressed if argument-based negotiation research is to reach its full potential. This paper focuses on new challenges in influence maximization inspired by non-profits' use of social networks to effect behavioral change in their target populations. Influence maximization is a multiagent problem where the challenge is to select the most influential agents from a population connected by a social network. Specifically, our work is motivated by the problem of spreading messages about HIV prevention among homeless youth using their social network. We show how to compute solutions which are provably close to optimal when the parameters of the influence process are unknown. We then extend our algorithm to a dynamic setting where information about the network is revealed at each stage. Simulation experiments using real world networks collected by the homeless shelter show the advantages of our approach. Procuring multiple agents with different ability levels to independently solve the same task is common in labor markets, crowdsourcing environments and research and development projects due to two reasons: some agents may fail to provide a satisfactory solution, and the redundancy increases the quality of the best solution found. However, incentivizing large number of agents to compete for one task is difficult; agents need fair ex-ante guaranteed payoffs that consider their ability levels and failure rates to exert efforts. We model such domains as a cooperative game called the Max-Game, where each agent has a weight representing its ability level, and the value of an agent coalition is the maximal weight of the agents in the coalition. When agents may fail, we redefine the value of a coalition as the expected maximal weight of its surviving members. We analyze the core, the Shapley value, and the Banzhaf index as methods of payoff division. Surprisingly, the latter two, which are usually computationally hard, can be computed in polynomial time. Finally, we initiate the study of a new form of sabotage where agents may be incentivized to influence the failure probabilities of their peers, and show that no such incentive is present in a restricted case of Max-Games. Two or more mobile entities, called agents or robots, starting at distinct initial positions, have to meet. This task is known in the literature as rendezvous. Among many alternative assumptions that have been used to study the rendezvous problem, two most significantly influence the methodology appropriate for its solution. The first of these assumptions concerns the environment in which the mobile entities navigate: it can be either a terrain in the plane, or a network modeled as an undirected graph. The second assumption concerns the way in which the entities move: it can be either deterministic or randomized. In this article, we survey results on deterministic rendezvous in networks. © 2012 Wiley Periodicals, Inc. NETWORKS, 2012 © 2012 Wiley Periodicals, Inc. Abstract Intelligent systems often depend on data provided by information agents, for example, sensor data or crowdsourced human computation. Providing accurate and relevant data requires costly effort that agents may not always be willing to provide. Thus, it becomes important not only to verify the correctness of data, but also to provide incentives so that agents that provide high-quality data are rewarded while those that do not are discouraged by low rewards. We cover different settings and the assumptions they admit, including sensing, human computation, peer grading, reviews, and predictions. We survey different incentive mechanisms, including proper scoring rules, prediction markets and peer prediction, Bayesian Truth Serum, Peer Truth Serum, Correlated Agreement, and the settings where each of them would be suitable. As an alternative, we also consider reputation mechanisms. We complement the game-theoretic analysis with practical examples of applications in prediction platforms, community sensin... We adopt a utilitarian perspective on social choice, assuming that agents have (possibly latent) utility functions over some space of alternatives. For many reasons one might consider mechanisms, or social choice functions, that only have access to the ordinal rankings of alternatives by the individual agents rather than their utility functions. In this context, one possible objective for a social choice function is the maximization of (expected) social welfare relative to the information contained in these rankings. We study such optimal social choice functions under three different models, and underscore the important role played by scoring functions. In our worst-case model, no assumptions are made about the underlying distribution and we analyze the worst-case distortion---or degree to which the selected alternative does not maximize social welfare---of optimal social choice functions. In our average-case model, we derive optimal functions under neutral (or impartial culture) distributional models. Finally, a very general learning-theoretic model allows for the computation of optimal social choice functions (i.e., that maximize expected social welfare) under arbitrary, sampleable distributions. In the latter case, we provide both algorithms and sample complexity results for the class of scoring functions, and further validate the approach empirically. CrowdGrader is a system that lets students submit and collaboratively review and grade homework. We describe the techniques and ideas used in CrowdGrader, and report on the experience of using CrowdGrader in disciplines ranging from Computer Science to Economics, Writing, and Technology. In CrowdGrader, students receive an overall crowd-grade that reflects both the quality of their homework, and the quality of their work as reviewers. This creates an incentive for students to provide accurate grades and helpful reviews of other students' work. Instructors can use the crowd-grades as final grades, or fine-tune the grades according to their wishes. Our results on seven classes show that students actively participate in the grading and write reviews that are generally helpful to the submissions' authors. The results also show that grades computed by CrowdGrader are sufficiently precise to be used as the homework component of class grades. Students report that the main benefits in using CrowdGrader are the quality of the reviews they receive, and the ability to learn from reviewing their peers' work. Instructors can leverage peer learning in their classes, and easily handle homework evaluation in large classes. Previous studies on informational cascades have stressed the importance of informational social influences in decision-making. When people use the product evaluations of others to indicate product quality on the Internet, online herd behavior occurs. This work presents four studies examining herd behavior of online book purchasing. The first two studies addressed how two cues frequently found on the Internet, i.e., star ratings and sales volume, influence consumer online product choices. The last two studies investigated the relative effectiveness of different recommendation sources. The experimental results demonstrated that subjects use the product evaluations and choices of others as cues in making purchasing book decisions on the Internet bookstore. Additionally, recommendations of other consumers exerted a greater influence on subject choices than recommendations of an expert. Finally, recommendations from recommender system influenced online consumer choices more than those from website owners. The results and implications of this research are discussed. It is important to investigate the influence of life-like agents interacting with a user since some studies suggest thai life-like agents can "persuade" people, in other words, they have potential to change people's attitude and behavior. In this study, the influence of overheard communication (OC), one of famous persuasion techniques, by life-like agents toward online shopping Web site users was examined, since the OC by people often changes attitude of receivers. An experiment to compare the effect of OC by two life-like agents (a persuader agent and a persuadee agent) with regular communication by one persuader agent were conducted. The result of this experiment implied that even the OC by life-like agents could promote Web site users' online shopping purchase likelihood. Moreover, despite impression to a life-like agent as a persuader was not strongly affected by OC, attractiveness toward a persuader agent evaluated by participants was positively correlated with their purchase likelihood. These results suggest the effectiveness of the OC by life-like agents and the new direction of studies of social response to life-like agents, from the viewpoints of presence, gaze, and appearance of lifelike agents. The need for automated grading tools for essay writing and open-ended assignments has received increasing attention due to the unprecedented scale of Massive Online Courses (MOOCs) and the fact that more and more students are relying on computers to complete and submit their school work. In this paper, we propose an efficient memory networks-powered automated grading model. The idea of our model stems from the philosophy that with enough graded samples for each score in the rubric, such samples can be used to grade future work that is found to be similar. For each possible score in the rubric, a student response graded with the same score is collected. These selected responses represent the grading criteria specified in the rubric and are stored in the memory component. Our model learns to predict a score for an ungraded response by computing the relevance between the ungraded response and each selected response in memory. The evaluation was conducted on the Kaggle Automated Student Assessment Prize (ASAP) dataset. The results show that our model achieves state-of-the-art performance in 7 out of 8 essay sets. In the on-line Explore and Exploit literature, central to Machine Learning, a central planner is faced with a set of alternatives, each yielding some unknown reward. The planner's goal is to learn the optimal alternative as soon as possible, via experimentation. A typical assumption in this model is that the planner has full control over the experiment design and implementation. When experiments are implemented by a society of self-motivated agents the planner can only recommend experimentation but has no power to enforce it. (JPE, 2014) introduce the first study of explore and exploit schemes that account for agents' incentives. In their model it is implicitly assumed that agents do not see nor communicate with each other. Their main result is a characterization of an optimal explore and exploit scheme. In this work we extend (JPE, 2014) by adding a layer of a social network according to which agents can observe each other. It turns out that when observability is factored in the scheme proposed by (JPE, 2014) is no longer incentive compatible. In our main result we provide a tight bound on how many other agents can each agent observe and still have an incentive-compatible algorithm and asymptotically optimal outcome. More technically, for a setting with N agents where the number of nodes with degree greater than N^alpha is bounded by N^beta and 2*alpha+beta < 1 we construct incentive-compatible asymptotically optimal mechanism. The bound 2*alpha+beta < 1 is shown to be tight. The actions of a rational agent reveal information about its mental states. These inferred mental states, particularly the agent’s intentions, play an important role in the evaluation of moral permissibility. While previous computational models have shown that beliefs and desires can be inferred from behavior under the assumption of rational action they have critically lacked a third mental state, intentions. In this work, we develop a novel formalism for intentions and show how they can be inferred as counterfactual contrasts over influence diagrams. This model is used to quantitatively explain judgments about intention and moral permissibility in classic and novel trolley problems. All agents being informed and the virtual leader traveling at a constant velocity are the two critical assumptions seen in the recent literature on flocking in multi-agent systems. Under these assumptions, Olfati-Saber in a recent IEEE Transactions on Automatic Control paper proposed a flocking algorithm which by incorporating a navigational feedback enables a group of agents to track a virtual leader. This paper revisits the problem of multi-agent flocking in the absence of the above two assumptions. We first show that, even when only a fraction of agents are informed, the Olfati-Saber flocking algorithm still enables all the informed agents to move with the desired constant velocity, and an uninformed agent to also move with the same desired velocity if it can be influenced by the informed agents from time to time during the evolution. Numerical simulation demonstrates that a very small group of the informed agents can cause most of the agents to move with the desired velocity and the larger the informed group is the bigger portion of agents will move with the desired velocity. In the situation where the virtual leader travels with a varying velocity, we propose modification to the Olfati-Saber algorithm and show that the resulting algorithm enables the asymptotic tracking of the virtual leader. That is, the position and velocity of the center of mass of all agents will converge exponentially to those of the virtual leader. The convergent rate is also given.