In many areas of data mining, data is collected from humans beings. In this contribution, we ask the question of how people actually respond to ordinal scales. The main problem observed is that users tend to be volatile in their choices, i.e. complex cognitions do not always lead to the same decisions, but to distributions of possible decision outputs. This human uncertainty may sometimes have quite an impact on common data mining approaches and thus, the question of effective modelling this so called human uncertainty emerges naturally. Our contribution introduces two different approaches for modelling the human uncertainty of user responses. In doing so, we develop techniques in order to measure this uncertainty at the level of user inputs as well as the level of user cognition. With support of comprehensive user experiments and large-scale simulations, we systematically compare both methodologies along with their implications for personalisation approaches. Our findings demonstrate that significant amounts of users do submit something completely different (action) than they really have in mind (cognition). Moreover, we demonstrate that statistically sound evidence with respect to algorithm assessment becomes quite hard to realise, especially when explicit rankings shall be built.
The impacts of human uncertainty for recommendation results have been frequently discussed in recent work from different perspectives. Observations presented in @cite_5 @cite_6 have shown that it can significantly influence results of recommender evaluation. The methodology applied there is based on repeating rating scenarios for same users-items-pairs and represents the current standard in latest research such as @cite_10 . In this paper, the same methodology is explored and compared to a new Bayesian method, which we have derived from the from latest research on cognitions of uncertainty in educational scenarios @cite_1 .
@cite_1 @cite_5 @cite_10 @cite_6
2164918230 1872493208 2148943147
The ideas of first year university students about measurement in the physics laboratory are explored. Student responses to written probes administered at the beginning of the year are compared to those written after a 12 week laboratory course. The 'point' and 'set' paradigms are used as a model to analyse the responses to the probes. At the heart of the point paradigm is that both action and reasoning are based solely on individual measurements in a data set. On the other hand, subscribing to the set paradigm implies an understanding that a series of measurements are to be viewed as a collective that can be modelled by theoretical constructs, such as the mean and standard deviation. The degree of consistent use of these paradigms by individual students across the sets of probes is investigated. Implications for effective teaching interventions in the physics laboratory are discussed. Recent growing interest in predicting and influencing consumer behavior has generated a parallel increase in research efforts on Recommender Systems. Many of the state-of-the-art Recommender Systems algorithms rely on obtaining user ratings in order to later predict unknown ratings. An underlying assumption in this approach is that the user ratings can be treated as ground truth of the user's taste. However, users are inconsistent in giving their feedback, thus introducing an unknown amount of noise that challenges the validity of this assumption. In this paper, we tackle the problem of analyzing and characterizing the noise in user feedback through ratings of movies. We present a user study aimed at quantifying the noise in user ratings that is due to inconsistencies. We measure RMSE values that range from 0.557 to 0.8156. We also analyze how factors such as item sorting and time of rating affect this noise. A common approach to designing Recommender Systems (RS) consists of asking users to explicitly rate items in order to collect feedback about their preferences. However, users have been shown to be inconsistent and to introduce a non-negligible amount of natural noise in their ratings that affects the accuracy of the predictions. In this paper, we present a novel approach to improve RS accuracy by reducing the natural noise in the input data via a preprocessing step. In order to quantitatively understand the impact of natural noise, we first analyze the response of common recommendation algorithms to this noise. Next, we propose a novel algorithm to denoise existing datasets by means of re-rating: i.e. by asking users to rate previously rated items again. This denoising step yields very significant accuracy improvements. However, re-rating all items in the original dataset is unpractical. Therefore, we study the accuracy gains obtained when re-rating only some of the ratings.In particular, we propose two partial denoising strategies: data and user-dependent denoising. Finally, we compare the value of adding a rating of an unseen item vs. re-rating an item. We conclude with a proposal for RS to improve the quality of their user data and hence their accuracy: asking users to re-rate items might, in some circumstances, be more beneficial than asking users to rate unseen items.