Toloka

Toloka is a crowdsourcing data labeling platform used by hi-tech businesses to quickly label large amounts of data. These data are then used for training machine learning and artificial intelligence algorithms that lie behind self-driving vehicles, computer vision, speech technologies, chatbots and voice assistant, retail and e-commerce, online services, web search, and more. Toloka is currently used by 2,400 businesses worldwide and has a pool of 9 million performers in 100 countries. It is a microtasking solution that breaks down labeling projects into small and simple tasks and distributes them among performers. Most of the tasks are related to analyzing and rating various content and usually do not require special training from the performer. For the correct completion of each task, a performer receives a reward. Toloka was launched by Yandex in 2014 and is incorporated in Switzerland.
About Toloka.
Origin of the platform's name.
 used to be a form of mutual assistance among villagers of Russia, Ukraine, Belarus, Estonia, Latvia, and Lithuania. It was organized in villages to perform urgent work requiring large numbers of workers, such as harvesting, logging, building houses, etc. Sometimes toloka was used for community works (building churches, schools, roads, etc.).
The idea of joint useful work for a common result is consistent with the principle of crowdsourcing.
Types of tasks and scope of results.
Improving search algorithms.
Data labeling helps improve search quality and effectively tune result ranking algorithms. For example, in August 2017, Yandex introduced a new search algorithm called "Korolyov". It's a neural network in creation of which, to train neural networks, the estimates of «tolokers» were used. In more than two years of service, the tolokers have put about two billion ratings, allowed developers to significantly improve the algorithm.
Machine learning.
To "teach" a machine to solve a particular problem, it is necessary to give it a huge number of examples, both positive and negative. Among other tasks, users of the Toloka service can receive ones to describe objects defined by a computer in an image. For example, the photo shows objects that artificial intelligence was able to recognize, but in order to use this data in work, precise information is needed about what kind of objects they are. Artificial intelligence can be trained on the mass of such recognitions provided by humans, who, in turn, will receive encouragement for their work. In tasks of another type, a context of the dialogue is given and a scale is proposed by which it is necessary to assess whether a chatbot's answer in this context is appropriate, interesting, and so on. Another group of tasks in Toloka is checking automatically received drafts of dictionary entries for Yandex.Translate.
Audit and marketing research.
Checking the quality of the online store, delivery service, writing reviews about products and services. For example, in 2017, Toloka users took part in checking the quality of the product return service on . Such experiments allow to control the quality of the service and identify weaknesses, over which work will be carried out in the future to improve and eliminate the identified problems. In another project, users were asked to write a review for a product from popular categories and provide it with their own photos.
Choosing designs and names for new products and services.
Service users also answer questions related to the choice of the most successful name and design for a new product or service, personal or consumer preferences of users. In one of these polls, tolokers chose a name for Yandex's voice assistant. The purpose of the assignment was to determine the character traits of a girl with one name or another. For the survey, they took both the necessary characteristics and completely inappropriate ones, and began to ask: if a girl has such a name, is she rather kind or aggressive? The name "Alice" won the poll by a wide margin.
Payment and money withdrawal.
Completion of tasks in Toloka are paid for - the corresponding amount is indicated next to each task. The amount they can earn is indicated next to the task. It is possible to withdraw funds using several payment systems: YooMoney, PayPal, Skrill, Papara and Payoneer. Also, money can be withdrawn to the mobile phone account of Russian operators MTS, MegaFon, Beeline, Tele2. At the stage of debugging processes on the part of Toloka, withdrawal of money could take up to 30 days, at present - from several minutes to several days.
Toloka mobile apps.
To solve problems that require work "in the field" (for example, go to the address and check the relevance of information about the organization), there is a mobile version of Toloka. The service has two app versions - for Android and iOS.
Users.
Toloka users, also known as performers or tolokers, are people who earn money by completing system testing and improvement tasks on the Toloka crowdsourcing platform. There are currently about 9 million registered Toloka users from Russia, Ukraine, Belarus, Kazakhstan, Uzbekistan, Turkey, France, Indonesia, India and other countries, who have completed a total of more than 2 billion evaluations. In 2018, more than a million people participated in Toloka projects. Most performers are young people under 35 (usually engineering students or mothers on maternity leave). Performers mainly see Toloka as an additional source of income, but many of them note that they like to do meaningful work and clean up the internet.
Requesters.
All tasks in Toloka are placed by requesters. The main requesters are various Yandex teams. The main uses of Toloka within Yandex projects are data collection and processing for machine learning, speech technology, computer vision, smart search algorithms, and other projects, as well as content moderation (conversations in Yandex Maps), field tasks (updating information about businesses and organizations in Yandex Business Directory), optimization of internal business processes (searching for posts on social media, labeling mentions of Yandex by tone as positive or negative). Since at least hundreds or even thousands of tolokers are simultaneously working on the problem, the results are ready in the worst case the next day. In April 2019, Toloka launched an affiliate program, within the framework of which it will redirect external customers to partner companies (or individual performers) to prepare relevant projects and complete tasks.
Research in Toloka.
Toloka is both a tool for solving applied problems and a platform for scientific research in various subject areas. In May 2019, the service's team started publishing datasets for non-commercial and academic purposes to support the scientific community and attract researchers to Toloka. Such datasets are addressed to researchers in different directions like linguistics, computer vision, testing of result aggregation models, and chatbot training