<jupyter_start><jupyter_text>Search relevanceWe have a set of search queries and products on a website. We need to determine the extent to which each query is relevant to the corresponding product on the website. We ask performers to look at the search query and the product image from the website and rate the relevance level. To get acquainted with Toloka tools for free, you can use the promo code **TOLOKAKIT1** on $20 on your [profile page](https://toloka.yandex.com/requester/profile?utm_source=github&utm_medium=site&utm_campaign=tolokakit) after registration. Prepare environment and import all we'll need.<jupyter_code>!pip install toloka-kit==0.1.15
!pip install crowd-kit==0.0.7

import datetime
import sys
import time
import logging
import urllib.request

import pandas
import numpy as np

import toloka.client as toloka
import toloka.client.project.template_builder as tb
from crowdkit.aggregation import DawidSkene

logging.basicConfig(
    format='[%(levelname)s] %(name)s: %(message)s',
    level=logging.INFO,
    stream=sys.stdout,
)<jupyter_output><empty_output><jupyter_text>Сreate toloka-client instance. All api calls will go through it. More about OAuth token in our [Learn the basics example](https://github.com/Toloka/toloka-kit/tree/main/examples/0.getting_started/0.learn_the_basics) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Toloka/toloka-kit/blob/main/examples/0.getting_started/0.learn_the_basics/learn_the_basics.ipynb)<jupyter_code>toloka_client = toloka.TolokaClient(input("Enter your token:"), 'PRODUCTION')  # Or switch to 'SANDBOX'
print(toloka_client.get_requester())<jupyter_output><empty_output><jupyter_text>Create a projectEnter a clear project name and description.> Note: The project name and description will be visible to the performers.<jupyter_code>project = toloka.Project(
    public_name='Classify search query relevance',
    public_description='Analyze a website with a product and decide to what extent it meets the search query',
)<jupyter_output><empty_output><jupyter_text>Create task interface.> Read about configuring the [task interface](https://toloka.ai/docs/guide/reference/interface-spec.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in the Requester’s Guide.> Check the [Interface section](https://toloka.ai/knowledgebase/interface?utm_source=github&utm_medium=site&utm_campaign=tolokakit) of our Knowledge Base for more tips on interface design.This interface contains a query, a picture of a product, and its title, which needs to be assessed. There is a button for checking this query in Google, which is handy because the query might not be obvious and performers will often need to look it up. There is also a plugin that checks if a label was really chosen.<jupyter_code># left column
product_image = tb.ImageViewV1(tb.InputData('imagepath'))
product_description = tb.MarkdownViewV1(tb.InputData('title'), label='Product title:')

# right column
request = tb.AlertViewV1(tb.TextViewV1(tb.InputData('query')), label='Search query', theme='info')
google_link = tb.ActionButtonViewV1(tb.OpenLinkActionV1(tb.InputData('search_url')), label='Search query in Google')
divider = tb.DividerViewV1()
label = tb.RadioGroupFieldV1(
    tb.OutputData('result_class'),
    label='Choose relevance class',
    options=[
        tb.GroupFieldOption('relevant', 'Relevant'),
        tb.GroupFieldOption('relevant_minus', 'Slightly relevant'),
        tb.GroupFieldOption('irrelevant', 'Irrelevant'),
    ],
    validation=tb.RequiredConditionV1()
    )

# create interface with two columns
general_interface = tb.SidebarLayoutV1(
    tb.ListViewV1([product_image, product_description], direction='vertical'),
    tb.ListViewV1([request, google_link, divider, label], direction='vertical'),
    min_width=400,
)

task_width_plugin = tb.TolokaPluginV1(
    layout=tb.TolokaPluginV1.TolokaPluginLayout(
        kind='scroll',
        task_width=600,
    )
)

project_interface = toloka.project.TemplateBuilderViewSpec(
    view=tb.ListViewV1([general_interface]),
    plugins=[task_width_plugin],
)<jupyter_output><empty_output><jupyter_text>For performers, our interface will look like this.      <img src="./img/performer_interface.png"         alt="Task page"  width="1000">        Figure 1. What the task can looks like.   Specifications are a description of input data that will be used in a project and the output data that will be collected from the performers.We are using screenshots to make this demo more robust against possible webpage changes. Another way is to use an iframe and let the performers assess the whole webpage.> Read more about [input and output data specifications](https://yandex.ru/support/toloka-tb/operations/create-specs.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in the Requester’s Guide.<jupyter_code>input_specification = {
    'imagepath': toloka.project.UrlSpec(),
    'title': toloka.project.StringSpec(),
    'query': toloka.project.StringSpec(),
    'search_url': toloka.project.UrlSpec(),
}
output_specification = {'result_class': toloka.project.StringSpec()}

project.task_spec = toloka.project.task_spec.TaskSpec(
    input_spec=input_specification,
    output_spec=output_specification,
    view_spec=project_interface,
)<jupyter_output><empty_output><jupyter_text>Write comprehensive instructions.Instructions are essential for complex tasks like relevance evaluation that are based on a set of rules and various criteria. Make sure to not only describe the general idea, but also go through examples and explain the evaluation logic in each case. We recommend trying to evaluate around two dozen cases yourself to get more insights for the instructions.> Get more tips on [designing instructions](https://toloka.ai/knowledgebase/instruction?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in our Knowledge Base.<jupyter_code>project.public_instructions = """Your task is to determine whether a product is relevant to the search query and to what degree.<br>
<br>
Imagine that you're searching for this product and get such an answer for your query.<br>
<br>
<b>Basic steps:</b>
<ul><li>Look at the title and the image of the product</li>
<li>Compare it with the query</li>
<li>Choose the most appropriate level of relevance from Relevant, Slightly relevant or Irrelevant&nbsp;.</li></ul>
<br>
<i>If image is too small click on the expand button!</i>
<br>
Relevant:<br>
<ol><li>The product fully matches the query</li></ol>
<br>
Slightly relevant:<br>
<ol><li>The product is somewhat right but some properties are different</li></ol>
<br>
Irrelevant:<br>
<ol><li>There is a completely different product in the image</li>
<li>The title doesn't match the query at all</li></ol>
"""
example_images = [
    {
        'label': 'Relevant',
        'product': 'Bodum Bistro Electric Burr Coffee Grinder-(Brand New)',
        'query': 'coffee grinder',
        'img_url': 'https://tlklab.s3.yandex.net/screenshots/1026.jpg'
    },
    {
        'label': 'Slightly relevant',
        'product': 'The Hobbit: The Desolation of Smaug',
        'query': 'Bluray Hobbit extended',
        'img_url': 'https://tlklab.s3.yandex.net/screenshots/1037.jpg'
    },
    {
        'label': 'Irrelevant',
        'product': 'NEW IKEA RUSCH BATTERY OPERATED WHITE WALL CLOCK',
        'query': 'stop watches',
        'img_url': 'https://tlklab.s3.yandex.net/screenshots/1066.jpg'
    },
]

table_rows = ''.join([
    f'<tr><td>{row["label"]}</td>'
    f'<td>{row["product"]}</td>'
    f'<td>{row["query"]}</td>'
    f'<td><img alt="{row["label"]}" src="{row["img_url"]}" width="200" height="205"></td></tr>\n'
    for row in example_images
])

project.public_instructions = project.public_instructions + f"""
<br>
<b>Examples:</b><br>
<table border="1">
<tr><td>Class</td><td>Product</td><td>Query</td><td>Image</td></tr>
{table_rows}
</table>
"""<jupyter_output><empty_output><jupyter_text>Create a project.<jupyter_code>project = toloka_client.create_project(project)<jupyter_output><empty_output><jupyter_text>Preparing dataThis example uses [eCommerce search relevance](https://data.world/crowdflower/ecommerce-search-relevance) that distributed under Public Domain License [![License: ODbL](https://img.shields.io/badge/License-PDDL-brightgreen.svg)](https://opendatacommons.org/licenses/pddl/)Let's load this dataset and split it.<jupyter_code>!curl https://tlk.s3.yandex.net/ext_dataset/ecommerce_search_relevance.csv --output dataset.csv

dataset = pandas.read_csv('dataset.csv')
dataset = dataset.sample(frac=1).reset_index(drop=True)

with pandas.option_context("max_colwidth", 100):
    display(dataset)<jupyter_output>% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 27.2M  100 27.2M    0     0  20.1M      0  0:00:01  0:00:01 --:--:-- 20.1M<jupyter_text>Because it's old dataset, we need to check images. Let's take 80 rows with valid images:- 10 - for training- 10 - for exam- 10 - for golden-set in the main pool- 50 - main tasks<jupyter_code>rows_cnt = 80
new_dataset = pandas.DataFrame(columns=['relevance', 'product_image', 'product_title', 'query'])
for row in dataset.itertuples():
    try:
        response = urllib.request.urlopen(row.product_image)
        data = response.read()
        if len(data) > 2000:
            new_dataset = new_dataset.append(
                {
                    'relevance': row.relevance,
                    'product_image': row.product_image,
                    'product_title': row.product_title,
                    'query': row.query,
                },
                ignore_index=True
            )
            print(len(new_dataset), row.product_image)
            if len(new_dataset) >= rows_cnt:
                break
    except:
        pass<jupyter_output>1 http://scene7.targetimg1.com/is/image/Target/16721129?wid=138&hei=138
2 http://ak1.ostkcdn.com/images/products/9800299/P16967916.jpg
3 http://scene7.targetimg1.com/is/image/Target/11108929?wid=138&hei=138
4 http://ak1.ostkcdn.com/images/products/8595192/Organize-It-All-Cherry-Open-Drawer-Storage-Cube-P15865898.jpg
5 http://ak1.ostkcdn.com/images/products/8597293/P15867585.jpg
6 http://ak1.ostkcdn.com/images/products/8396033/Tubular-6-port-Sunflower-Seed-Feeder-P15697821.jpg
7 http://ak1.ostkcdn.com/images/products/8756258/Coffee-Grinder-Vinyl-Wall-Decal-P15999634.jpg
8 http://ak1.ostkcdn.com/images/products/P13759467.jpg
9 http://ak1.ostkcdn.com/images/products/7860036/7860036/White-Mark-Womens-Ibiza-Yellow-and-Turquoise-Printed-Sleeveless-Dress-P15245439.jpg
10 http://ak1.ostkcdn.com/images/products/9490830/P16671734.jpg
11 http://scene7.targetimg1.com/is/image/Target/16413566?wid=138&hei=138
12 http://ak1.ostkcdn.com/images/products/5900276/75/372/NCAA-Idaho-Vandals-Round-Patio-Set[...]<jupyter_text>Split dataset into 4 parts.<jupyter_code>dataset_with_answers = new_dataset[~new_dataset['relevance'].isna()].head(30)
main_dataset = new_dataset.drop(dataset_with_answers.index)
training_dataset, exam_dataset, gold_dataset = np.split(dataset_with_answers, [10, 20], axis=0)

print(f'training_dataset - {len(training_dataset)}')
print(f'exam_dataset - {len(exam_dataset)}')
print(f'gold_dataset - {len(gold_dataset)}')
print(f'main_dataset - {len(main_dataset)}')<jupyter_output>training_dataset - 10
exam_dataset - 10
gold_dataset - 10
main_dataset - 50<jupyter_text>In the dataset relevance is a float, where 1.0 is "irrelevant" and 4.0 is absolutely "relevant". But in our project, we need three string labels. Let's prepare function to convert one to another.<jupyter_code>def str_relevance(relevance: float) -> str:
    if relevance > 3:
        return 'relevant'
    if relevance > 2:
        return 'relevant_minus'
    return 'irrelevant'

print(str_relevance(1.0))
print(str_relevance(3.0))
print(str_relevance(3.66))<jupyter_output>irrelevant
relevant_minus
relevant<jupyter_text>Create a training poolSince relevance evaluation is based on rules, not just common sense or certain skills, we recommend investing some time on learning how to explain all the rules. Training needs to involve both common and extreme cases. The comments should explain the underlying logic rather than just state the correct answers.> A well-grounded training exercise is also a great tool for scaling your task, because you can run it any time you need new performers.Read more about [selecting performers](https://toloka.ai/knowledgebase/quality-control?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in our Knowledge Base.Read more about [training pools](https://toloka.ai/docs/guide/concepts/train.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in our Requester’s Guide.<jupyter_code>training = toloka.Training(
    project_id=project.id,
    private_name='Search relevance training',
    may_contain_adult_content=False,
    assignment_max_duration_seconds=60*10,
    mix_tasks_in_creation_order=True,
    shuffle_tasks_in_task_suite=True,
    training_tasks_in_task_suite_count=2,
    task_suites_required_to_pass=5,
    retry_training_after_days=2,
    inherited_instructions=True,
)
training = toloka_client.create_training(training)<jupyter_output><empty_output><jupyter_text>Upload training tasks to the pool, without opening the training pool.> We recommend opening the training pool along with the main pool. Otherwise Tolokers will spend their time on training but get no access to real tasks, which is frustrating. Also, do not forget to close the training pool when there are no main tasks available anymore.<jupyter_code>hint_messages = {
    'irrelevant': 'The product does not fit the request.',
    'relevant_minus': 'The product is similar, but does not fully satisfy the request.',
    'relevant': 'Product is satisfied.',
}

training_tasks = [
    toloka.Task(
        pool_id=training.id,
        input_values={
            'imagepath': row.product_image,
            'title': row.product_title,
            'query': row.query,
            'search_url': f'https://www.google.ru/search?q={row.query}',
        },
        known_solutions = [toloka.task.BaseTask.KnownSolution(output_values={'result_class': str_relevance(row.relevance)})],
        message_on_unknown_solution=hint_messages[str_relevance(row.relevance)],
    )
    for row in training_dataset.itertuples()
]
result = toloka_client.create_tasks(training_tasks, allow_defaults=True)
print(len(result.items))<jupyter_output>10<jupyter_text>Create an exam poolWe recommend adding an exam pool along with the training because relevance evaluation projects are usually more complicated than most crowdsourcing projects, and it takes a certain effort to master all the guidelines. The more guidelines there are, the greater will be the need to check if the performers have really learned them.Set up exam quality calculation via skill.Create new skill.<jupyter_code>exam_skill = next(toloka_client.get_skills(name='Search relevance exam'), None)
if exam_skill:
    print('Skill already exists')
else:
    exam_skill = toloka_client.create_skill(
        name='Search relevance exam',
        hidden=True,
        public_requester_description={'EN': 'How performer deal with search relevance exam'},
    )<jupyter_output><empty_output><jupyter_text>Set the price per task suite (for example, $0.03).> You can use a zero price as well. However, if the exam is time-consuming, a zero price might be unfair, as the performers will spend  a lot of time completing it.Read more about [pricing principles](https://toloka.ai/knowledgebase/pricing?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in our Knowledge Base.<jupyter_code>exam = toloka.Pool(
    project_id=project.id,
    # Give the pool any convenient name. You are the only one who will see it.
    private_name='Classify search query relevance - exam',
    may_contain_adult_content=False,
    # Set the price per task page.
    reward_per_assignment=0.03,
    will_expire=datetime.datetime.utcnow() + datetime.timedelta(days=365),
    # Time allowed for completing a task page
    assignment_max_duration_seconds=600,
    filter=(toloka.filter.Languages.in_('EN')),
)

exam.set_mixer_config(golden_tasks_count=10)<jupyter_output><empty_output><jupyter_text>Attach the training pool.> The quality of the training can be low to just filter out potential deception, because we expect performers to make mistakes and learn from them (yet again, relevance is a complicated task type).<jupyter_code>exam.set_training_requirement(training_pool_id=training.id, training_passing_skill_value=10)<jupyter_output><empty_output><jupyter_text>We will have 10 tasks in the exam pool, so the quality will be calculated after the whole exam has been passed.> We will then use this parameter as an entry filter for the main pool.<jupyter_code>exam.quality_control.add_action(
    collector=toloka.collectors.GoldenSet(history_size=10),
    conditions=[toloka.conditions.TotalAnswersCount >= 10,],
    action=toloka.actions.SetSkillFromOutputField(
        skill_id=exam_skill.id,
        from_field='correct_answers_rate',
    ),
)
exam = toloka_client.create_pool(exam)<jupyter_output><empty_output><jupyter_text>Add tasks to exam.<jupyter_code>exam_tasks = [
    toloka.Task(
        pool_id=exam.id,
        input_values={
            'imagepath': row.product_image,
            'title': row.product_title,
            'query': row.query,
            'search_url': f'https://www.google.ru/search?q={row.query}',
        },
        known_solutions = [toloka.task.BaseTask.KnownSolution(output_values={'result_class': str_relevance(row.relevance)})],
        infinite_overlap=True,
    )
    for row in exam_dataset.itertuples()
]
result = toloka_client.create_tasks(exam_tasks, allow_defaults=True)
print(len(result.items))<jupyter_output>10<jupyter_text>Create the main poolA pool is a set of paid tasks grouped into task pages. These tasks are sent out for completion at the same time.>Note: All tasks within a pool have the same settings (price, quality control, etc.)Set the price per task suite for 0.03$. Read more about [pricing principles](https://toloka.ai/knowledgebase/pricing?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in our Knowledge Base.Sets an overlap of 3. This is the number of users who will complete the same task. We will aggregate the results after the pool is completed. To understand [how this rule works](https://toloka.ai/docs/guide/concepts/mvote.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit), go to the Requester’s Guide.Let's add language filter so performers who speak English will be invited to complete this task.<jupyter_code>pool = toloka.Pool(
    project_id=project.id,
    # Give the pool any convenient name. You are the only one who will see it.
    private_name='Classify search query relevance',
    may_contain_adult_content=False,
    # Set the price per task page.
    reward_per_assignment=0.03,
    will_expire=datetime.datetime.utcnow() + datetime.timedelta(days=365),
    # Overlap. This is the number of users who will complete the same task.
    defaults=toloka.Pool.Defaults(default_overlap_for_new_task_suites=3),
    # Time allowed for completing a task page
    assignment_max_duration_seconds=600,
    filter=(
        (toloka.filter.Languages.in_('EN')) &
        (toloka.filter.Skill(exam_skill.id) >= 90)
    )
)<jupyter_output><empty_output><jupyter_text>Set up [Quality control](https://toloka.ai/docs/guide/concepts/control.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit):  - Set the number of responses and the percentage of correct responses. We will record a quality parameter in the same skill we used in the quality filter.  - Set up the [Fast responses](https://toloka.ai/docs/guide/concepts/quick-answers.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit) rule. This rule allows you to ban performers who submit tasks at a suspiciously high speed.Read more about [quality control principles](https://toloka.ai/knowledgebase/quality-control?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in our Knowledge Base or check out [control tasks settings](https://toloka.ai/docs/guide/concepts/goldenset.html?utm_source=github&utm_medium=site&utm_campaign=tolokakit) in the Requester’s Guide.<jupyter_code>pool.quality_control.add_action(
    collector=toloka.collectors.GoldenSet(history_size=20),
    conditions=[toloka.conditions.TotalAnswersCount >= 1,],
    action=toloka.actions.SetSkillFromOutputField(
        skill_id=exam_skill.id,
        from_field='correct_answers_rate',
    ),
)

pool.quality_control.add_action(
    collector=toloka.collectors.AssignmentSubmitTime(history_size=5, fast_submit_threshold_seconds=10),
    conditions=[toloka.conditions.FastSubmittedCount >= 1],
    action=toloka.actions.RestrictionV2(
        scope=toloka.user_restriction.UserRestriction.PROJECT,
        duration=10,
        duration_unit='DAYS',
        private_comment='Fast responses',
    )
)<jupyter_output><empty_output><jupyter_text>Specify	the number of tasks per page. We recommend putting as many tasks on one page as a performer can complete in 1 to 5 minutes. That way, performers are less likely to get tired, and they won’t lose a significant amount of data if a technical issue occurs.To learn more about [grouping tasks](https://toloka.ai/docs/search/?utm_source=github&utm_medium=site&utm_campaign=tolokakit&query=smart+mixing) into suites, read the Requester’s Guide.<jupyter_code>pool.set_mixer_config(
    real_tasks_count=4,
    golden_tasks_count=1,
)<jupyter_output><empty_output><jupyter_text>Create pool<jupyter_code>pool = toloka_client.create_pool(pool)<jupyter_output><empty_output><jupyter_text>Preparing and uploading tasksWe recommend putting as many tasks on one page as a performer can complete in 1 to 5 minutes. That way, performers are less likely to get tired, and they won’t lose a significant amount of data if a technical issue occurs.To learn more about [grouping tasks](https://toloka.ai/docs/search/?utm_source=github&utm_medium=site&utm_campaign=tolokakit&query=smart+mixing) into suites, read the Requester’s Guide.<jupyter_code>golden_tasks = [
    toloka.Task(
        pool_id=pool.id,
        input_values={
            'imagepath': row.product_image,
            'title': row.product_title,
            'query': row.query,
            'search_url': f'https://www.google.ru/search?q={row.query}',
        },
        known_solutions = [toloka.task.BaseTask.KnownSolution(output_values={'result_class': str_relevance(row.relevance)})],
        infinite_overlap=True,
    )
    for row in gold_dataset.itertuples()
]

tasks = [
    toloka.Task(
        pool_id=pool.id,
        input_values={
            'imagepath': row.product_image,
            'title': row.product_title,
            'query': row.query,
            'search_url': f'https://www.google.ru/search?q={row.query}',
        },
    )
    for row in main_dataset.itertuples()
]
created_tasks = toloka_client.create_tasks(golden_tasks + tasks, allow_defaults=True)
print(len(created_tasks.items))<jupyter_output>44<jupyter_text>You can visit web interface and preview task suites.      <img src="./img/task_suite_interface.png"         alt="How performers will see your tasks"  height="600">        Figure 2. How performers will see your tasks   Start the pools.**Important.** Remember that real Toloka performers will complete the tasks.Double check that everything is correctwith your project configuration before you start the pool<jupyter_code>training = toloka_client.open_training(training.id)
print(f'training - {training.status}')

exam = toloka_client.open_pool(exam.id)
print(f'exam - {exam.status}')

pool = toloka_client.open_pool(pool.id)
print(f'main pool - {pool.status}')<jupyter_output>training - Status.OPEN
exam - Status.OPEN
main pool - Status.OPEN<jupyter_text>Receiving responses Wait until the pool is completed.<jupyter_code>pool_id = pool.id

def wait_pool_for_close(pool_id, minutes_to_wait=1):
    sleep_time = 60 * minutes_to_wait
    pool = toloka_client.get_pool(pool_id)
    while not pool.is_closed():
        op = toloka_client.get_analytics([toloka.analytics_request.CompletionPercentagePoolAnalytics(subject_id=pool.id)])
        op = toloka_client.wait_operation(op)
        percentage = op.details['value'][0]['result']['value']
        print(
            f'   {datetime.datetime.now().strftime("%H:%M:%S")}\t'
            f'Pool {pool.id} - {percentage}%'
        )
        time.sleep(sleep_time)
        pool = toloka_client.get_pool(pool.id)
    print('Pool was closed.')

wait_pool_for_close(pool_id)

exam = toloka_client.close_pool(exam.id)
print(f'exam - {exam.status}')

training = toloka_client.close_training(training.id)
print(f'training - {training.status}')<jupyter_output><empty_output><jupyter_text>Get responsesWhen all the tasks are completed, look at the responses from performers.<jupyter_code>answers_df = toloka_client.get_assignments_df(pool.id, field=['ASSIGNMENT:task_id', 'ASSIGNMENT:worker_id'])

answers_df = answers_df[answers_df['GOLDEN:result_class'].isna()]

answers_df = answers_df.rename(columns={
    'ASSIGNMENT:task_id': 'task',
    'OUTPUT:result_class': 'label',
    'ASSIGNMENT:worker_id': 'performer',
    'INPUT:query': 'query',
    'INPUT:imagepath': 'imagepath',
    'INPUT:title': 'title',
})

answers_to_aggregate = answers_df[['task', 'label', 'performer']]

with pandas.option_context("max_colwidth", None):
    display(answers_df)<jupyter_output>[WARNING] toloka.client: Experimental method<jupyter_text>Aggregation results using the Dawid-Skene model. We use this aggregation model because our questions are of comparable difficulty, and we don't have many control tasks.Read more about the [Dawid-Skene model](https://toloka.ai/docs/guide/concepts/result-aggregation.html?utm_source=github&utm_medium=site&utm_campaign=tolokakitaggr__dawid-skene) in the Requester’s Guide or get at an overview of different [aggregation models](https://toloka.ai/knowledgebase/aggregation) our Knowledge Base.More aggregation models in [Crowd-Kit](https://github.com/Toloka/crowd-kitcrowd-kit-computational-quality-control-for-crowdsourcing).<jupyter_code>predicted_answers = DawidSkene(n_iter=20).fit_predict(answers_to_aggregate).reset_index(name='result')

predicted_answers = pandas.merge(predicted_answers, answers_df.drop_duplicates(subset='task'), on='task')

with pandas.option_context("max_colwidth", None):
    display(predicted_answers[['query', 'imagepath', 'title', 'result']])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10827596527985633, 'fraction_numerical': 0.01590092786590841, 'mean_word_length': 4.830933682373473, 'pattern_counts': {'":': 0, '<': 166, '<?xml version=': 0, '>': 186, 'https://': 33, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '479482', 'license': 'Apache-2.0', 'n_tokens_mistral': 8854, 'n_tokens_neox': 8371, 'n_words': 2060, 'path': 'examples/8.search_relevance/search_relevance.ipynb', 'repo_name': 'lehasm/toloka-kit', 'size': '26728'}