<jupyter_start><jupyter_text>Цель на ноутбукНеобходимо отобрать те посты и комментарии, над которыми будем делать topic modeling и topic representation 1. Посты в каналахПредлагается взять день начала конфликта - 24 февраля и последний полный день выборки.  2. Комментарии к постамБерем также 24 февраля и 7 марта, по следующим правилам чистим от ботов:1. Комментарии, отправленные в течение очень короткого промежутка времени после публикации поста2. Наличие ссылок в сообщении3. Наличие номеров карт и телефонов4. Медиа без текстаЭмбеддинги будем получать бертом, так что процессить текст не надо.<jupyter_code>import json
import re
from datetime import (
    datetime, timezone
)

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style="whitegrid")
pd.set_option("display.max_columns", 50)
pd.set_option("display.max_rows", 150)<jupyter_output><empty_output><jupyter_text>1. Служебные функции<jupyter_code>def is_comment_correct(comment: str) -> bool:
    pass<jupyter_output><empty_output><jupyter_text>2. Константы<jupyter_code>MESSAGES_PARQUET_FILE = "../data/processed/messages.parquet"
COMMENTS_PARQUET_FILE = "../data/processed/comments.parquet"

START_DATE_LEFT = datetime(
    2022, 2, 24, 0, 0, 0, 
    tzinfo=timezone.utc
)

START_DATE_RIGHT = datetime(
    2022, 2, 25, 0, 0, 0, 
    tzinfo=timezone.utc
)

END_DATE_LEFT = datetime(
    2022, 3, 7, 0, 0, 0, 
    tzinfo=timezone.utc
)

END_DATE_RIGHT = datetime(
    2022, 3, 8, 0, 0, 0, 
    tzinfo=timezone.utc
)<jupyter_output><empty_output><jupyter_text>3. Выгружаем данныеДополнително делаем `join_key`, так как у каждого канала своя нумерация постов и они могут дублироваться.<jupyter_code>messages = pd.read_parquet(
    MESSAGES_PARQUET_FILE
)

messages["join_key"] = messages.apply(
    lambda row: f"{row['channel_name']}_{row['id']}",
    axis=1
)

print(messages["join_key"].nunique())
print(messages.shape)
messages.head(5)
comments = pd.read_parquet(
    COMMENTS_PARQUET_FILE
)

comments = comments[
    (
        (
            (comments["comment_date"] >= START_DATE_LEFT)
            & (comments["comment_date"] < START_DATE_RIGHT)
        )
        | (
            (comments["comment_date"] >= END_DATE_LEFT)
            & (comments["comment_date"] < END_DATE_RIGHT)
        )
    )
]

comments["join_key"] = comments.apply(
    lambda row: f"{row['channel_name']}_{row['message_id']}",
    axis=1
)

comments["comment_date_truncated"] = comments["comment_date"].apply(
    lambda value: f"{value.year}-{value.month}-{value.day}, {value.hour}:00 - {value.hour}:59"
)

print(comments["join_key"].nunique())
print(comments.shape)
comments.head(5)<jupyter_output>967
(104536, 10)<jupyter_text>4. Смотрим у каких каналов есть комментарииКомментарии есть только у двух каналов - NEXTA и Readovka, при этом 24 не все посты в NEXTA были открыты для комментирования.<jupyter_code>date_plot_order = pd.unique(
    comments.sort_values("comment_date")["comment_date_truncated"]
)


figure = plt.figure(figsize=(15, 8))
grid = figure.add_gridspec(2, 1)

ax_channel = figure.add_subplot(grid[0, 0])
sns.countplot(x="channel_name", data=comments, ax=ax_channel)

ax_date = figure.add_subplot(grid[1, 0])
sns.countplot(
    x="comment_date_truncated", data=comments, 
    ax=ax_date, order=date_plot_order,
    hue="channel_name"
)

plt.xticks(rotation=90)
plt.axvline(
    x=23.5, color="black", linestyle="--"
)<jupyter_output><empty_output><jupyter_text>5. Чистим комментарии<jupyter_code>comments_enriched = pd.merge(
    comments, messages, 
    on="join_key",
    how="inner"
)

print(comments_enriched.shape)
assert comments_enriched.shape[0] == comments.shape[0]

comments_enriched.head(5)
comments_enriched["comment_message"] = comments_enriched["comment_message"].str.replace(
    "\xa0", " ", regex=False
).str.replace(
    "\n", " ", regex=False
).str.replace(
    # emoji
    "\s+", 
    " ", regex=True
).str.strip()<jupyter_output><empty_output><jupyter_text>5.1. Выкидываем выбросы по времени<jupyter_code>comments_enriched["comment_delta"] = (
    comments_enriched["comment_date"] - comments_enriched["date"]
).dt.total_seconds()

(
    comments_enriched["comment_delta"].quantile(0.05), 
    comments_enriched["comment_delta"].quantile(0.95)
)
comments_enriched = comments_enriched[
    (
        (comments_enriched["comment_delta"] > comments_enriched["comment_delta"].quantile(0.05))
        & (comments_enriched["comment_delta"] < comments_enriched["comment_delta"].quantile(0.95))
    )
]

comments_enriched.shape<jupyter_output><empty_output><jupyter_text>5.2. Выкидываем медиа без текста<jupyter_code>comments_enriched = comments_enriched[
    ~(
        comments_enriched["comment_message"] == ""
    )
]

comments_enriched.shape<jupyter_output><empty_output><jupyter_text>5.3. Выкидываем ссылки<jupyter_code>comments_enriched = comments_enriched[
    ~comments_enriched["comment_message"].str.contains("https?", regex=True, flags=re.IGNORECASE)
]

comments_enriched.shape<jupyter_output><empty_output><jupyter_text>5.4. Выкидываем номера телефонов и карт<jupyter_code>comments_enriched = comments_enriched[
    ~comments_enriched["comment_message"].str.contains("(?:\d\s*){8,}", regex=True, flags=re.IGNORECASE)
]

comments_enriched.shape<jupyter_output><empty_output><jupyter_text>5.5. Спам и реклама<jupyter_code>comment_messages_vc = comments_enriched["comment_message"].value_counts()
comment_messages_vc = comment_messages_vc[comment_messages_vc > 2].reset_index(drop=False)

comment_messages_vc["len"] = comment_messages_vc["index"].apply(len)

comment_messages_vc = comment_messages_vc[
    comment_messages_vc["len"] > comment_messages_vc["len"].quantile(0.85)
]

print(
    comment_messages_vc.shape, 
    comment_messages_vc["comment_message"].sum()
)

comment_messages_vc.head(5)
comments_enriched = comments_enriched[
    ~comments_enriched["comment_message"].isin(
        comment_messages_vc["index"].values
    )
]

comments_enriched.shape
comments_enriched = comments_enriched[
    ~comments_enriched["comment_message"].str.contains("@[a-z0-9]", regex=True, flags=re.IGNORECASE)
]

comments_enriched.shape<jupyter_output><empty_output><jupyter_text>5.6. Выборка для topic modeling<jupyter_code>comments_to_modeling = comments_enriched[
    [
        "channel_name_x", "channel_orientation_x", 
        "comment_date", "comment_message", "id"
    ]
]

print(comments_to_modeling.shape)
comments_to_modeling.sample(5)<jupyter_output>(87947, 5)<jupyter_text>6. Чистим посты<jupyter_code>messages = messages[
    (
        (
            (messages["date"] >= START_DATE_LEFT)
            & (messages["date"] < START_DATE_RIGHT)
        )
        | (
            (messages["date"] >= END_DATE_LEFT)
            & (messages["date"] < END_DATE_RIGHT)
        )
    )
]

messages["date_truncated"] = messages["date"].apply(
    lambda value: f"{value.year}-{value.month}-{value.day}, {value.hour}:00 - {value.hour}:59"
)

messages.shape
date_plot_order = pd.unique(
    messages.sort_values("date")["date_truncated"]
)


figure = plt.figure(figsize=(15, 8))
grid = figure.add_gridspec(3, 1)

ax_orient = figure.add_subplot(grid[0, 0])
sns.countplot(x="channel_orientation", data=messages, ax=ax_orient)

ax_channel = figure.add_subplot(grid[1, 0])
sns.countplot(x="channel_name", data=messages, ax=ax_channel)

ax_date = figure.add_subplot(grid[2, 0])
sns.countplot(
    x="date_truncated", data=messages, 
    ax=ax_date, order=date_plot_order,
    hue="channel_name"
)

plt.xticks(rotation=90)
plt.axvline(
    x=23.5, color="black", linestyle="--"
)<jupyter_output><empty_output><jupyter_text>6.1. Убираем из постов отличительные черты каналов (названия, приписки, emoji и т.п)<jupyter_code>with open("../data/raw/codes.json", "r") as codes_file:
    emoji_codes = json.load(codes_file)
    
regex_emoji_pattern = r"|".join(
    re.escape(code) for code in emoji_codes.keys()
)
messages["message"] = messages["message"].str.replace(
    "УС > Подписаться", " ", regex=False
).str.replace(
    "Реальная Война", " ", regex=False
).str.replace(
    "Readovka", " ", regex=False
).str.replace(
    "\xa0", " ", regex=False
).str.replace(
    "\n", " ", regex=False
).str.replace(
    "!!", " ", regex=False
).str.replace(
    "‼", " ", regex=False
).str.replace(
    # emoji
    regex_emoji_pattern, 
    " ", regex=True
).str.replace(
    # emoji
    "\s+", 
    " ", regex=True
).str.strip()<jupyter_output><empty_output><jupyter_text>6.2. Сносим пустые посты (медиа)<jupyter_code>messages = messages[
    messages["message"] != ""
]

messages.shape<jupyter_output><empty_output><jupyter_text>6.3. Выборка для topic modeling<jupyter_code>messages_to_modeling = messages[
    [
        "channel_name", "channel_orientation",
        "date", "message"
    ]
]

print(messages_to_modeling.shape)
messages_to_modeling.sample(5)<jupyter_output>(3473, 4)<jupyter_text>7. Выгружаем<jupyter_code>comments_to_modeling.to_parquet(
    "../data/processed/comments_to_modeling.parquet"
)

messages_to_modeling.to_parquet(
    "../data/processed/messages_to_modeling.parquet"
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13899529386012915, 'fraction_numerical': 0.018824559483419066, 'mean_word_length': 5.170155300472653, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '664975', 'license': 'Apache-2.0', 'n_tokens_mistral': 3584, 'n_tokens_neox': 3542, 'n_words': 505, 'path': 'notebooks/3. topic_modeling_data_selection.ipynb', 'repo_name': 'erqups/special_operation_topic_modeling', 'size': '9137'}