<jupyter_start><jupyter_text>Exploratory data analysis<jupyter_code>import pandas as pd
import pickle
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
from datetime import datetime as dt
import re

warnings.filterwarnings('ignore')
pd.set_option('display.max_colwidth', 500)

RANDOM_STATE = 1<jupyter_output><empty_output><jupyter_text>Load Data<jupyter_code>def read_pickle(path):
    f = open(path, "rb")
    return pickle.load(f)

d = read_pickle('/home/jovyan/work/data/docs_dict.pkl')
raw = pd.DataFrame(d)
df = raw.copy()
df['date'] = pd.to_datetime(df['date'])
# Show sample rows
display(df.sample(5, random_state=RANDOM_STATE))<jupyter_output><empty_output><jupyter_text>Preprocessing The motion texts often starts with *'Motion till riksdagen \'* and then state the author and the associated party *'av \ (\)'* followed by the title. For example, the text of the motion with title *Norrbotniabanan* starts with:  Motion till riksdagen 2010/11:T230 av <NAME>-Brömster (FP) Norrbotniabanan  Let's filter the first sentences which are not really part of the motion text by:  * splitting the motion text at the first occurance of 'av \ (\)'  * then, if the new string starts with the title, drop that part tooWe'll also drop the author's name which is often present at the end of the text<jupyter_code># Make all text lower case
for col in df.columns:
    if df[col].dtype == 'object':
        df[col] = df[col].str.lower()
def trim_motion_text_by_subtitle(row):
    split = row['text'].split(row['subtitle'], 1)
    return split[-1].strip()

df['text'] = df.apply(trim_motion_text_by_subtitle, axis=1)
display(df.sample(5, random_state=RANDOM_STATE))
def trim_motion_text_by_title(row):
    if row['text'].startswith(row['title']):
        text = row['text'].split(row['title'], 1)[-1].strip()
    else:
        text = row['text']
    return text

df['text'] = df.apply(trim_motion_text_by_title, axis=1)
display(df.sample(5, random_state=RANDOM_STATE))<jupyter_output><empty_output><jupyter_text>That's better, but now some of the texts starts with a recommendation for a parliament decision. We wan't to delete that part as well<jupyter_code>def trim_motion_text_by_proposed_decision(row):
    split = re.split("förslag till riksdagsbeslut riksdagen [A-ö0-9\s,]+\.\s|" +\
                     "riksdagen tillkännager för regeringen som sin mening [A-ö0-9\s,]+\.\s|" +\
                     "riksdagen ställer sig bakom det som anförs [A-ö0-9\s,]+\.\s",
                     row['text'])
    return split[-1].strip()

df['text'] = df.apply(trim_motion_text_by_proposed_decision, axis=1)
display(df.sample(5, random_state=RANDOM_STATE))
def trim_motion_text_by_author(row):
    end_string = f"{row['main_author']} ({row['author_party']})"
    if row['text'].endswith(end_string):
        text = row['text'].removesuffix(end_string).strip()
    else:
        text = row['text']
    return text

df['text'] = df.apply(trim_motion_text_by_author, axis=1)
display(df.sample(5, random_state=RANDOM_STATE))<jupyter_output><empty_output><jupyter_text>Finally, consider all (title, text) pairs and drop duplicates from the data set.<jupyter_code>print("Number of motions that are duplicates when only considering the 'title' and 'text' columns:", 
      df.duplicated(subset=['title', 'text']).sum())

# Drop those rows
df = df.drop_duplicates(subset=['title', 'text'], keep='first').copy()<jupyter_output>Number of motions that are duplicates when only considering the 'title' and 'text' columns: 1002<jupyter_text>Describe Data<jupyter_code>display(df.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 166391 entries, 0 to 167392
Data columns (total 7 columns):
 #   Column        Non-Null Count   Dtype         
---  ------        --------------   -----         
 0   id            166391 non-null  object        
 1   date          166391 non-null  datetime64[ns]
 2   title         166391 non-null  object        
 3   subtitle      165865 non-null  object        
 4   text          166391 non-null  object        
 5   main_author   166390 non-null  object        
 6   author_party  166298 non-null  object        
dtypes: datetime64[ns](1), object(6)
memory usage: 10.2+ MB<jupyter_text>About 167k samples. Let's just go ahead and drop any row with a missing value, for simplicity.<jupyter_code>df = df.dropna()
display(df.info())
display(df.describe())<jupyter_output><empty_output><jupyter_text>There's a few things to note here.* The ID's are unique. * On the yop date 25th Jan 1991 there were 3 719 motions filed. That warrants a closer look.  * The titles are not unique. In fact 200 motions share the same title "med anledning av prop. 1989/90:90 om forskning"  * There's som cleaning up left to do based on the text-column. Now 125 only has 'dokument sakans' (document missing) as string.  We'll want to drop those from the data.  Missing Documents<jupyter_code># Drop missing docs
df = df.loc[df['text'].isin(['dokument saknas', '', 'motionen utgår.', 'motionen återtagen', '.']) == False].copy()
display(df[(df['text'].str.len() < 200)])<jupyter_output><empty_output><jupyter_text>There's still lots of junk where the text is very short. Let's make a cut at minimum 200 characters and take another look.<jupyter_code>df = df[df['text'].str.len() > 200]
display(df[(df['text'].str.len() < 250)].sample(10))<jupyter_output><empty_output><jupyter_text>Better!  The many documents with title 'med anledning av prop. 1989/90:90 om forskning'<jupyter_code>df.loc[df['title'] == 'med anledning av prop. 1989/90:90 om forskning'].head(10)<jupyter_output><empty_output><jupyter_text>At least after eye-balling just a few of these motions they seem fine. They're different documents, just on the same topic. But since we cant look at them all, let's drop any rows with identical combination of title and text.  Over 3k motions on 1991-01-25Let's first see how may motions are usually filed on any particular date<jupyter_code>sns.histplot(df.groupby('date').size(), kde=False, log_scale=True)
plt.title('Motions per day distribution')
plt.xlabel('Number of motions (log scale)')
plt.ylabel('Day count')
plt.show()<jupyter_output><empty_output><jupyter_text>While 3k+ on a day is certainly an outlier, it's not the only date that has a large number of motions associated with it. Below is the top ten most common titles for motions on 25th Jan 1991.<jupyter_code># Show top ten titles on 1991-01-25
display(df[df['date'].dt.date == dt(1991, 1, 25).date()].title.value_counts().head(10))<jupyter_output><empty_output><jupyter_text>There's not a high number of motions on any one title. I'm going to assume that these motions are valid data (maybe just filed on the wrong date).  Motions per year<jupyter_code>sns.lineplot(data=df.groupby(pd.Grouper(key='date', freq='Y')).size())
plt.title('Number of motions per year')
plt.xlabel('Year')
plt.ylabel('Count')
plt.show()<jupyter_output><empty_output><jupyter_text>Title lengths<jupyter_code># Plot distribution of title lengths
sns.histplot(df['title'].str.len(), binwidth=1)
plt.title('Motion title length distribution')
plt.xlabel('Number of characters')
plt.ylabel('Motion count')
plt.show()<jupyter_output><empty_output><jupyter_text>The distribution as a whole looks reasonable. We've seen that certain titles are the same for multiple documents which may explain the slightly ragged shape.  Motion text length<jupyter_code># Plot distribution of text lengths
sns.histplot(df['text'].str.len(), log_scale=True)
plt.title('Motion title length distribution')
plt.xlabel('Number of characters (log scale)')
plt.ylabel('Motion count')
plt.xlim(0, 100_000)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12347283597608526, 'fraction_numerical': 0.02742396672731999, 'mean_word_length': 4.7169390787518575, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '129116', 'license': 'MIT', 'n_tokens_mistral': 2732, 'n_tokens_neox': 2494, 'n_words': 748, 'path': 'notebooks/exploratory-data-analysis.ipynb', 'repo_name': 'erikgrip/swedish_parliament_motion_summarization', 'size': '7703'}