<jupyter_start><jupyter_text>Initial Data descriptive table - All reports<jupyter_code>df_master = pd.read_csv("/Users/david/Nextcloud/Dokumente/Education/Uni Bern/Master Thesis/Analyzing Financial Climate Disclosures with NLP/Data/stoxx_inference/Firm_AnnualReport.csv")
# TODO: Detail, but report here the original dataset or the one that could be labelled/inferred? There are 1-2 reports with issues... df_reports_count = df_master.groupby('year')['is_inferred'].sum()

no_of_reports = len(df_master)

df = df_master.groupby("firm_name").first()
df = df[['country','icb_industry']]

df['years'] = df_master.groupby('firm_name')['year'].apply(list).apply(lambda l: f"{min(l)} - {max(l)}")
df['country'] = df['country'].str.upper()
df['icb_industry'] = df['icb_industry'].str.slice(start=3)
df.rename(columns={"country": "Country", "icb_industry": "Industry", "years": "Years"}, inplace=True)

# Add a total column at the end
total_row = df.nunique()
total_row["Years"] = no_of_reports

df.loc[f"Unique firms: {len(df)}"] = total_row

tables.export_to_latex(df, filename="stoxx50_firms.tex", index_names=False, add_midrule_at=1, make_bold_row_at=[2, -3])<jupyter_output><empty_output><jupyter_text>Pages<jupyter_code>import os
import yaml 

data_dir = "/Users/david/Projects/fin-disclosures-nlp/input_files/annual_reports"

def get_total_page_no(row):
    path = os.path.join(data_dir, f"{row.country}_{row.company}", f"{row.orig_report_type}_{row.year}", row.output_file)
    with open(path, 'r') as stream:
        content = yaml.safe_load(stream)
        return len(content['pages'])
    
df_master["total_page_no"] = df_master.apply(lambda x: get_total_page_no(x), axis=1)
    
df_master["total_page_no"].mean()
df_master<jupyter_output><empty_output><jupyter_text>Keyword list<jupyter_code>import pandas as pd
from data import dataframe_preparation
from data.utils import tables

pd.set_option('display.max_colwidth',1000)
vocabulary = dataframe_preparation.get_keywords_from_file("/Users/david/Projects/fin-disclosures-nlp/data/keyword_vocabulary.txt")
unigrams = [word for word in vocabulary if len(word.split(" ")) < 2]
bigrams = [word for word in vocabulary if len(word.split(" ")) > 1]

df_keyword_a = pd.DataFrame(data={"Unigrams": [", ".join(unigrams)]})
df_keyword_b = pd.DataFrame(data={"Bigrams": [", ".join(bigrams)]})
df_keyword = pd.concat([df_keyword_a, df_keyword_b], axis=1, sort=False)
tables.export_to_latex(df_keyword, filename="filter_vocabulary.tex")
print(len(unigrams) + len(bigrams))<jupyter_output>70<jupyter_text>Labels<jupyter_code>data_dir = os.path.join("/Users/david/Nextcloud/Dokumente/Education/Uni Bern/Master Thesis/Analyzing Financial Climate Disclosures with NLP/Labelling/annual reports/")
df_train = pd.read_pickle(os.path.join(data_dir, 'Firm_AnnualReport_Labels_Training.pkl'))

df_train.query("cro != 'OP'").groupby("cro_sub_type_combined").count()
t_d, t_l, tt_d, tt_l = cro_dataset.prepare_datasets(task="multi-label", cro_category_level="cro", should_filter_op=True, train_neg_sampling_strategy="all", test_neg_sampling_strategy="all")

t_l.sum()
t_l.sum()
t_l.sum()<jupyter_output><empty_output><jupyter_text>StatusWe now have 933 samples with additional samples of about 15/100 reports that will be added as soon as possible, so more than 1000 samples are likely.However, the categories both on main CRO level as well as the sub-categories are unbalanced.In comparison to the other main categories, physical risks (PR) are often **indirect** (about 55%), i.e. there is no explicit mentioning of climate change but rather a consequence. Often, these examples are also labelled as *vague*, i.e. more of the boilerplate type.<jupyter_code>fig, axes = plt.subplots(nrows=1, ncols=2)
groups.plot(ax=axes[0], kind='bar')
sub_groups.plot(ax=axes[1], kind='bar')<jupyter_output><empty_output><jupyter_text>Experimental setup For the high-level experimental setup, there are multiple options:- Step-by-Step: First classifiy if CRO relevant or not, then second step classify the main category. Then the sub-category...- End-To-End: Directly try to infer category--> I would propose for now that we attempt a Step-By-Step approach and actually start with the second task of classifying the main CRO categories to get an idea of how well the different methods perform. A benefit would this also have that getting the negative samples ise relatively straight-forward, i.e. just the opposite classes.As document inputs, I would propose that we use paragraphs for now but having a sentence approach then as comparison. *Challenge: How would the labels be assigned for sentences?*As evaluation metrics, F1-Scores (and Accuracy) are probably most useful. PreprocessingDepending on the downstream classification approach, preprocessing such as stop-words removal, stemming/lemmatization is required. *Challenge: How can we make sure that the tense of a sentense is not removed in this step?* ApproachesA set of approaches is selected to compare performance and whether the "heavy-weights" are actually necessary. Embedding options:        - BoW    - TF-IDF    - Word2Vec    - Doc2Vec    - Contextualized Word Embeddings (from BERT for example, i.e. only use the word embeddings from these models) without transfer learning, i.e. then select a classifier from belowClassifiers:        - Logistic regression    - SVM    - Random Forest    - XGBoost    - Neural NetAnd then the state of the art NLU models such as BERT/Roberta/...<jupyter_code>import sys
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

sys.path.append('..')
import data
from data.labels_postprocessing import process
df = pd.read_pickle("/Users/david/Nextcloud/Dokumente/Education/Uni Bern/Master Thesis/Analyzing Financial Climate Disclosures with NLP/Labelling/annual reports/Firm_AnnualReport_Labels_100_combined.pkl")
df = process(df)
groups = df.groupby(['cro']).size()
print(groups)
groups.plot(kind='barh')
sub_groups = df.groupby(['cro','cro_sub_type_combined']).size()
print(sub_groups)
sub_groups.plot(kind='barh')
indirects = df.groupby(['cro','indirect']).size()
print(indirects)
indirects.plot(kind='barh')
vague = df.groupby(['cro','vague']).size()
print(vague)
vague.plot(kind='barh')
past = df.groupby(['cro','past']).size()
print(past)
past.plot(kind='barh')
len(df.span_id.unique())
# spans = df[df.duplicated(subset=['report_id','span_id'], keep=False)]
# spans
df.keyword.count()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12339172221361029, 'fraction_numerical': 0.0065106185087583325, 'mean_word_length': 6.184855233853007, 'pattern_counts': {'":': 5, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '16518', 'license': 'MIT', 'n_tokens_mistral': 2200, 'n_tokens_neox': 2105, 'n_words': 552, 'path': 'notebooks/DescriptiveStatistics.ipynb', 'repo_name': 'dafrie/fin-disclosures-nlp', 'size': '6451'}