<jupyter_start><jupyter_text>Labels distribution is very unbalanced with Positive sentiment dominating<jupyter_code>plt.figure(figsize=[16,5])
for cl in class_names:
    _d = dat[dat['label_str']== cl]
    plt.bar(_d['ID'], _d['num_words'])
plt.legend(class_names)
plt.ylabel('number of words')
plt.title('Lenghth of sentences, colored by class_label')<jupyter_output><empty_output><jupyter_text>Findings from explorative data analysis:  Non-random order:There are some stong indications for a non-random order of the data, since we see that there are blocks of sentences with the class label - see between 180 and 200, were there is a block of negartive and neutral sentences.  Length of sentences is not indicative for sentimentAs expected, the length of a sentence does not provide class-relevent information.<jupyter_code>## investigating non-random order. Assumption is that neighboring sentences are from the same document

for i in range(180,200):
        print(i, ':   ', dat['Sentence'][i])
print('  ++++++ NOT SHOWN DUE TO PRIVACY POLICIES   ++++++ ')<jupyter_output>180 :    Data from the ongoing studies in infantile-onset SMA support that treatment with nusinersen has the potential to prolong overall and event-free survival
181 :    They also support the need for early treatment with nusinersen, since shortly after the loading phase was complete, the rate of ventilation among nusinersen-treated subjects separated from that of the controls
182 :    Taken together, these data demonstrate consistently that nusinersen has produced meaningful benefits across a broad range of SMA phenotypes
183 :    The consistency among the various key clinically meaningful endpoints, each of which provide independent measures central to the pathology of SMA, support significant efficacy in both pre-symptomatic and symptomatic subjects and across infantile-onset and later-onset SMA
184 :    The current studies do not provide evidence that the treatment effects persist with indefinite length of treatment, neither in infantile nor in later onset SMA
185 :    The CHMP co[...]<jupyter_text>Data is non-IIDThe above excerpt of the sentences indicate that all these sentences are related as most of them contain the term  "SMA" (which is likely the abbrevation for "spinal muscular atrophy"). Moreover, the sentences with the IDs 186 and 188 both mention the specific abbrevation "PAES", which also indicate that those sentences are occuring in the same context Problems due to non-IID samplesWithout knowing the origin of the sentences, it is not possible to split the data into valid training/validation and test sets. Certainly, splitting the data randomly will introduce a large bias. A chronological split may be more suitable, however sample size (i.e. number of unique documents) is unknown and it is expected to be very small (<15).  Loading data from pdf documents<jupyter_code># here, we get text data from pdf documents and thereby extract unlabeled sentences 

def get_sentences_from_pdf(pdf_path):
    os.system('pdftotext -f 3 {}'.format(pdf_path))
    with open(pdf_path.replace('.pdf', '.txt')) as f:
        text = f.read()
    #reformat text: several newlines are also end-of sentence. This is important for Headers 
    text = text.replace('\n\n', '. ')
    # remove remaining single-newlines
    text = text.replace('\n', ' ')
    sentences = text.split('. ')
    # remove sentences with less that 4 words
    sentences = [s for s in sentences if len(s.split())>=4]
    return sentences

unlabeled_data = {}
for filename in ['WC500057122.pdf', 'WC500135744.pdf']:
    unlabeled_data[filename] = get_sentences_from_pdf('../DS_NLP_Assignment/{}'.format(filename))<jupyter_output><empty_output><jupyter_text>Train/Val/Test SplittingAs discussed above, splitting the data is highly troublesome since the data is non-IID and additional knowledge of is required for a reasonable data split. However, in order to try trainig and evaluating some ML models, I proceed with using the last 66 samples for testing and the first 200 sentences for training for a joint testset. Thus, we pooling validation and trainnig to one holdout testset   Wordclouds for the Positive and Negative class<jupyter_code>#!pip3 install wordcloud==2.2.2
import wordcloud
plt.figure()
cloud = wordcloud.WordCloud(max_font_size=50, max_words=100, background_color="white").generate(' '.join(dat['Sentence'][dat.label_str == 'Positive']))
plt.figure(figsize=[10,7])
plt.imshow(cloud, interpolation="bilinear")
plt.title('wordcloud Positive')


cloud = wordcloud.WordCloud(max_font_size=50, max_words=100, background_color="white").generate(' '.join(dat['Sentence'][dat.label_str == 'Negative']))
plt.figure(figsize=[10,7])
plt.imshow(cloud, interpolation="bilinear")
plt.title('wordcloud Negative')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08241644176873572, 'fraction_numerical': 0.018061033838488685, 'mean_word_length': 5.240932642487047, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '466554', 'license': 'MIT', 'n_tokens_mistral': 1409, 'n_tokens_neox': 1272, 'n_words': 558, 'path': 'Data_Exploration.ipynb', 'repo_name': 'sharpheisenberg/EMA_health_feedback', 'size': '4817'}