<jupyter_start><jupyter_text>Score AI: NLP Essay Scorer Why Score AITeachers remain stretched thin and under-supported. Automating some of the work they do will help them focus their time and energy on the most important aspects of their work -- building relationships with students that allow them to draw out their best selves. Scoring essays is one of the most time-intensive tasks that a teacher has to do, and the delay in feedback makes it harder for the student to use and internalize that feedback. Challenges- There is a lack of labelled essay data- There are no straightforward scoring metrics for essay organization- There are vastly different writing styles and essay lengths- Good writing transcends formulaic approaches Responses- Scraped data- Used background knowledge about essay grading to develop a scoring metric- Chose to only score argumentative essays to start and made scoring length-independent- Suggest use of scorer only for non-expert writers ScrapingOf the dozen or so essay sites that I investigated for scraping, only 2 of them freely offered the entire text of essays without a login. I used a scrapy script within the terminal to run spiders over those two sites to collect essays on about 40 different search terms for popular essay topics and save the necessary information in json files. The scrapy scripts are located in the same Github repo as this project. In order to get a ground truth for topic modelling, I used the spiders to scrape tags from the essay pages. Unfortunately, many of the essay tags turned out to be blank or meaningless.<jupyter_code>!pip install pyLDAvis

import pandas as pd
import spacy
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import csv
import sklearn
import re
import nltk
import pickle
import gensim
import gensim.corpora as corpora
import pyLDAvis
import pyLDAvis.gensim

from sklearn.preprocessing import StandardScaler, MinMaxScaler
from gensim.models import CoherenceModel, LsiModel, LdaModel
from string import punctuation
from itertools import starmap, combinations

%matplotlib inline
!python -m spacy download en_core_web_md
# Read in essay data from both sites, concatenate into one data frame
sm_df = pd.read_json('studymoose.json', orient='records')
phd_df = pd.read_json('phdessay.json', orient='records')
raw_df = pd.concat([sm_df, phd_df], ignore_index=True)
len(raw_df)<jupyter_output><empty_output><jupyter_text>Data Cleaning and PreprocessingThe essay texts are not in good shape to be topic modeled and tokenized. Some paragraphs are just newlines, some sentences are too short to be meaningful. Some sentences are citations/references. Essays are too long or contain too many one-sentence paragraphs. I will be removing these all from the dataset because they are interspersed randomly through the data and will not too greatly affect the makeup of the corpus.<jupyter_code># Find text that only newline characters
error_df = raw_df[raw_df['Text'].str.contains('\\n', regex=True)]
error_df = error_df[error_df['Text'].str.len() < 50]
drop_indices = error_df.index.tolist()
# Drop newlines
raw_df.drop(drop_indices, axis=0, inplace=True)
raw_df.reset_index(drop=True, inplace=True)
raw_df.tail()
# Split each paragraph into sentences
sentences = raw_df.Text.str.split(r'[.!?] ', expand=True)
# Create a sentence-level data frame by joining the sentences to the raw_df
sent_df = raw_df.drop('Text', axis=1).join(
    sentences.stack().reset_index(drop=True, level=1).rename('Text'))
sent_df.head()
# Remove lines that are too short
short_df = sent_df[sent_df['Text'].str.len() < 15]
short_indices = short_df.index.tolist()
sent_df.drop(short_indices, axis=0, inplace=True)

# Reset index
sent_df.reset_index(drop=True, inplace=True)
sent_df.tail()
# Remove lines that have too much punctuation (90% citations)
ref_indices = []
for i in range(len(sent_df)):
    text = sent_df.iloc[i, 4]
    punct = [c for c in text if c in punctuation]
    ratio = len(punct) / len(text)
    if ratio > 0.1:
        ref_indices.append(i)
        
sent_df.drop(ref_indices, axis=0, inplace=True)
len(sent_df)
# Count the sentences
sent_df['Sentences'] = sent_df.groupby(
    ['URL', 'Paragraph'])['Text'].transform('count')

sent_df.head()
# Remove essays that are too long
long_essays = sent_df[sent_df['Paragraph']>=66]['URL'].unique().tolist()
sent_df = sent_df[~sent_df['URL'].isin(long_essays)]
len(sent_df)
# Number each essay
essays = sent_df.URL.unique()
essay_dict = dict(zip(essays, range(len(essays))))


# Create an Essay column with the number of each essay
sent_df['Essay'] = sent_df['URL'].replace(essay_dict)
sent_df.reset_index(drop= True, inplace=True)
sent_df.tail()
# Find essays that have too many one sentence paragraphs
choppy_essays = []
for i, grp in sent_df.groupby('Essay'):
    short_paras = grp[grp['Sentences']==1]
    paras = grp['Paragraph'].unique().tolist()
    if len(short_paras) / len(paras) > 0.5:
        choppy_essays.append(i)
    
len(choppy_essays)
# Remove essays that have too many one sentence paragraphs

choppy_indices = sent_df[sent_df['Essay'].isin(choppy_essays)].index.tolist()

sent_df.drop(choppy_indices, inplace=True)

len(sent_df)
# Remove remaining one sentence paragraphs
one_liners = sent_df[sent_df['Sentences']==1].index.tolist()

sent_df.drop(one_liners, inplace=True)

len(sent_df)
# Number each essay
essays = sent_df.URL.unique()
essay_dict = dict(zip(essays, range(len(essays))))


# Create an Essay column with the number of each essay
sent_df['Essay'] = sent_df['URL'].replace(essay_dict)
sent_df.reset_index(drop= True, inplace=True)
sent_df.tail()
# Find the paragraphs per essay and sentences per paragraph
maxima = sent_df.groupby('Essay')['Paragraph'].max()

sents_per_para = sent_df.Sentences.value_counts().to_dict()

sents_per_para = {k: int(v/k) for k, v in sents_per_para.items()}

s_counts = [v for v in sents_per_para.values()]<jupyter_output><empty_output><jupyter_text>Distributions of Essay LengthFrom the plots above, it is easy to see that there is no standard size for essays in terms of the number of paragraphs in the essay or the number of sentences in a paragraph. After exploring the distribution of paragraphs that are one sentence long and the essays that are highly composed of one sentence paragraphs, I decided to remove them from the data set. I believe I can do so without significantly impacting the performance of my model. Removal of one sentence paragraphs and essays with a high percentage of on e sentence paragraphs in this model stems from the data collection process. I believe many of the one sentence paragraph to be a result of delineating paragraphs based on the html paragraph tag. In future iterations of this model where essay are collected from users, one sentence paragraphs will be scored at a z-score of -3.<jupyter_code># Visualize distributions of paragraphs and sentences per para
sns.set_style('darkgrid')

plt.figure(figsize=(15, 5))

plt.subplot(121)
sns.distplot(maxima, 
             kde=False,
             norm_hist=False)
plt.title('Distribution of Paragraphs per Essay')
plt.xlabel('Number of paragraphs')

plt.subplot(122)
sns.barplot([k for k in sents_per_para.keys()], 
            s_counts, 
            color='green')
plt.title('Distribution  of Sentences per Paragraph')
plt.xlabel('Sentences per paragraph')

plt.show()

print('Paragraphs per Essay:\nMean -- {}\nMedian -- {}\n\n'.format( 
maxima.mean(), maxima.median()))

s_list = []
for i in range (1, 33):
    if i in sents_per_para.keys():
        for j in range(sents_per_para[i]):
            s_list.append(i)

s_mean = np.mean(s_list)
mid = len(s_list)//2
s_median = (s_list[mid] + s_list[mid-1]) / 2


print('Sentences per Para:\nMean -- {}\nMedian -- {}\n\n'.format( 
    s_mean, s_median))<jupyter_output>/home/jeff/anaconda3/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if isinstance(obj, collections.Iterator):
/home/jeff/anaconda3/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return list(data) if isinstance(data, collections.MappingView) else data
/home/jeff/anaconda3/lib/python3.7/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval[...]<jupyter_text>Tokenizing and Topic Modeling Spacy TokensSince the model depends on the cosine similarity of sentence vectors, I needed to use one of the larger spacy tokenizers. I needed part of speech tags to get only semantically substantive words for topic modeling, but I could disable the dependency parser and named entity recognizer to cut down on memory usage and processing time. Topic ModelingI used the gensim package for topic modeling predominantly for its CoherenceModel implementation that would help me visualize how well the topic models describe the corpus.<jupyter_code># Load spacy model for tokenizing
try:
    with open('essays.pickle', 'rb') as fp:
        tokenized_sents = pickle.load(fp)
    print('Pretrained Model loaded!')
  
except:
    nlp = spacy.load(
        'en_core_web_md', 
        disable=['parser', 'ner'])
    
    # Run NLP to get tokens
    tokenized_sents = []
    i = 0
    for sent in sent_df.Text.tolist():
        i += 1
        if i % 5000 == 0:
            print('Processing {}'.format(i))
        doc = nlp(sent)
        # Get lemmas for each word
        tokenized_sents.append(doc)
    
    with open('essays.pickle', 'wb') as fp:
        pickle.dump(tokenized_sents, fp)
    
    print('New model trained!')
# Set sent_df Text column to tokenized sentences
sent_df['Text'] = tokenized_sents
sent_df.head()
# Group texts by essay to send essays through as documents for the corpus
essays = sent_df.groupby('Essay')['Text'].apply(lambda x: list(x))
# Extract only substantive words for topic modeling
essay_lemmas = []
for essay in essays:
    essay_lemmas.append([x.lemma_ for e in essay for x in e 
                       if not x.is_punct and 
                       x.pos_ in ['PRON', 'NOUN', 'VERB', 'ADJ'] and 
                       x.lemma_ not in ['be', 'have', 'do']])

# Create Dictionary
id2word = corpora.Dictionary([essay_lemmas[0]])

# Add the rest of the essays to the dictionary
for i in range(1, len(essays)):
    try:
        id2word.add_documents([essay_lemmas[i]])
    except:
        print('Error: {}'.format(i))
    if i % 500 == 0:
        print('Processed line {}'.format(i))
        
# Create Corpus
texts = []
for essay in essay_lemmas:
    texts.append([e for e in essay])

# Term Document Frequency
corpus = []
for i in range(len(texts)):
    try:
        corpus.append(id2word.doc2bow(texts[i]))
    except:
        print('Error: {}'.format(i))

# View
print(corpus[:1])
def get_lda_model(num_topics):
  
    '''Trains an LDA model and outputs model and list of information 
    to be used for model info data frame that will be useful for 
    selecting best topic model'''
  
    model_type = 'LDA'
    model_name = 'checkpoints/lda' + str(num_topics) + '.model'
  
    try:
        return LdaModel.load(model_name), [model_type, num_topics]
    
    except:
        print('Training new LDA {} Model'.format(num_topics))
        # Build LDA model
        lda_model = LdaModel(corpus=corpus,
                         id2word=id2word,
                         num_topics=num_topics,
                         update_every=1,
                         chunksize=100,
                         passes=10,
                         alpha='auto',
                         per_word_topics=True)
        # Save model
        lda_model.save(model_name)
    
        return lda_model, [model_type, num_topics]
# Get LDA Models
lda12_model, lda12_info = get_lda_model(num_topics=12)
lda16_model, lda16_info = get_lda_model(num_topics=16)
lda20_model, lda20_info = get_lda_model(num_topics=20)
lda22_model, lda22_info = get_lda_model(num_topics=22)
lda24_model, lda24_info = get_lda_model(num_topics=24)
lda26_model, lda26_info = get_lda_model(num_topics=26)
lda28_model, lda28_info = get_lda_model(num_topics=28)
def get_lsa_model(num_topics):
    '''Trains an LSA model and outputs model and list of information 
    to be used for model info data frame that will be useful for 
    selecting best topic model'''
  
    model_type = 'LSA'
    model_name = 'checkpoints/lsa' + str(num_topics) + '.model'
  
    try:
        return LsiModel.load(model_name), [model_type, num_topics]
    
    except:
        print('Training new LSA {} Model'.format(num_topics))

        # Build LSA Model
        lsa_model = LsiModel(corpus=corpus,
                        id2word=id2word,
                        num_topics=num_topics, 
                        chunksize=100)

        lsa_model.save(model_name)
    
        return lsa_model, [model_type, num_topics]
# Get LSA Models
lsa12_model, lsa12_info = get_lsa_model(num_topics=12)
lsa16_model, lsa16_info = get_lsa_model(num_topics=16)
lsa20_model, lsa20_info = get_lsa_model(num_topics=20)
lsa22_model, lsa22_info = get_lsa_model(num_topics=22)
lsa24_model, lsa24_info = get_lsa_model(num_topics=24)
lsa26_model, lsa26_info = get_lsa_model(num_topics=26)
lsa28_model, lsa28_info = get_lsa_model(num_topics=28)
def get_model_info(df, topic_model, model_type, n_topics, corpus):
    '''Add LDA and LSA model information to a data frame 
    in order to compare, plot, and select the best model'''
  
    print('Getting model info for {} {}'.format(model_type, n_topics))
  
    # Add model info to dataframe
    model = model_type
    n = n_topics
    topics = topic_model.print_topics(num_topics=n_topics)
    if model == 'LDA':
        perplexity = topic_model.log_perplexity(corpus)
    else:
        perplexity = 'N/A'

    coherence_model = CoherenceModel(model=topic_model,
                                      texts=texts, 
                                      dictionary=id2word, 
                                      coherence='c_v')

    coherence = coherence_model.get_coherence()

    return pd.concat([df, 
                    pd.DataFrame.from_dict({'Model Type': model, 
                                            'Num_Topics': n, 
                                            'Topics': topics, 
                                            'Perplexity': perplexity, 
                                            'Coherence': coherence})], 
                   ignore_index=True)
  
# Retrieve model_info data frame or create a new one
try:
    model_info = pd.read_csv('model_info.csv')
  
except:
  
    models = [[lda12_model, lda12_info], 
            [lda16_model, lda16_info], 
            [lda20_model, lda20_info], 
            [lda22_model, lda22_info], 
            [lda24_model, lda24_info], 
            [lda26_model, lda26_info], 
            [lda28_model, lda28_info], 
            [lsa12_model, lsa12_info], 
            [lsa16_model, lsa16_info], 
            [lsa20_model, lsa20_info], 
            [lsa22_model, lsa22_info], 
            [lsa24_model, lsa24_info], 
            [lsa26_model, lsa26_info], 
            [lsa28_model, lsa28_info]]

    model_info = pd.DataFrame(columns=['Model Type', 
                                   'Num_Topics', 
                                   'Topics', 
                                   'Perplexity', 
                                   'Coherence'])

    for model in models:
        model_info = get_model_info(model_info, 
                                model[0], 
                                model[1][0], 
                                model[1][1], 
                                corpus=corpus)
    
    model_info.to_csv('model_info.csv')

model_info.head()<jupyter_output><empty_output><jupyter_text>Comparing Topic ModelsLDA will achieve different results with each model because of the random weight initiation, but through 10 different versions of these LDA models, there was always a peak at 16, a relatively similar peak at 22 and an upward trend from 26 to 28 -- all hovering around 0.50 coherence. While the LSA models do not perform as well, they seem to confirm a peak at 16. Prior to having run any topic models, my mentor and I discussed how many topics we might expect. There were 40 search terms, but many of them could be lumped into similar topics or were even synonyms to try to capture more essays about the same topic. We determined that there were about 16 topics and were surprised to see that intuition confirmed by the topic models. Though other numbers of topics for LDA may produce similar or better coherences, we decided to go with the consensus of two different modeling functions and human expertise to use the LDA16 model going forward.<jupyter_code># Compare topic models
plt.figure(figsize=(10, 5))

sns.lineplot(x='Num_Topics', 
             y='Coherence', 
             hue='Model Type', 
             data=model_info)
plt.title('Model Coherence by Number of Topics')

plt.show()<jupyter_output>/home/jeff/anaconda3/lib/python3.7/site-packages/scipy/stats/stats.py:1713: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
  return np.add.reduce(sorted[indexer] * weights, axis=axis) / sumval<jupyter_text>Topics Worth KeepingThough I tried dropping parts of speech and modal verbs that would not contribute to determining the topic, the model still generated a few topics based on keywords too generic to provide a meaningful topic. For this reason, I will refrain from assigning topics 1,  7,  8, or 13 as the dominant topic for any essay.<jupyter_code># Visualize the topics
pyLDAvis.enable_notebook()
vis = pyLDAvis.gensim.prepare(lda16_model, corpus, id2word)
vis
# Code attributed to:
# https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/

def format_topics_sentences(ldamodel=lda16_model, corpus=corpus, texts=texts):
    # Init output
    df = pd.DataFrame()

    # Get main topic in each document
    for i, row in enumerate(ldamodel[corpus[:-1]]):
        row = sorted(row[0], key=lambda x: (x[1]), reverse=True)
        # Get the Dominant topic, 
        # Percent Contribution and Keywords for each document
        for j, (topic_num, prop_topic) in enumerate(row):
            # Prevent dominant topics 7 or 8, they're meaningless
            if j in [0, 1, 2, 3, 4] and topic_num not in [1.0, 7.0, 8.0, 13.0]:
                wp = ldamodel.show_topic(topic_num)
                topic_keywords = ", ".join([word for word, prop in wp])
                df = df.append(
                    pd.Series([int(topic_num), 
                               round(prop_topic,4), 
                               topic_keywords]), 
                    ignore_index=True)
                break
                
            else:
                continue
                
    df.columns = ['Dominant_Topic', 
                  'Percent_Contrib', 
                  'Topic_Keywords']

    # Add original text to the end of the output
    contents = pd.Series(texts)
    df = pd.concat([df, contents], axis=1)
    return(df)


topics_df = format_topics_sentences(ldamodel=lda16_model, 
                                    corpus=corpus, 
                                    texts=texts)

# Format
topics_df.reset_index(drop= True, inplace=True)

# Set column names
topics_df.columns = ['Dominant_Topic', 
                     'Topic_Percent_Contrib', 
                     'Keywords', 
                     'Text']

# Create Essay number column
topics_df['Essay'] = sent_df.Essay.unique().tolist()

# Show
topics_df.head(10)
pd.options.display.max_rows = 100
# Code attributed to: 
# https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/
# Group top 5 essays under each topic
sorted_topics = pd.DataFrame()

grouped = topics_df.groupby('Dominant_Topic')

for i, grp in grouped:
    sorted_topics = pd.concat([sorted_topics, 
                               grp.sort_values(['Topic_Percent_Contrib'], 
                                               ascending=[0]).head(5)], 
                              axis=0)

# Reset Index    
sorted_topics.reset_index(drop=True, inplace=True)

# Format
sorted_topics.columns = ['Topic_Num', 
                         'Topic_Percent_Contrib', 
                         'Keywords', 
                         'Text', 
                         'Essay']

# Show
sorted_topics<jupyter_output><empty_output><jupyter_text>Dominant Topic DistributionThe dominant topics are relatively well distributed once the more generic topics are removed. No one topic dominates the essay and a cursory look confirms the distribution I would expect based on the number of essays I was able to scrape for each topic.<jupyter_code># Number of Documents for Each Topic
topic_counts = topics_df.groupby(
    'Keywords')['Dominant_Topic'].count()
topic_counts
# Isolate tags for tokenization and topic modeling
tags = sent_df.groupby('Essay')['Tags'].apply(lambda x: np.unique(x)[0])
# Tokenize tags and prepare for topic model
tokenized_tags =[]

nlp = spacy.load('en')
i = 0
for tag in tags:
    i += 1
    if i % 500 == 0:
        print('Processing {}'.format(i))
    tag = ' '.join(tag)
    doc = nlp(tag)
    # Get lemmas for each word
    tokenized_tags.append([token.lemma_ for token in doc if not token.is_punct])
    
tag_corpus = []
for i in range(len(tokenized_tags)):
    try:
        tag_corpus.append(id2word.doc2bow(tokenized_tags[i]))
    except:
        print('Error: {}'.format(i))

# Get topics for tags
tags_df = format_topics_sentences(ldamodel=lda16_model, 
                                         corpus=tag_corpus, 
                                         texts=tokenized_tags)

tags_df.columns = ['Tag_Topic', 
                   'Tag_Percent_Contrib', 
                   'Tag_Keywords', 
                   'Tags']

tags_df.head()
# Add tags and topics to the essay-topic data frame
topics_df = topics_df.join(tags_df)
topics_df.head()<jupyter_output><empty_output><jupyter_text>Topic AccuracyThe topic accuracy rose from about 25% to around 38% when the generic topics were pruned. Still, I believe the main thing holding back the accuracy of the topics to be the fact that my labels (the tags scraped from the websites) were not that descriptive of the essay topics to begin with. Given more time, I would find a way to include the search term with the output of the scraper so I could use that as the label instead of the scraped tags. For now, I will show examples of essays and tags where the topic labels match and examples where they don't match to try to illustrate the issue of poor labels over poor topic models.<jupyter_code># Determine the percentage of essay topics that match their tag topic
matches = np.where(
    topics_df['Dominant_Topic'] == topics_df['Tag_Topic'])[0]

l = len(matches)

print('Tag topic to essay topic accuracy: {}'.format(l/len(topics_df)))
# Separate matches and non-matches
match_df = topics_df.loc[matches]
nonmatch_df = topics_df[~topics_df.index.isin(matches)]

# Show a random sample of matches
match_df.sample(25)
# Show a random sample of non-matches
nonmatch_df.sample(25)
# Read in the paragraph organization scores or generate them
try:
    score_df = pd.read_csv('score_df.csv')
  
except:
  
    def get_similarity(x, y):
        # Get cosine similarity between sentence vectors
        return x.similarity(y)
  
  # Establish a column for paragraph level scores
    sent_df['Para_score'] = 0

    for i, grp in sent_df.groupby(['Essay', 'Paragraph']):
        idx = grp.index
  
        if i[0] % 500  == 0:
            print('Processing {}'.format(i))
  
        # Set the paragraph level score as the mean of the cosine
        # similarity scores of all combinations of sentences within
        # a paragraph
        sent_df.loc[idx, 'Para_score'] = np.mean([x for x in starmap(
            get_similarity, combinations(grp['Text'], 2))])
  
        if i[0] % 500  == 0:
          print('Para_score {} = {}'.format(idx, sent_df.loc[idx, 'Para_score']))
    
    sent_df.to_csv('score_df.csv')
  
    score_df = sent_df.copy()

score_df.head()<jupyter_output><empty_output><jupyter_text>Paragraph Scores DistributionThe scores based on the average cosine similarity of sentence vectors within the paragraph tends to be a relatively normal distribution close to one with a long left tail. For this reason, I left all words in the sentence rather than reducing it to semantically-relevant lemmas as in the topic model. When the sentences are reduced in that way, it shifts the distribution even closer to one and increases the kurtosis. The negative skew seems to come at least in part from paragraphs containing citations/references, headings, websites, or other text that would not contribute to the body of a paragraph. This issue stems again from the data collection method and these are the pieces of text I was unable to filter out in the earlier cleaning and preprocessing. Essay Scores DistributionThe distribution of essay scores is similar to that of paragraph scores while clipping some of the tail on the left. This lets me know that the poor paragraph scores were not all concentrated in a few essays.<jupyter_code># Plot the distribution of Paragraph scores
plt.figure(figsize=(10,5))

sns.distplot(score_df['Para_score'])
plt.title('Distribution of paragraph-level organization scores')
plt.show()

print('Paragraph Score Min: {}'.format(score_df['Para_score'].min()))
print('Paragraph Score Max: {}'.format(score_df['Para_score'].max()))
print('Paragraph Score Mean: {}'.format(score_df['Para_score'].mean()))

# Explore paragraphs that comprise the long tail
score_df[score_df['Para_score'] < 0.6]
# Initialize an essay-level organization score column
score_df['Essay_score'] = 0

for i, grp in score_df.groupby('Essay'):
    idx = grp.index
  
    if i % 500 == 0:
        print('Processing {}'.format(i))
  
  # Set essay score to the mean of paragraph-level organization scores
    score_df.loc[idx, 'Essay_score'] = np.mean(
      [x for x in grp['Para_score'].unique()])
  
    
score_df.head()
# Plot the distribution of Essay scores
plt.figure(figsize=(10,5))

sns.distplot(score_df['Essay_score'])
plt.title('Distribution of essay-level organization scores')
plt.show()

print('Essay Score Min: {}'.format(score_df['Essay_score'].min()))
print('Essay Score Max: {}'.format(score_df['Essay_score'].max()))
print('Essay Score Mean: {}'.format(score_df['Essay_score'].mean()))
# Scale Essay scores based on their z-score
scaler = StandardScaler()

score_df['z_score'] = scaler.fit_transform(
    np.array(score_df['Essay_score']).reshape(-1,1))

print(score_df['z_score'].min())
print(score_df['z_score'].max())
# Cut of the tails of the Essay z-scores by assigning them to 3 or -3
score_df['z_score'] = score_df['z_score'].apply(lambda x: x if x <= 3 else 3)
score_df['z_score'] = score_df['z_score'].apply(lambda x: x if x > -3 else -3)

print(score_df['z_score'].min())
print(score_df['z_score'].max())
# Add 3 to all z-scores to get rid of negative values
score_df['z_score'] += 3

print(score_df['z_score'].min())
print(score_df['z_score'].max())
# Add essay topic coherence scores to the data frame
on_topic_dict = dict(zip(
    topics_df['Essay'], topics_df['Topic_Percent_Contrib']))

score_df['Topic_score'] = score_df['Essay'].replace(on_topic_dict)

score_df.head()
# Drop the one essay with null values
score_df.dropna(how='any', axis=0, inplace=True)
# Scale essay topic coherence scores between 0 and 1
mmscaler = MinMaxScaler()

score_df['on_topic'] = mmscaler.fit_transform(
    np.array(score_df['Topic_score']).reshape(-1,1))

# Get a final organization score by mutliplying the essay org z-score
# by the essay topic coherence score
score_df['Org_score'] = score_df['z_score'] * score_df['on_topic']

score_df.head()<jupyter_output><empty_output><jupyter_text>Final Organization ScoreThe topic model is not perfect, so the distribution of final organization scores (comprised of essay-level org z-scores with the tails clipped and made positive times topic contrib scores scaled between 0 and 1) did not end up on a 0 - 6 scale as I had hoped. However, it seemed to me there was a way to divide the distribution of organization scores to give it the types of grades teachers might give to their students on for essay organization. - 0 -- This writer cannot demonstrate having learned about organization-1 -- This writer demonstrates minimal organization -- the essay is divided into paragraphs, some remain on topic- 2 -- This writer is approaching sufficient organization -- many paragraphs remain on topic- 3 -- This writer is proficient at organization -- nearly all paragraphs are on topic and contribute to the thesis- 4 -- This writer is has advanced organization -- all paragraphs remain on topic and contribute to the thesisAs with the topic model accuracy, I will have to pull out examples to qualitatively determine whether these labels are accurate.<jupyter_code># Plot the distribution of Organization scores
m = score_df['Org_score'].mean()
maximum = score_df['Org_score'].max()
minimum = score_df['Org_score'].min()
std = score_df['Org_score'].std()

plt.figure(figsize=(15,7))

sns.distplot(score_df['Org_score'])

# Assign teacher interpretable scores -- "Grade on a curve"
plt.axvspan(minimum, (m-(1.5*std)), facecolor='r', alpha=0.4, label='0')
plt.axvspan((m-(1.5*std)), (m-std), facecolor='orange', alpha=0.4, label='1')
plt.axvspan((m-std), (m+std), facecolor='y', alpha=0.4, label='2')
plt.axvspan((m+std), (m+(3*std)), facecolor='g', alpha=0.4, label='3')
plt.axvspan((m+(3*std)), maximum, facecolor='lime', alpha=0.4, label='4')

plt.legend(title='Grades')

plt.title('Distribution of final organization scores')
plt.show()

print('Minimum: {}'.format(minimum))
print('Maximum: {}'.format(maximum))
print('Mean: {}'.format(m))
print('Standard Deviation: {}'.format(std))
# Sample best, average, and worst essays
expert = score_df[score_df['Org_score']>3]
average = score_df[score_df['Org_score']<1.1]
average = average[average['Org_score']>0.8]
worst = score_df[score_df['Org_score']<0.25]

# Show a random sample of matches
ex_sample = expert.sample(25)['URL'].tolist()
av_sample = average.sample(25)['URL'].tolist()
w_sample = worst.sample(25)['URL'].tolist()
print('Expert essays:\n')
for url in ex_sample:
    print(url + '\n')
print('\n\nAverage essays:\n')
for url in av_sample:
    print(url + '\n')
print('\n\nBad essays:\n')
for url in w_sample:
    print(url + '\n')<jupyter_output>Expert essays:

https://studymoose.com/the-relationship-between-media-and-politics-essay

https://studymoose.com/girl-child-essay

https://studymoose.com/euthanasia-argumentative-essay-essay

https://studymoose.com/properties-of-hydrates-essay

https://studymoose.com/how-has-china-been-affected-by-globalization-essay

https://studymoose.com/the-relationship-between-media-and-politics-essay

https://studymoose.com/equality-and-human-rights-commissio-essay

https://studymoose.com/the-power-of-religion-essay

https://studymoose.com/the-relationship-between-media-and-politics-essay

https://studymoose.com/the-fukushima-nuclear-power-plant-accident-essay

https://studymoose.com/cultural-diversity-race-and-ethnicity-4-essay

https://studymoose.com/freedom-of-speech-2-essay

https://studymoose.com/were-the-upper-and-lower-canada-rebellions-in-1837-a-success-essay

https://studymoose.com/euthanasia-argumentative-essay-essay

https://studymoose.com/the-color-of-water-analysis-essay

https://phd[...]
{'fraction_non_alphanumeric': 0.09956763818470248, 'fraction_numerical': 0.013281906124607298, 'mean_word_length': 3.4615598112683874, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 18, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '42678', 'license': 'Apache-2.0', 'n_tokens_mistral': 10349, 'n_tokens_neox': 9734, 'n_words': 3130, 'path': 'org_score/ScoreAI.ipynb', 'repo_name': 'tjeffkessler/score_ai', 'size': '32149'}