<jupyter_start><jupyter_text>Assignment 1Instructions:- Please read the problem description carefully- Make sure to complete all requirement (shown as bullets) . In general, it would be much easier if you complete the requirements in the order as shown in the problem description  Q1. Define a function to analyze the frequency of words in a string (3 points) - Define a function which does the following:     * has a string as an input     * splits the string into a list of tokens by space.          - e.g., "it's a hello world!!!" will be split into two tokens ["it's", "a","hello","world!!!"]        * if a token starts with or ends with one or more punctuations, remove these punctuations, e.g. "world!!!" -> "world".(hint, you can import module *string*, use *string.punctuation* to get a list of punctuations (say *puncts*), and then use function *strip(puncts)* to remove leading or trailing punctuations )      * remove the space surrounding each token     * only keep tokens with 2 or more characters, i.e. *len*(token)>1      * converts all tokens into lower case      * create a dictionary to save the count of each uninque word      * sort the dictionary by word count in descending order     * return the sorted dictionary<jupyter_code>from string import punctuation as punc

def stripPunc(token: str)-> str:
    res = ''
    for t in token:
        if t not in punc:
            res+=t
    return res.lower().strip() 

def tokensList(string: str)->dict:
    string = list(map(stripPunc,string.split(" ")))
    res = {}
    for s in string:
        if not s: continue
        if res.get(s) == None:
            res[s] = 1
        else:
            res[s] += 1
    res = dict(sorted(res.items(),key = lambda x: -x[1]))
    return res

# test your code
text = '''People with moderately to severely compromised immune systems are especially vulnerable to COVID-19, 
    and may not build the same level of immunity to 2-dose vaccine series compared to people 
    who are not immunocompromised. This additional dose intended to improve immunocompromised 
    people’s response to their initial vaccine series. CDC recommends that people with moderately to 
    severely compromised immune systems receive an additional dose '''

tokensList(text)<jupyter_output><empty_output><jupyter_text>Q2. Define a function to analyze a numpy array (4 points) - Assume we have an array $X$ which contains term frequency of each document. In this array, each row presents a document, each column denotes a word, and each value, say $x_{i,j}$,  denotes the frequency of the word $j$ in document $i$. Therefore, if there are  $m$ documents, $n$ words, $X$ has a shape of $(m, n)$.  Define a function which:      * Take $X$ as an input.      * Divides word frequency $x_{i,j}$ by the total number of words in document $i$. Save the result as an array named $tf$ ($tf$ has shape of $(m,n)$).      * Calculate the document frequency $df_j$ for word $j$, e.g. how many documents contain word $j$. Save the result to array $df$ ($df$ has shape of $(n,)$).      * Calculate $idf_j =  ln(\frac{|m|}{df_j})+1$. m is the number of documents. The reason is, if a word appears in most documents, it does not have the discriminative power and often is called a `stop` word. The inverse of $df$ can downgrade the weight of such words. ($idf_j$ has shape of $(n,)$).      * For each $x_{i,j}$, calculates $tf\_idf_{i,j} = tf_(i,j) * idf_j$. ($tf\_idf$ has shape of $(m,n)$).      * Now, please print the following:          * print the index of the longest document          * print the indexes of words with the top 3 largest $df$ values          * for the longest document, print the indexes of words with top 3 largest values in the $tf\_idf$ array (use the index you got previously).       * Return the $tf\_idf$ array. - Note, for all the steps, **do not use any loop**. Just use array functions and broadcasting for high performance computation.<jupyter_code>import numpy as np
import pandas as pd


def text_analyzer_q2(X): #takes array as an input

    tf_idf = None

    

    a1 = np.sum(X,axis=1) #for normalizing friquency selects each elements

    idx = np.where(a1 == max(a1))[0] # index with the longest document

    print("Indexes of the longest documents:", idx)

    

    tf = np.divide(X.T,a1) #decides word frequency by length of the document and saves result in tf

    

    a2 = np.where(tf.T>0,1,0)

    df = np.sum(a2,axis=0) #calculates the document frequency of each word in df

    

    df_max = np.argsort(df)

    print("Indexes of words with the top 3 largest df values", df_max[:-4:-1])

    tf_idf = tf.T/df #tf/df

    f = np.argsort(tf_idf) #it will count the indexs according to word which was appered most of the time in ascending order

    

    print("Indexes of words with top 3 largest tf_idf values in the longest document:", f[idx,-3:][0]) # prints only indexes of words with top 3 largest values in the tf_idf array

    return tf_idf

# dtm.csv is a csv file for test. 
# It contains word counts in a few documents
dtm = pd.read_csv("C://Users//prans//Downloads//dtm.csv")
text_analyzer_q2(dtm.values)<jupyter_output>Indexes of the longest documents: [0 1]
Indexes of words with the top 3 largest df values [ 4  5 15]
Indexes of words with top 3 largest tf_idf values in the longest document: [2 3 6]<jupyter_text>Q3. Define a function to analyze a dataset using pandas (3 points)- The dataset "emotion.csv" contains a number of text and ten types of sentiment scores. Define a function named `emotion_analysis` to do the follows:   * Read "emotion.csv" as a dataframe with the first row in the csv file as column names   * Count the number of samples labeled for each emotion (i.e. each value in the column "emotion). Print the counts.   * Add a column "length" that calculates the number of words for each text. (hint: "apply" function to split the text by space and then count elements in the resulting list)   * Show the min, max, and mean values of sadness, happiness, and text length for each emotion. Print the results.   * Create a cross tabulation of average anxiety scores. Use "emotion" as row index, "worry" as column index, and "anxiety" as values. Print the table. - This function does not have any return. Just print out the result of each calculation step.<jupyter_code>import pandas as pd
import numpy as np

def emotion_analysis():
    # Read "emotion.csv" as a dataframe with the first row in the csv file as column names
    data = pd.read_csv("C://Users//prans//Downloads//emotion.csv")

    # Count the number of samples labeled for each emotion
    emotions = list(data['emotion'].unique())
    for emotion in emotions:
        df_len = len(data[data['emotion']==emotion])
        print('Number of rows for emotion',emotion,'are: ',df_len)

    print('\n')
    # Add a column "length" that calculates the number of words for each text.
    length = [len(row.split()) for row in data['text']]
    data['length'] = length
    #print(data['length'])

    # Show the min, max, and mean values of sadness, happiness, and text length for each emotion.
    emotions = ['Sadness','Happiness'] # using jou instead of happiness
    for emotion in emotions:
        df_length = list(data[data['emotion']==emotion]['length'])
        print('For emotion',emotion,': min value: ',min(df_length),', max value: ',max(df_length),', mean value: ',np.mean(df_length))

    # Create a cross tabulation of average anxiety scores. Use "emotion" as row index, "length" as column index, and "anxiety" as values.

    return(None)

emotion_analysis()
emotion_analysis()<jupyter_output>worry  emotion  anger  disgust  fear  anxiety  sadness  happiness  \
0      3  Sadness      5        5     3        7        7          2   
1      8  Anxiety      6        7     7        8        6          4   

   relaxation  desire                                               text  
0           4       5  It is less an much an issue of how it affects ...  
1           3       1  I am concerned that the true impact of the cur...  
===The number of samples labeled for each emotion===
Anxiety       1381
Sadness        357
Relaxation     333
Fear           230
Anger          107
Happiness       39
Desire          27
Disgust         17
Name: emotion, dtype: int64


=== min, max, and mean values of sadness, happiness, and text length for each emotion===
             sadness           happiness                length           
                mean amin amax      mean amin amax        mean amin  amax
emotion                                                                  
Anger       [...]<jupyter_text>Bonus question (3 points)1. Suppose your machine learning model returns a list of probabilities as the output. Write a function to do the following:    - Given a threshold, say $th$, if a probability > $th$, the prediction is positive; otherwise, negative    - Compare the prediction with the ground truth labels to calculate the confusion matrix as [[TN, FN],[FP,TP]], where:        * True Positives (TP): the number of correct positive predictions        * False Positives (FP): the number of postive predictives which actually are negatives        * True Negatives (TN): the number of correct negative predictions        * False Negatives (FN): the number of negative predictives which actually are positives    - Calculate **precision** as $TP/(TP+FP)$ and **recall** as $TP/(TP+FN)$    - return precision and recall. 2. Call this function with $th$ varying from 0.05 to 0.95 with an increase of 0.05. Plot a line chart to see how precision and recall change by $th$<jupyter_code>prob =np.array([0.28997326, 0.10166073, 0.10759583, 0.0694934 , 0.6767239 ,
       0.01446897, 0.15268748, 0.15570522, 0.12159665, 0.22593857,
       0.98162019, 0.47418329, 0.09376987, 0.80440782, 0.88361167,
       0.21579844, 0.72343069, 0.06605903, 0.15447797, 0.10967575,
       0.93020135, 0.06570391, 0.05283854, 0.09668829, 0.05974545,
       0.04874688, 0.07562255, 0.11103822, 0.71674525, 0.08507381,
       0.630128  , 0.16447478, 0.16914903, 0.1715767 , 0.08040751,
       0.7001173 , 0.04428363, 0.19469664, 0.12247959, 0.14000294,
       0.02411263, 0.26276603, 0.11377073, 0.07055441, 0.2021157 ,
       0.11636899, 0.90348488, 0.10191679, 0.88744523, 0.18938904])

truth = np.array([1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 0,
       0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 0, 1, 0])
def evaluate_performance(prob, truth, th):
    conf = [[0, 0], [0, 0]]
    
    # add your code here

   

    return 

# Test with threhold 0.5<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09481787457754412, 'fraction_numerical': 0.056984603830266614, 'mean_word_length': 2.9324473975636765, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '567371', 'license': 'MIT', 'n_tokens_mistral': 3575, 'n_tokens_neox': 3136, 'n_words': 1406, 'path': 'HW1/Homework_1.ipynb', 'repo_name': 'ipsavani/BIA-660-WebMining', 'size': '10652'}