<jupyter_start><jupyter_text>Sociology 128D: Mining Culture Through Text Data: Introduction to Social Data Science  Notebook 12: Modeling Ratings and Sentiment using PythonIn this notebook, we are going to explore a range of tools for statistical analysis in Python. To do this, we are going to use the text of Yelp reviews as well as various metadata. We will conclude with regression tables that can be displayed in the notebook, copied to a word processor, or rendered in HTML or LaTeX. In the optional exercises, you will adapt the code provided in the notebook to answer your own research question.  SetupYou will likely need to install `contractions`,  `num2words`, `pingouin`, `stargazer`, `statsmodels`, `unidecode`, and `vaderSentiment`. You can install most of these using `conda` (if using Anaconda), but you will need to install `contractions` and `stargazer` using `pip` (see below).`conda install -c conda-forge pingouin` `conda install -c conda-forge num2words` `conda install -c conda-forge statsmodels` `conda install -c conda-forge unidecode` `conda install -c conda-forge vadersentiment` `pip install contractions` `pip install stargazer`<jupyter_code>import contractions
import json
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import pingouin
import re
import seaborn as sns
import spacy
import statsmodels.formula.api as smf

from collections import defaultdict, Counter
from num2words import num2words
from pingouin import cronbach_alpha
from scipy.stats import pearsonr, spearmanr
from spacy.lang.en.stop_words import STOP_WORDS as spacy_stopwords
from stargazer.stargazer import Stargazer
from statsmodels.stats.weightstats import ttest_ind
from unidecode import unidecode
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
np.random.seed(1054)<jupyter_output><empty_output><jupyter_text>DataFor this notebook, we are going to use the Yelp Open Dataset, which you can find [here](https://www.yelp.com/dataset). You'll have to click 'Download Dataset', agree to the terms, and click 'Download JSON'. It's a large download: ~5GB compressed and ~11GB once you've uncompressed it. The dataset has 8,635,403 reviews of businesses including text, a rating out of five stars, and various other information. The main file, yelp_academic_dataset_review.json, is quite large, and we are going to take a random sample of it.<jupyter_code>os.listdir("yelp_dataset")<jupyter_output><empty_output><jupyter_text>First, let's confirm the number of reviews. We'll iterate through the file line by line, counting the lines by incrementing num_reviews. Then we'll use `np.random.choice` to identify the *indices* of random elements from an array of the same length. Then we'll loop back through the lines of the main file and keep the lines whose index is a match for our sample. This approach allows us to avoid loading the full dataset into memory.<jupyter_code>num_reviews = 0

with open("yelp_dataset/yelp_academic_dataset_review.json", "r", encoding="utf-8") as reader:
    for line in reader:
        num_reviews += 1
num_reviews<jupyter_output><empty_output><jupyter_text>Using `np.arange` with num_reviews gives us an array of the same length as num_reviews from 0 to num_reviews-1. This is more or less doing the same thing as the more familiar `range` function, but it returns a NumPy array. You can use either, but `np.arange` is a good tool to have if you start working with NumPy more.`np.random.choice` will take a sample of the values from the array of size size without replacement.<jupyter_code>sample_indices = np.random.choice(np.arange(num_reviews), size=10000, replace=False)
len(sample_indices)<jupyter_output><empty_output><jupyter_text>Now, we will loop through the lines of the main file using `enumerate` to count as we go. For the first line, i will be 0. For the last line, i will be 8635402. For each line, if the corresponding value of i is in sample_indices, we will append the line to the list we've called sample.<jupyter_code>sample = []

with open("yelp_dataset/yelp_academic_dataset_review.json", "r", encoding="utf-8") as reader:
    for i, line in enumerate(reader):
        if i in sample_indices:
            sample.append(line.strip())<jupyter_output><empty_output><jupyter_text>JSON is a bit like a dictionary. If we use `json.loads`, we can turn each line (a string) into a dictionary.<jupyter_code>sample[0]
sample = [json.loads(s) for s in sample]
sample[0]
sample[0]["stars"]
sample[0]["text"]<jupyter_output><empty_output><jupyter_text>sample is a list of 10,000 dictionaries, each corresponding to one review. However, each fo these dictionaries has the same keys. We can use `pd.Dataframe` directly on this list to create a dataframe, and it will use the keys of the dictionaries as the columns.<jupyter_code>df = pd.DataFrame(sample)
df.head()
df.date.min()
df.date.max()<jupyter_output><empty_output><jupyter_text>Some quick preprocessing...<jupyter_code>def fix_ordinal_nums(word: str) -> str:
    ord_num_reg = r"\d+[(st)(nd)(rd)(th)]"
    try:
        if any(re.findall(ord_num_reg, word)):
            word = re.sub("[(st)(nd)(rd)(th)]", "", word)
            word = num2words(word, lang="en", to="ordinal")
            
        return word
    
    except:
        
        return word


def preprocess_post(post: str) -> str:
    """
    Tokenize, lemmatize, remove stop words, 
    remove non-alphabetic characters.
    """
    post = unidecode(str(post))
    post = contractions.fix(post)
    post = [word.lemma_ for word in nlp(post) if (word.text not in spacy_stopwords) & (len(word.text) > 1)]
    post = " ".join([fix_ordinal_nums(word) for word in post])
    post = re.sub("[^a-z]", " ", post.lower())
    
    return re.sub("\s+", " ", post).strip()
    

nlp = spacy.load("en_core_web_sm", disable=["ner", "parser"])
# Convert to list
docs = df.text.tolist()

%time df["preprocessed"] = [preprocess_post(doc) for doc in docs]<jupyter_output><empty_output><jupyter_text>We aren't going to use the user_id column, and review_id is also an eyesore when we try to display the dataframe. Let's drop those columns and save our work.<jupyter_code>df.drop(labels=["review_id", "user_id"], axis=1, inplace=True)
df.to_json("yelp_sample.json")
df.head()<jupyter_output><empty_output><jupyter_text>Business MetadataNow we're going to fetch some info about the businesses themselves. First, we'll create a variable storing all of the unique business IDs in our sample. Next, we'll loop through the yelp_academic_dataset_business.json file and get info about the businesses whose IDs are in our sample.The categories and attributes fields could be useful. They provide a lot of information about different properties of the businesses, from accessibility to ambience to type of business. categories is provided as a single string, so we're going to split that into individual categories. attributes has fewer total values, but the results are dictionaries. Some are True/False (e.g., 'Outdoor Seating') and some are more varied values (e.g., 'hours'). One of the attribute dictionaries, 'Ambience', has key/value pairs indicating whether a business is touristy, 'hipster', romantic, divey, intimate, trendy, upscale, classy, and/or casual (each True/False). We'll treat each of those as a separate attribute because we're going to use them in our analyses below. If you are interested in some of the other attributes, you may need to modify the code below to handle them how 'Ambience' is handled.<jupyter_code>businesses_in_sample = df.business_id.unique()
len(businesses_in_sample)<jupyter_output><empty_output><jupyter_text>Let's take a look at the categories and attributes fields. The code below prints the results for the first review.<jupyter_code>with open("yelp_dataset/yelp_academic_dataset_business.json", "r", encoding="utf-8") as reader:
    for line in reader:
        line = json.loads(line.strip())
        print(line["categories"])
        print()
        print(line["attributes"])
        break
biz_categories_dict = defaultdict(lambda: {})
biz_attributes_dict = defaultdict(lambda: {})

with open("yelp_dataset/yelp_academic_dataset_business.json", "r", encoding="utf-8") as reader:
    for line in reader:
        line = json.loads(line.strip())
        biz_id = line["business_id"]
        if biz_id in businesses_in_sample:
            categories = line["categories"]
            attributes = line["attributes"]
            
            # check if "categories" is empty/None
            if categories:
                categories = categories.lower().split(",")
                categories = [cat.strip() for cat in categories]
                biz_categories_dict[biz_id] = set(categories)
            
            # check if "attributes" is empty/None
            if attributes:
                for key, value in attributes.items():
                    if key == "Ambience":
                        amb_categories = eval(value)
                        if type(amb_categories) == dict:
                            for cat, cat_val in amb_categories.items():
                                biz_attributes_dict[cat][biz_id] = cat_val
                    else:
                        biz_attributes_dict[key][biz_id] = value<jupyter_output><empty_output><jupyter_text>Let's put all of the categories from categories together, count their frequencies (using `Counter`), and take a look at the most frequent.<jupyter_code>all_categories = []
for cats in biz_categories_dict.values():
    all_categories += cats
len(set(all_categories))
c = Counter(all_categories)
c = sorted(list(c.items()), key=lambda x: x[1], reverse=True)
c[:200]<jupyter_output><empty_output><jupyter_text>We can use biz_categories_dict and each review's business_id field to check whether each category was used to describe the business.We can do the same for attributes.<jupyter_code>for attribute in biz_attributes_dict.keys():
    print(attribute, len(biz_attributes_dict[attribute]))<jupyter_output><empty_output><jupyter_text>Let's add a few variables (columns) to our dataframe reflecting different properties of the businesses. Note that with RestaurantsPriceRange2, we have to change the datatype from `str` to `float`.<jupyter_code>df["RestaurantsPriceRange2"] = df.business_id.apply(lambda x: biz_attributes_dict["RestaurantsPriceRange2"].get(x, np.nan))
df = df.astype({"RestaurantsPriceRange2": "float"})
df.RestaurantsPriceRange2.unique()
df["BYOBCorkage"] = df.business_id.apply(lambda x: biz_attributes_dict["BYOBCorkage"].get(x, np.nan))
df.BYOBCorkage.unique()
d = {"'no'": 0, "'yes_free'": 1, "'yes_corkage'": 1, 'None': np.nan, "u'no'": 0}
df.BYOBCorkage = df.BYOBCorkage.apply(lambda x: d.get(x, np.nan))
Counter(df.BYOBCorkage.dropna())
# From 'Ambience' in the raw data
df["classy"] = df.business_id.apply(lambda x: biz_attributes_dict["classy"].get(x, np.nan))
df.head()
df = df[df.business_id.isin(biz_categories_dict)]
df.shape
df["karaoke"] = df.business_id.apply(lambda x: "karaoke" in biz_categories_dict[x])
df["breweries"] = df.business_id.apply(lambda x: "breweries" in biz_categories_dict[x])
df["restaurant"] = df.business_id.apply(lambda x: "restaurant" in biz_categories_dict[x])
df["coffee"] = df.business_id.apply(lambda x: "coffee" in biz_categories_dict[x])
df["breakfast"] = df.business_id.apply(lambda x: "breakfast & brunch" in biz_categories_dict[x])<jupyter_output><empty_output><jupyter_text>Sentiment with VADERTo expand our toolkit a bit more in general and increase the range of variables we can explore in this notebook in particular, we are going to use VADER to analyze sentiment. You can check out the original paper here: [VADER: A Parsimonious Rule-based Model for  Sentiment Analysis of Social Media Text](http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf). You can read more about it on the GitHub page [here](https://github.com/cjhutto/vaderSentiment).VADER provides positive, neutral, and negative sentiment scores as well as a 'compound' score. According to the [GitHub page](https://github.com/cjhutto/vaderSentiment):> The compound score is computed by summing the valence scores of each word in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive). This is the most useful metric if you want a single unidimensional measure of sentiment for a given sentence. Calling it a 'normalized, weighted composite score' is accurate.We'll focus on the compound score in our analyses. Let's print a couple of example reviews, the scores from VADER, and the number of stars the reviewer assigned.<jupyter_code>analyzer = SentimentIntensityAnalyzer()
for idx, row in df.sample(2).iterrows():
    print(f"Review {idx}\n--")
    print(row.text)
    vs = analyzer.polarity_scores(row.text)
    print(vs)
    print(f"Stars: {row.stars}")
    print()<jupyter_output><empty_output><jupyter_text>...and let's go ahead and compute the scores for each review in our sample.<jupyter_code>df[["neg", "neu", "pos", "compound"]] = df.preprocessed.apply(lambda x: pd.Series(analyzer.polarity_scores(x)))<jupyter_output><empty_output><jupyter_text>We'll also a variable for the day of the week that could be useful for focusing on reviews of particular types of events (e.g., dinner on Friday vs. Sunday brunch).<jupyter_code>df["date"] = pd.to_datetime(df["date"])
df["day_name"] = df["date"].dt.day_name()
df.head()
df.to_json("yelp_sample.json")<jupyter_output><empty_output><jupyter_text>Descriptive Statistics Pandas dataframes have a `describe` method that provides basic descriptive statistics. You can also compute them directly (e.g., by calling `.mean()` or `.median()`) or by using a library like NumPy directly.<jupyter_code>df[["stars", "neg", "neu", "pos", "compound"]].describe()
print(f"Mean: {df.stars.mean():.2f}")
print(f"Median: {df.stars.median():.2f}")
print(f"Min: {df.stars.min():.2f}")
print(f"Max: {df.stars.max():.2f}")
print(f"Mean: {df.stars.std():.5f}")
print()

print(f"Mean: {np.mean(df.stars):.2f}")
print(f"Mean: {np.median(df.stars):.2f}")
print(f"Min: {np.min(df.stars):.2f}")
print(f"Max: {np.max(df.stars):.2f}")
print(f"Standard deviation: {np.std(df.stars):.5f}") # default: formula for population standard deviation
print(f"Standard deviation: {np.std(df.stars, ddof=1):.5f}") # use N-1 for sample standard deviation<jupyter_output><empty_output><jupyter_text>Correlation You can also calculate correlation coefficients directly using the [`.corr()` method](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html) with your dataframe. By default, this will produce a correlation matrix using Pearson's *r*. You can add the "spearman" argument for Spearman's rho or "kendall" for the Kendall rank correlation coefficient.<jupyter_code>df[["stars", "neg", "neu", "pos", "compound", "RestaurantsPriceRange2"]].corr()<jupyter_output><empty_output><jupyter_text>We can also save the whole correlation matrix as a variable (here, correlation_matrix) and plot it using Seaborn's `heatmap` function.<jupyter_code>correlation_matrix = df[["stars", "neg", "neu", "pos", "compound"]].corr()
sns.heatmap(correlation_matrix, cmap="coolwarm", annot=True, square=True)
plt.show()
df[["stars", "neg", "neu", "pos", "compound", "RestaurantsPriceRange2"]].corr("spearman")<jupyter_output><empty_output><jupyter_text>The `pingouin` library adds features like the [`rcorr` method](https://pingouin-stats.org/generated/pingouin.rcorr.html), which display correlation coefficients in the lower triangle and significance levels (represented by asterisks) in the upper triangle.<jupyter_code>df[["stars", "compound", "RestaurantsPriceRange2"]].rcorr("spearman")<jupyter_output><empty_output><jupyter_text>`scipy.stats` makes it easy to access the correlation coefficient and p-value for specific pairs of variables.<jupyter_code>pearsonr(df.stars, df.compound)
spearmanr(df.stars, df.compound)
r, pvalue = spearmanr(df.stars, df.compound)
r<jupyter_output><empty_output><jupyter_text>**Note: The p-value is being rounded down to zero. It's not precisely zero.**<jupyter_code>pvalue<jupyter_output><empty_output><jupyter_text>Reliability (Cronbach's alpha) If you want to know whether multiple three or more variables are measuring the same thing, you can use `pinguoin`'s implementation of Cronbach's alpha.<jupyter_code>cronbach_alpha(df[["stars", "pos", "compound"]])<jupyter_output><empty_output><jupyter_text>t-tests To test for differences in the means between two groups, we'll use the `statsmodels` implementation of independent samples *t*-tests, `ttest_ind`. The code below checks whether there are any reviews for businesses that are describes as breweries *and* as having karaoke. There are none in the sample. Next we'll compare the mean star-rating for each of these groups and then conduct a *t*-test. `ttest_ind` returns the *t*-statistic, the *p*-value, and the degrees of freedom. In this sample, there is a significant difference in the mean star-rating of breweries and businesses offering karaoke. On average, breweries get an extra 0.58 stars.<jupyter_code>df[(df.karaoke==True) & (df.breweries==True)].shape
print(df[df.karaoke==True].stars.mean())
print(df[df.breweries==True].stars.mean())
print(df[df.breweries==True].stars.mean()-df[df.karaoke==True].stars.mean())
t, pvalue, _ = ttest_ind(df[df.karaoke==True].stars, df[df.breweries==True].stars, usevar="unequal")
print(f"t = {t:.2f}, p = {pvalue:.2f}")<jupyter_output><empty_output><jupyter_text>How about restaurants that are described as classy or not?<jupyter_code>print(df[df.classy==True].stars.mean())
print(df[df.classy==False].stars.mean())
print(df[df.classy==True].stars.mean()-df[df.classy==False].stars.mean())
t, pvalue, _ = ttest_ind(df[df.classy==True].stars, df[df.classy==False].stars, usevar="unequal")
print(f"t = {t:.2f}, p = {pvalue:.2f}")<jupyter_output><empty_output><jupyter_text>On average, restaurants described as classy receive an extra 0.26 stars.  Linear Regression Now we will turn to linear regression using ordinary least squares (OLS). OLS tests the strength and significance of the relationship between the dependent variable and each independent variable in a way that can seem more flexible and interpretable than the previous tests. Whereas correlation coefficientas are "unit-free," a regression coefficient is interpreted as the unit change in the y variable for a unit change in the x variable. The intercept (or "constant") in the model is the conditional mean of the dependent variable.We are going to use the [`statsmodels` library](https://www.statsmodels.org/stable/index.html) to train our regression models and the [`stargazer` library](https://github.com/mwburke/stargazer) to format regression tables. compound is a continuous variable ranging between -1 and 1. RestaurantsPriceRange2 is an ordinal variable ranging from 1 to 4, though we can treat it as a continuous variable. breakfast and classy are stored as `bools` (True/False), and `statsmodels` will implicitly treat them as categorical variables where `False` == 0 and `True` == 1. This means that they will be treated as dummy variables, and the coefficient for each variable indicates the change in the outcome when the variable is `True` (or 1), as opposed to `False` (or 0).More specifically, we are going to use `statsmodels.formula.api`, which allows us to write our regression equations using R-like syntax: e.g., `y ~ x1 + x2`. See [here](https://www.statsmodels.org/devel/example_formulas.html).<jupyter_code># We are going to create a subset of the dataframe without missing data for the variables in our models
tmp = df[["compound", "RestaurantsPriceRange2", "breakfast", "classy"]].dropna()

formula = "compound ~ RestaurantsPriceRange2 + breakfast"
ols1 = smf.ols(formula, data=tmp).fit()

formula = "compound ~ RestaurantsPriceRange2 + breakfast + classy"
ols2 = smf.ols(formula, data=tmp).fit()
reg_table = Stargazer([ols1, ols2])
reg_table<jupyter_output><empty_output><jupyter_text>`stargazer` offers numerous methods for editing the appearance of the table, for example the order the independent variables appear in, the names of the independent variables, and the title of the table.<jupyter_code>reg_table = Stargazer([ols1, ols2])
reg_table.covariate_order(["RestaurantsPriceRange2", "breakfast[T.True]", "classy[T.True]", "Intercept"])
reg_table.rename_covariates({"RestaurantsPriceRange2": "Price", "breakfast[T.True]": "Breakfast", "classy[T.True]": "Classy"})
reg_table.title("Compound Sentiment Regressed on Price, Ambience, and Breakfast")
reg_table.show_model_numbers(False)
reg_table.significance_levels([0.05, 0.01, 0.001])
reg_table.custom_columns(["Model 1", "Model 2"], [1, 1])
reg_table<jupyter_output><empty_output><jupyter_text>**Interpretation:**For each tier increase in the measure of the restaurant's price range, we would expect the compound sentiment score of a review to increase by 0.064 on average according to Model 1 or by 0.042 on average according to Model 2, net of the other variables in each model.We have no evidence that serving breakfast and brunch is associated with a difference in the compound sentiment score.Businesses characterized as having a "classy" ambience score 0.076 higher on the compound score of sentiment on average, net of price and whether or not the business serves breakfast.Based on a comparison of the coefficients in these two models, it appears that classy accounts for some of the association between the outcome and the other independent variables. Specifically, the coefficient on price decreases from 0.064 to 0.042 when we control for classy. This could be interpreted in different ways.For example, classiness may be associated both with higher prices and with factors that lead to more positive reviews (i.e., higher compound scores). This would be an example of confounding. In contrast, it may be that higher prices make a restaurant seem more classy, which in turn leads to more positive reviews. This would be an example of partial mediation, and there are more formal ways to test for this. The larger point is that this is observational data and as much as we may like to offer causal interpretations, we can't be certain. Our job when using tools like linear regression is to come up with the best hypothesis we can, motivated by the best theory, and then use our models (and the results of significance tests) as components of an argument. We can also produce LaTeX code for the table directly:<jupyter_code>print(reg_table.render_latex())<jupyter_output><empty_output><jupyter_text>Logistic RegressionFinally, we will take a quick look at an example of logistic regression using `statsmodels`. Coefficients are reported in log-odds, but interpretations are otherwise similar: a unit change in x is associated with a change in the log-odds of y of some amount.For our dependent variable, we'll use the BYOBCorkage attribute, which we have dichotomized to no == 0 and yes == 1. What factors affect whether the restaurant will uncork your BYOB wine for you?<jupyter_code># We are going to create a subset of the dataframe without missing data for the variables in our models
tmp = df[["BYOBCorkage", "RestaurantsPriceRange2", "breakfast", "classy"]].dropna()

formula = "BYOBCorkage ~ RestaurantsPriceRange2 + breakfast"
logit1 = smf.logit(formula, data=tmp).fit()

formula = "BYOBCorkage ~ RestaurantsPriceRange2 + breakfast + classy"
logit2 = smf.logit(formula, data=tmp).fit()

logit_table = Stargazer([logit1, logit2])
logit_table.covariate_order(["RestaurantsPriceRange2", "breakfast[T.True]", "classy[T.True]", "Intercept"])
logit_table.rename_covariates({"RestaurantsPriceRange2": "Price", "breakfast[T.True]": "Breakfast", "classy[T.True]": "Classy"})
logit_table.title("BYOBCorkage Regressed on Price, Ambience, and Breakfast")
logit_table.show_model_numbers(False)
logit_table.significance_levels([0.05, 0.01, 0.001])
logit_table.custom_columns(["Model 1", "Model 2"], [1, 1])
logit_table<jupyter_output><empty_output><jupyter_text>**Interpretation:**According to the first model, an increase in the price range of the business is associated with a 0.223 decrease in the log-odds that the business will uncork your BYOB wine. When we control for whether the restaurant is considered classy (Model 2), we no longer observe a significant relationship between price and the outcome.  Exercises     For the exercises in this notebook, you will use the categories and attributes data to create new variables to answer a social research question. You will then use t-tests or correlation coefficients and finally a linear or logistic regression to test your hypothesis.     Exercise 1    Keeping in mind the variables you will create, what is your research question? You might find it helpful to pose your question in terms of the relationship between two variables. At least one of these should be a continuous variable (or a variable that can be treated as continuous). It may also be helpful to use one of the pre-existing continuous variables (e.g., star ratings or sentiment) as the outcome (dependent variable). Why did you choose these variables? Why might they be related? Are there variables that you may need to control for in order to model the relationship between your two variables? You might also consider using a subset of the data confined to a particular period of time. If needed, you can start with a sample larger than the 10,000-review sample taken at the start of the notebook. (You may also find it helpful to wait to fully finish this question until you've made sure you can create the variables you have in mind.) *Your answer here*     Exercise 2    2.1 Now create two variables. If you are using one of the pre-existing variables as your independent or (more likely) dependent variable, you might create the other variable and then create an additional variable to use as a control variable. The variables should fit your research question.     Remember, you can create variables using code like the following:     ```pythondf["classy"] = df.business_id.apply(lambda x: biz_attributes_dict["classy"].get(x, np.nan))``````pythondf["karaoke"] = df.business_id.apply(lambda x: "karaoke" in biz_categories_dict[x])```<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>2.2 Be sure to check the values of your new variables and recode them if needed. You can using code like df.NEW_VARIABLE.unique() to display the unique values.<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>Exercise 3    3.1 If your independent and dependent variables are both continuous (or can be treated as continuous, e.g. RestaurantsPriceRange2), are they correlated with one another? Use pearsonr or spearmanr from scipy.stats and report both the correlation coefficient and p-value. If only one of these variables is continuous, then select two groups from your categorical variable (e.g., the True and False groups) to compare with respect to the continuous variable. What are the group means of the continuous variable? Is one mean substantially higher? Is there a significant difference according to a t-test? Use ttest_ind. Report the group means, the difference, the results of the t-test. What do the results say about your research question? Do they fit with your expectations?<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>*Your answer here*     3.2 Now, assuming your dependent variable is continuous, train at least two linear regression models using OLS (and smf.ols). If your dependent variable is categorical, then make sure you are using a binary version of it (i.e., only two groups) and train at least two logistic regression models using smf.logit instead. The models should be trained using the same observations and should only differ in that the second (or third, etc.) model has one additional control variable. The addition of a control variable may reflect what you write in Exercise 1. You can train the models on the same subset of observations by using code like the code below to exclude any observations missing data for any of the variables you will use:    ```pythontmp = df[["compound", "RestaurantsPriceRange2", "breakfast", "classy"]].dropna()```<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output><jupyter_text>3.2 Now use stargazer to make a regression table presenting the results of your models.<jupyter_code># YOUR CODE HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09625968380832194, 'fraction_numerical': 0.007326392540400323, 'mean_word_length': 5.227461253001528, 'pattern_counts': {'":': 12, '<': 141, '<?xml version=': 0, '>': 145, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '417457', 'license': 'Unlicense', 'n_tokens_mistral': 8604, 'n_tokens_neox': 8196, 'n_words': 3229, 'path': 'notebooks/soc128d_notebook_12_modeling_ratings_and_sentiment_using_python.ipynb', 'repo_name': 'soc128d/soc128d.github.io', 'size': '28527'}