<jupyter_start><jupyter_text>1: Jeopardy Questions<jupyter_code>import pandas as pd
jeopardy = pd.read_csv('jeopardy.csv')
jeopardy.head()
jeopardy.columns
jeopardy.columns = ['Show Number', 'Air Date', 'Round', 'Category', 'Value', 'Question', 'Answer']<jupyter_output><empty_output><jupyter_text>2: Normalizing Text<jupyter_code>import re
def norm(string):
    string = string.lower()
    string = re.sub("[^A-Za-z0-9\s]", "", string)
    return(string)
jeopardy["clean_question"] = jeopardy["Question"].apply(norm)
jeopardy["clean_answer"] = jeopardy["Answer"].apply(norm)<jupyter_output><empty_output><jupyter_text>3: Normalizing Columns<jupyter_code>def norm_value(string):
    string = re.sub("[^A-Za-z0-9\s]", "", string)
    try:
        string =int(string)
    except Exception:
        string = 0
    return(string)   
jeopardy["clean_value"] = jeopardy["Value"].apply(norm_value)
jeopardy["Air Date"] = pd.to_datetime(jeopardy["Air Date"])
jeopardy.head()
jeopardy.dtypes<jupyter_output><empty_output><jupyter_text>4: Answers In Questions<jupyter_code># How often the answer is deducible from the question.
def count_matches(row):
    split_answer = row['clean_answer'].split(' ')
    split_question = row['clean_question'].split(' ')
    match_count = 0
    if 'the' in split_answer: split_answer.remove("the")
    if len(split_answer)==0: return 0
    for item in split_answer:
        if item in split_answer:
            match_count +=1
    return match_count / len(split_answer)
jeopardy["answer_in_question"] = jeopardy.apply(count_matches, axis=1)

jeopardy["answer_in_question"].mean()<jupyter_output><empty_output><jupyter_text>How often the answer is deducible from the questionThe answer only appears in the question about 6% of the time. This isn't a huge number, and means that we probably can't just hope that hearing a question will enable us to figure out the answer. We'll probably have to study.  5: Recycled Questions<jupyter_code>question_overlap = []
terms_used = set()
for i, row in jeopardy.iterrows():
        split_question = row["clean_question"].split(" ")
        split_question = [q for q in split_question if len(q) > 5]
        match_count = 0
        for word in split_question:
            if word in terms_used:
                match_count += 1
        for word in split_question:
            terms_used.add(word)
        if len(split_question) > 0:
            match_count /= len(split_question)
        question_overlap.append(match_count)
jeopardy["question_overlap"] = question_overlap

jeopardy["question_overlap"].mean()<jupyter_output><empty_output><jupyter_text>How often new questions are repeats of older questions.There is about 70% overlap between terms in new questions and terms in old questions. This only looks at a small set of questions, and it doesn't look at phrases, it looks at single terms. This makes it relatively insignificant, but it does mean that it's worth looking more into the recycling of questions.  6: Low Value Vs High Value Questions<jupyter_code>def determine_value(row):
    value = 0
    if row["clean_value"] > 800:
        value = 1
    return value

jeopardy["high_value"] = jeopardy.apply(determine_value, axis=1)
def count_usage(term):
    low_count = 0
    high_count = 0
    for i, row in jeopardy.iterrows():
        if term in row["clean_question"].split(" "):
            if row["high_value"] == 1:
                high_count += 1
            else:
                low_count += 1
    return high_count, low_count

comparison_terms = list(terms_used)[:5]
observed_expected = []
for term in comparison_terms:
    observed_expected.append(count_usage(term))

observed_expected<jupyter_output><empty_output><jupyter_text>7: Applying The Chi-Squared Test<jupyter_code>from scipy.stats import chisquare
import numpy as np

high_value_count = jeopardy[jeopardy["high_value"] == 1].shape[0]
low_value_count = jeopardy[jeopardy["high_value"] == 0].shape[0]

chi_squared = []
for obs in observed_expected:
    total = sum(obs)
    total_prop = total / jeopardy.shape[0]
    high_value_exp = total_prop * high_value_count
    low_value_exp = total_prop * low_value_count
    
    observed = np.array([obs[0], obs[1]])
    expected = np.array([high_value_exp, low_value_exp])
    chi_squared.append(chisquare(observed, expected))

chi_squared<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11044776119402985, 'fraction_numerical': 0.009873708381171067, 'mean_word_length': 4.30572472594397, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '33058', 'license': 'MIT', 'n_tokens_mistral': 1481, 'n_tokens_neox': 1373, 'n_words': 357, 'path': 'Winning Jeopardy/Basics.ipynb', 'repo_name': 'vipmunot/Data-Science-Projects', 'size': '4355'}