<jupyter_start><jupyter_text>__European Social Survey__This is a biannual survey to measure the attitudes, beliefs and behavior patterns of the various populations in Europe. Using selected questions from the 2012 and 2014 editions, address the following questions. Here is the data file, and here is the codebook with information about the variable coding and content.Note: the field idno should not be taken as a unique identifier.1. Did people become less trusting from 2012 to 2014?2. Did people become happier from 2012 to 2014?3. Pick three or four of the countries in the sample and compare how often people met socially in 2014. Are there differences, and if so, which countries stand out?4. Pick three or four of the countries in the sample and compare how often people took part in social activities, relative to others their age, in 2014. Are there differences, and if so, which countries stand out?__Ruff Figural Fluency Test__ This test provides information about nonverbal capacity for initiation, planning, and divergent reasoning. You can find the data here. The Casenr is a unique identifier.Of particular use for comparing data points between measurements will be pivot() from Pandas.1. Is there a significant difference in the number of unique designs drawn by the same participants from Measurement 2 to Measurement 3?2. Is there a significant difference in the number of perseverative errors drawn by the same participants from Measurement 2 to Measurement 3?<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from scipy import stats
europe = pd.read_csv("https://tf-assets-prod.s3.amazonaws.com/tf-curric/data-science/ess_combined_2012_2014.csv")
europe.head()<jupyter_output><empty_output><jupyter_text>1. Did people become less trusting from 2012 to 2014?<jupyter_code>twenty12 = europe[europe['year'] == 6]

twenty14 = europe[europe['year'] == 7]

print(stats.shapiro(twenty12['ppltrst']))
print(stats.shapiro(twenty14['ppltrst']))

print(stats.describe(twenty12['ppltrst']))
print(stats.describe(twenty14['ppltrst']))

plt.hist(twenty12['ppltrst'], alpha = .5)
plt.hist(twenty14['ppltrst'], alpha = .5)
plt.show()<jupyter_output>(0.9520939588546753, 1.4426241793085802e-34)
(0.9580563902854919, 9.222525335052542e-33)
DescribeResult(nobs=4071, minmax=(0, 10), mean=5.578481945467944, variance=4.893040545067073, skewness=-0.5431086926347095, kurtosis=-0.1210956960964813)
DescribeResult(nobs=4076, minmax=(0, 10), mean=5.5713935230618254, variance=4.927171652708959, skewness=-0.4601859476512641, kurtosis=-0.25006786100787215)<jupyter_text>Shapiro tests are very sensitive to large samples. All other indications suggest this is a normal distribution<jupyter_code>europe.groupby(['year']).agg(['count'])<jupyter_output><empty_output><jupyter_text>Distribution doesn't look normal<jupyter_code>stats.ttest_ind(twenty12['ppltrst'], twenty14['ppltrst'])<jupyter_output><empty_output><jupyter_text>Failed to reject the null, there is no statistically significant difference  2. Did people become happier from 2012 to 2014?<jupyter_code>print(stats.shapiro(twenty12['happy']))
print(stats.shapiro(twenty14['happy']))

print(stats.describe(twenty12['happy']))
print(stats.describe(twenty14['happy']))

plt.hist(twenty12['happy'], alpha = .5)
plt.hist(twenty14['happy'], alpha = .5)
plt.show()<jupyter_output>(0.8943395018577576, 0.0)
(0.9060227870941162, 1.2611686178923354e-44)
DescribeResult(nobs=4071, minmax=(0, 10), mean=7.731515598133137, variance=3.0819524689826823, skewness=-1.1505518232802756, kurtosis=1.9055890346061979)
DescribeResult(nobs=4076, minmax=(0, 10), mean=7.6854759568204125, variance=2.840436853164115, skewness=-1.0296465664009635, kurtosis=1.5218046596567287)<jupyter_text>very skewed, not a normal distribution<jupyter_code>stats.kruskal(
    twenty12['happy'],
    twenty14['happy'])<jupyter_output><empty_output><jupyter_text>Very close p value, however looking at the two histograms I have decided to reject the null.  3. Pick three or four of the countries in the sample and compare how often people met socially in 2014. Are there differences, and if so, which countries stand out?<jupyter_code>europe['cntry'].unique()
selected = ['CH', 'CZ', 'DE', 'ES']

countries = twenty14[twenty14['cntry'].isin(selected)]
countries['sclmeet'].hist(by = countries['cntry'])
stats.kruskal(
    countries.loc[countries['cntry'] == 'CZ',['sclmeet']],
    countries.loc[countries['cntry'] == 'DE',['sclmeet']],
    countries.loc[countries['cntry'] == 'ES',['sclmeet']],
    countries.loc[countries['cntry'] == 'CH',['sclmeet']])<jupyter_output><empty_output><jupyter_text>Not all medians are even, time to boxplot to see the odd man out<jupyter_code>countries.boxplot('sclmeet', by = 'cntry', figsize = (15,15))<jupyter_output><empty_output><jupyter_text>4. Pick three or four of the countries in the sample and compare how often people took part in social activities, relative to others their age, in 2014. Are there differences, and if so, which countries stand out?<jupyter_code>countries['sclact'].hist(by = countries['cntry'])<jupyter_output><empty_output><jupyter_text>These look to be more normal distributions based on the eyeball test<jupyter_code>stats.f_oneway(
    countries.loc[countries['cntry'] == 'CZ',['sclact']],
    countries.loc[countries['cntry'] == 'DE',['sclact']],
    countries.loc[countries['cntry'] == 'ES',['sclact']],
    countries.loc[countries['cntry'] == 'CH',['sclact']])<jupyter_output><empty_output><jupyter_text>Based on these results there doesn't seem to be any significant differences between the groups<jupyter_code>from statsmodels.stats.multicomp import pairwise_tukeyhsd

tukey = pairwise_tukeyhsd(endog = countries['sclact'],      # Data
                          groups = countries['cntry'],   # Groups
                          alpha=0.05) 


tukey.summary()<jupyter_output><empty_output><jupyter_text>Ruff Figural Fluency Test  1. Is there a significant difference in the number of unique designs drawn by the same participants from Measurement 2 to Measurement 3?<jupyter_code>rf = pd.read_csv("https://tf-assets-prod.s3.amazonaws.com/tf-curric/data-science/rfft.csv")
rf.head()
rfmeasurements = rf.pivot(index='Casenr', columns='Measurement', values='Unique')
rfmeasurements.head()
plt.hist(rfmeasurements['Measurement 3 (2008-2012)'] - rfmeasurements['Measurement 2 (2006-2008)'])
print(stats.describe(rfmeasurements['Measurement 3 (2008-2012)'] - rfmeasurements['Measurement 2 (2006-2008)']))<jupyter_output>DescribeResult(nobs=2515, minmax=(-75.5, 68.5), mean=4.015506958250497, variance=342.2368318331855, skewness=-0.15837265462838976, kurtosis=0.7823714353134701)<jupyter_text>Looks normally distributed to me, I'll use a t-test to check for differences<jupyter_code>stats.ttest_rel(rfmeasurements['Measurement 3 (2008-2012)'], rfmeasurements['Measurement 2 (2006-2008)'])<jupyter_output><empty_output><jupyter_text>There seem to be a difference between measurement two and three  2. Is there a significant difference in the number of perseverative errors drawn by the same participants from Measurement 2 to Measurement 3?<jupyter_code>rfpersev = rf.pivot(index='Casenr', columns='Measurement', values='Perseverative')
rfpersev.head()
plt.hist(rfpersev['Measurement 3 (2008-2012)'], alpha = .5)
plt.hist(rfpersev['Measurement 2 (2006-2008)'], alpha = .5)<jupyter_output><empty_output><jupyter_text>Looks fairly normal and the distributions are similar enough<jupyter_code>stats.wilcoxon(rfpersev['Measurement 3 (2008-2012)'], rfpersev['Measurement 2 (2006-2008)'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11836787902489897, 'fraction_numerical': 0.09516360318081085, 'mean_word_length': 6.860655737704918, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '309385', 'license': 'MIT', 'n_tokens_mistral': 3003, 'n_tokens_neox': 2487, 'n_words': 662, 'path': 'Beyond_AB_Tests.ipynb', 'repo_name': 'sgf-afk/Class_Assignments', 'size': '7671'}