<jupyter_start><jupyter_text>Notebook to explore the Gather data sets providedGeneral information: - there are multiple files containing preprocessed data<jupyter_code>pwd
data_dir = '../data/wsup/tidy/'
ls $data_dir
import pandas_profiling
import pandas as pd<jupyter_output><empty_output><jupyter_text>Overview of the data files<jupyter_code>data_overview = pd.read_csv(data_dir + 'questionnaire_codebook.csv', index_col = 0)
data_overview<jupyter_output><empty_output><jupyter_text>Detailed data General background info on respondents and types of toilet - note this is not aggregated by plot - what is the unique id?<jupyter_code>data1_orig = pd.read_csv(data_dir + 'tidy_data_questionnaire_section_2.csv')
pandas_profiling.ProfileReport(data1_orig)
data1_orig.id.unique().shape, data1_orig.shape<jupyter_output><empty_output><jupyter_text>Find the unique index - is it id and respondent?It looks like type of toilet might also have a new row<jupyter_code>for key, grp in data1_orig.groupby(['id', 'respondent', 'type_of_toilet']):
    if grp.shape[0] > 1:
        print(grp[grp.columns.difference(['id', 'respondent'])].drop_duplicates().shape[0])<jupyter_output><empty_output><jupyter_text>We can see that within plot id and type of toilet that there can be multiple respondents but they always give the same responses<jupyter_code>for key, grp in data1_orig.groupby(['id','type_of_toilet']):
    if grp.shape[0] > 1:
        dups=grp[grp.columns.difference(['id','type_of_toilet', 'respondent', 'plot_months'])].drop_duplicates().shape[0]
        if dups > 1:
            print(grp)
data1_orig.groupby(['id', 'respondent','type_of_toilet']).size().max()
results = data1_orig.groupby('id')[['respondent', 'type_of_toilet']].agg(lambda x:len(set(x)))
results.columns= ['n_respondents', 'n_toilet_types']
data1_orig = data1_orig.set_index('id').join(results)
data1_orig['n_respondents'].value_counts()
data1_orig['n_toilet_types'].value_counts()
data1 = data1_orig.reset_index()
data1['respondent_ttype_id'] = data1.groupby('id')['respondent'].transform(lambda x:range(len(x)))
data1['respondent_ttype_id'].value_counts()
data1.groupby(['id', 'respondent_ttype_id']).size().max()
data1.set_index(['id', 'respondent_ttype_id'], drop=False, inplace=True)
data1.index.unique().shape, data1.shape<jupyter_output><empty_output><jupyter_text>More specific info on toilet use - unique by toilet user and plot ID?<jupyter_code>data3 = pd.read_csv(data_dir + 'tidy_data_questionnaire_section_3.csv')
pandas_profiling.ProfileReport(data3)<jupyter_output><empty_output><jupyter_text>It looks like waste disposal should be in data4, on the toilet type<jupyter_code>temp = data3[['id', 'solid_waste_disp']].drop_duplicates()
temp.shape
data3.drop('solid_waste_disp', 1, inplace=True)<jupyter_output><empty_output><jupyter_text>Unique index check<jupyter_code>data3.set_index(['id', 'toilet_user'], inplace=True)
data3.index.unique().shape, data3.shape<jupyter_output><empty_output><jupyter_text>Data on toilets that is aggregated by plot<jupyter_code>data4 = pd.read_csv(data_dir + 'tidy_data_questionnaire_section_4.csv')
pandas_profiling.ProfileReport(data4)<jupyter_output><empty_output><jupyter_text>Set the index<jupyter_code>data4 = data4.set_index('id')
data4 = data4.join(temp.set_index('id'))
data4.index.unique().shape, data4.shape
data4.head()<jupyter_output><empty_output><jupyter_text>Toilets<jupyter_code>data5 = pd.read_csv(data_dir + 'tidy_data_questionnaire_section_5.csv')
pandas_profiling.ProfileReport(data5)<jupyter_output><empty_output><jupyter_text>Questions  (1) How consistent are responses across different people from the same plot<jupyter_code>data1.columns
data1_dups = data1[data1.n_responders>1]
for f in ['no_of_hhs',  'no_of_ppl', 'no_of_toilets']:
    temp = data1_dups.groupby('id')[f].apply(lambda x:list(x))
    temprange = temp.map(lambda x:max(x) - min(x))
    print(f, temprange.max())<jupyter_output>/usr/local/anaconda3/envs/py3/lib/python3.6/site-packages/ipykernel_launcher.py:2: FutureWarning: 'id' is both a column name and an index level.
Defaulting to column but this will raise an ambiguity error in a future version<jupyter_text>There is variance in the no of toilets<jupyter_code>data1[data1.id.isin(temprange[temprange==10].index)]<jupyter_output><empty_output><jupyter_text>Ah... the data set is also split based on type of toilet  (2) What is the average number of toilets per plot<jupyter_code>data1<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.128268156424581, 'fraction_numerical': 0.015865921787709496, 'mean_word_length': 8.079107505070994, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '251943', 'license': 'MIT', 'n_tokens_mistral': 1652, 'n_tokens_neox': 1622, 'n_words': 298, 'path': 'helen_notebooks/EDA for the Gather dataset.ipynb', 'repo_name': 'helenlram/data_dive_risk', 'size': '4475'}