<jupyter_start><jupyter_text>Regiment  Introduction:Special thanks to: http://chrisalbon.com/ for sharing the dataset and materials. Step 1. Import the necessary libraries<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Step 2. Create the DataFrame with the following values:<jupyter_code>raw_data = {'regiment': ['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks', 'Dragoons', 'Dragoons', 'Dragoons', 'Dragoons', 'Scouts', 'Scouts', 'Scouts', 'Scouts'], 
        'company': ['1st', '1st', '2nd', '2nd', '1st', '1st', '2nd', '2nd','1st', '1st', '2nd', '2nd'], 
        'name': ['Miller', 'Jacobson', 'Ali', 'Milner', 'Cooze', 'Jacon', 'Ryaner', 'Sone', 'Sloan', 'Piger', 'Riani', 'Ali'], 
        'preTestScore': [4, 24, 31, 2, 3, 4, 24, 31, 2, 3, 2, 3],
        'postTestScore': [25, 94, 57, 62, 70, 25, 94, 57, 62, 70, 62, 70]}<jupyter_output><empty_output><jupyter_text>Step 3. Assign it to a variable called regiment. Don't forget to name each column<jupyter_code>regiment = pd.DataFrame(raw_data, columns = raw_data.keys())
regiment.head(10)<jupyter_output><empty_output><jupyter_text>Step 4. What is the mean preTestScore from the regiment Nighthawks?<jupyter_code># show all rows with regiment is Nighthawks
regiment['regiment'] == 'Nighthawks'
regiment[regiment['regiment'] == 'Nighthawks']
regiment[regiment['regiment'] == 'Nighthawks'].mean()
# or beautiful format,
regiment[regiment.regiment == 'Nighthawks'].groupby('regiment').mean()<jupyter_output><empty_output><jupyter_text>Step 5. Present general statistics by company<jupyter_code>regiment.groupby('company').describe()<jupyter_output><empty_output><jupyter_text>Step 6. What is the mean of each company's preTestScore?<jupyter_code>regiment.groupby('company').preTestScore.mean()<jupyter_output><empty_output><jupyter_text>Step 7. Present the mean preTestScores grouped by regiment and company<jupyter_code>regiment.groupby(['regiment','company']).preTestScore.mean()<jupyter_output><empty_output><jupyter_text>Step 8. Present the mean preTestScores grouped by regiment and company without heirarchical indexing<jupyter_code>regiment.groupby(['regiment','company']).preTestScore.mean().unstack()<jupyter_output><empty_output><jupyter_text>Step 9. Group the entire dataframe by regiment and company<jupyter_code>regiment.groupby(['regiment','company']).mean()<jupyter_output><empty_output><jupyter_text>Step 10. What is the number of observations in each regiment and company<jupyter_code>regiment.groupby(['company', 'regiment']).size()<jupyter_output><empty_output><jupyter_text>Step 11. Iterate over a group and print the name and the whole data from the regiment<jupyter_code>for name, group in regiment.groupby('regiment'):
    print(name)
    print(group)

df = regiment
df.head(12)
diff_score = df.postTestScore - df.preTestScore
df['diff_score'] = diff_score
df.head()
## Option 1:
# df.loc[df['preTestScore'] >= 50, 'preresult'] = 'pass' 
# df.loc[df['preTestScore'] < 50, 'preresult'] = 'fail'
# df.loc[df['postTestScore'] >= 50, 'postresult'] = 'pass' 
# df.loc[df['postTestScore'] < 50, 'postresult'] = 'fail'

## or
df['preresult'] = df['preTestScore'].apply(lambda x: 'fail' if x <= 50  else 'pass') 
df['postresult'] = df['postTestScore'].apply(lambda x: 'fail' if x <= 50  else 'pass') 
df.head(12)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16176033303597978, 'fraction_numerical': 0.024977698483496878, 'mean_word_length': 7.285714285714286, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '640774', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1284, 'n_tokens_neox': 1174, 'n_words': 271, 'path': '03_Grouping/Regiment/Osbert_answer_20200430.ipynb', 'repo_name': 'osbertc/pandas_exercises', 'size': '3363'}