<jupyter_start><jupyter_text>Python practical: Data analysis and visualisationIn this practical, you will practice using pandas, numpy, matplotlib and seaborn for data analysis and visualisation.Please read the notebook and complete the code for each question.<jupyter_code># Import libraries
import numpy as np
import scipy as sp
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats<jupyter_output><empty_output><jupyter_text>Q1: Analysis and visualise academic salariesThis an academic salary dataset. It comes from http://vincentarelbundock.github.io/Rdatasets Q1.1: A quick check<jupyter_code># Load dataset
df = pd.read_csv('Salaries.csv', index_col=0)
# Display the shape of the data

# Display the first 10 rows of the data

# Display the last 10 rows of the data

# Display the data type for each column

# Display the summary statistics for all numeric columns

# Display the summary statistics just for the salary column<jupyter_output><empty_output><jupyter_text>Q1.2 Grouping of dataWe will first show you how to display the salary statistics for the male and female groups. Then we will ask you to complete the code for analysing other data groups.<jupyter_code># Display the salary statistics for male and female groups
df_sex = df.groupby('sex')
df_sex['salary'].describe()
# Display the salary statistics for different academic ranks

# Display the years since PhD for different academic ranks

# Display the mean of all numeric columns for different academic ranks

# Group by both sex and academic ranks and display the mean statistics<jupyter_output><empty_output><jupyter_text>Q1.3 Data selection<jupyter_code># Select data with salary > 100K and display the first 5 rows

# Select data for female assistant professors and display the first 5 rows

# Display the mean salary for discipline A

# Display the region from index 10 to index 20, at the sex column

# Display the region from row 10 to row 20, at the 3rd and 4th columns<jupyter_output><empty_output><jupyter_text>Q1.4 Sorting<jupyter_code># Sort the data by salries in descending order and display the head
df_sorted = df.sort_values(by='salary', ascending=False)
df_sorted.head()
# Sort the data by yrs.service in ascending order and display the head

# Sort the data by yrs.service in ascending order and salary in descending order, display the head<jupyter_output><empty_output><jupyter_text>Q1.5 Data visualisation<jupyter_code># Display the histogram for salary using matplotlib
plt.hist(df['salary'])
# Display the histogram for salary using seaborn
sns.distplot(df['salary'])
# Bar plot of salary against academic ranks using seaborn

# Bar plot os salary against academic ranks, using different hues for males and females

# Box plot os salary against academic ranks, using different hues for males and females

# Scatter plot of salary against yrs.since.phd, colouring by sex and styling by rank

# Violin plot of salary against sex

# Pair plot to show joint relationships and histograms<jupyter_output><empty_output><jupyter_text>Q1.6 Student T-test<jupyter_code># Perform t-test between female salaries and male salaries<jupyter_output><empty_output><jupyter_text>Q1.7 A larger and more complex datasetThis is the Kaggle San Francisco city employee salary dataset.https://www.kaggle.com/kaggle/sf-salaries<jupyter_code># Load the dataset
df = pd.read_csv('Salaries_SF.csv')
# Display the shape of the data

# Display the first 10 rows of the data

# Display the statistics of the TotalPayBenefits for FIREFIGHTER

# Violin plot of TotalPayBenefits against Year

# Scatter plot of OtherPay against 'BasePay', only for subjects with positive OtherPay and with numeric data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05537720706260032, 'fraction_numerical': 0.009898341359015517, 'mean_word_length': 5.571177504393673, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '126085', 'license': 'Apache-2.0', 'n_tokens_mistral': 1088, 'n_tokens_neox': 1044, 'n_words': 472, 'path': 'python_practical_3/python_practical_3.ipynb', 'repo_name': 'baiwenjia/clinical_data_management', 'size': '3738'}