<jupyter_start><jupyter_text>A Series is a one-dimensional array-like object containing * a sequence of values (ofsimilar types to NumPy types) * and an associated array of data labels, called its index<jupyter_code>obj = pd.Series([4, 7, -5, 3])
print(obj)
print(obj.values)
print(obj.index)<jupyter_output><empty_output><jupyter_text>What is a dataframe?The primary objects in pandas are * the DataFrame, a tabular, column-oriented data structure with both row and column labels,* and a Series, a one-dimensional labeled array object.<jupyter_code>data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
        'year': [2000, 2001, 2002, 2001, 2002, 2003],
        'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
data = pd.DataFrame(data)
data.head()
data.columns
data['state']
data.state
data.iloc[0]
data = pd.DataFrame(data)
data.drop(['state'], axis = 1, inplace=True)
data.head()
data.describe()
data = pd.DataFrame(np.arange(16).reshape((4, 4)),
                    index=['Ohio', 'Colorado', 'Utah', 'New York'],
                    columns=['one', 'two', 'three', 'four'])
data.head()

data['two']
data[['three', 'one']]
data[:2]
data<5 # filtering
data[data<5] = 0 # filtering and setting new value using boolean operation
data
data.loc['Colorado', ['two', 'three']]
data.iloc[2, [3, 0, 1]]
data.iloc[[1, 2], [3, 0, 1]]
data.iloc[:, :3][data.three > 5]
data.sum()
data.sum(axis='columns')<jupyter_output><empty_output><jupyter_text>Some important pandas methods ![alt text](img/pandas_functions.png "Title")  Hypothesis testing The basis of hypothesis testing has two attributes: (a) Null Hypothesis and (b) Alternative Hypothesis. $H_0$: Null hypothesis (no change, no difference);   $H_1$: Research hypothesis (investigator's belief); $\alpha$ =0.05 Upper-tailed, Lower-tailed, Two-tailed Tests The research or alternative hypothesis can take one of three forms. An investigator might believe that the parameter has increased, decreased or changed. For example, an investigator might hypothesize:  H1: μ > μ 0 , where μ0 is the comparator or null value (e.g., μ0 =males weight 10 years ago) and an increase is hypothesized - this type of test is called an upper-tailed test;H1: μ < μ0 , where a decrease is hypothesized and this is called a lower-tailed test; orH1: μ ≠ μ 0, where a difference is hypothesized and this is called a two-tailed test.  The exact form of the research hypothesis depends on the investigator's belief about the parameter of interest and whether it has possibly increased, decreased or is different from the null value. The research hypothesis is set up by the investigator before any data are collected.   If $p>\alpha$, we don't reject the null hypothesis.  If $p< \alpha$, we reject the null hypothesis and accept the alternative hypothesis  Types of errors (Error I and Error II) First Case: The person is innocent and the judge identifies the person as innocent Second Case: The person is innocent and the judge identifies the person as guilty Third Case: The person is guilty and the judge identifies the person as innocent Fourth Case: The person is guilty and the judge identifies the person as guilty  Two Sample Z Test  We perform a Two Sample Z test when we want to compare the mean of two samples.  Example: The diabetes database Do males have significantly higher mean Boday Mass Index (BMI) than females? Null Hypothesis: $\mu_m = \mu_f$ Alternative Hypothesis: $\mu_m \neq \mu_f$<jupyter_code>dataName = 'nhanes_2015_2016.csv'
df = pd.read_csv(dataName)
df.head(15)

# Print column names
df.columns
# Check for missing or Nan values
print(df["RIAGENDR"].isnull().values.any())
print(df["BMXBMI"].isnull().values.any())
BMI_NAN = df["BMXBMI"].isnull()
print(BMI_NAN)
df.index[BMI_NAN==True]
df.dropna(subset=["BMXBMI", "RIAGENDR"])
print(df["BMXBMI"].isnull().values.any())
df.dropna(subset=["BMXBMI", "RIAGENDR"], inplace = True)
print(df["BMXBMI"].isnull().values.any())
males = df[df["RIAGENDR"] == 1]
males.head(20)
n1 = len(males)
mu1 = males["BMXBMI"].mean()
sd1 = males["BMXBMI"].std()

(n1, mu1, sd1)
females = df[df["RIAGENDR"] == 2]
females.head(20)
n2 = len(females)
mu2 = females["BMXBMI"].mean()
sd2 = females["BMXBMI"].std()

(n2, mu2, sd2)
plt.figure(figsize=(7,4))
plt.title("Female BMI histogram",fontsize=16)
plt.hist(females["BMXBMI"].dropna(),edgecolor='k',color='pink',bins=50)
plt.show()

plt.figure(figsize=(7,4))
plt.title("Male BMI histogram",fontsize=16)
plt.hist(male["BMXBMI"].dropna(),edgecolor='k',color='blue',bins=50)
plt.show()
sm.stats.ztest(females["BMXBMI"], males["BMXBMI"],alternative='two-sided')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13896693321806786, 'fraction_numerical': 0.025070239896261077, 'mean_word_length': 5.305177111716621, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '373259', 'license': 'MIT', 'n_tokens_mistral': 1632, 'n_tokens_neox': 1483, 'n_words': 524, 'path': 'notebooks/3_Stats_exercises_pandas.ipynb', 'repo_name': 'abdelrahman-ayad/MiCM-StatsPython-F21', 'size': '4627'}