<jupyter_start><jupyter_text>Mean Median Mode![Mean_median_mode](Statistics/Meanmedianmode.PNG)<jupyter_code>data = np.array([4,5,1,2,7,2,6,9,3])
data
# Mean, Median, Mode Calculatoion Function
def mean_median_mode(data):
    # Calculate Mean
    dt_mean = np.mean(data) ; print ("Mean :",round(dt_mean,2))
    # Calculate Median                 
    dt_median = np.median(data) ; print ("Median :",dt_median)        
    # Calculate Mode                     
    dt_mode =  stats.mode(data); print ("Mode :",dt_mode[0][0])

mean_median_mode(data)<jupyter_output>Mean : 4.33
Median : 4.0
Mode : 2<jupyter_text>Variance, Standard Deviation,Range![Mean_median_mode](Statistics/Varinace_stdev.PNG)<jupyter_code>import numpy as np
import statistics as st
game_points = np.array([35,56,43,59,63,79,35,41,64,43,93,60,77,24,82])
game_points
def variance(data):
    # Calculate Variance
    dt_var = st.variance(data) ; print ("Sample variance:", round(dt_var,2))
    # Calculate Standard Deviation
    dt_std = st.stdev(data) ; print ("Sample std.dev:",round(dt_std,2))
    # Calculate Range
    dt_rng = np.max(data,axis=0) - np.min(data,axis=0) ; print ("Range:",dt_rng)

variance(game_points)<jupyter_output>Sample variance: 400
Sample std.dev: 20.0
Range: 69<jupyter_text>Percentile, IQR![IQR](Statistics/Quantile.PNG)<jupyter_code>print ("Quantiles:")
for val in [20,50,100]:
    dt_qntls = np.percentile(game_points,val) 
    print (str(val)+"%" ,dt_qntls)
                                
# Calculate IQR                           
q75, q25 = np.percentile(game_points, [75 ,25]); print ("Inter quartile range:",q75-q25 )<jupyter_output>Quantiles:
20% 39.800000000000004
50% 59.0
100% 93.0
Inter quartile range: 28.5<jupyter_text>Hypothesis testing![hypothesis](Statistics/Hypothesis testing.PNG)<jupyter_code>from scipy import stats

xbar = 990; mu0 = 1000; s = 12.5; n = 30
# Test Statistic
t_smple  = (xbar-mu0)/(s/np.sqrt(float(n))); print ("Test Statistic:",round(t_smple,2))
# Critical value from t-table
alpha = 0.05
t_alpha = stats.t.ppf(alpha,n-1); print ("Critical value from t-table:",round(t_alpha,3))          
#Lower tail p-value from t-table                        
p_val = stats.t.sf(np.abs(t_smple), n-1); print ("Lower tail p-value from t-table", p_val)<jupyter_output>Test Statistic: -4.38
Critical value from t-table: -1.699
Lower tail p-value from t-table 7.035025729010886e-05<jupyter_text>Normal Distribution![Normal Distribution](Statistics/Normalization.PNG)<jupyter_code>def z_score(xbar, mu0, s):
    z = (xbar-mu0)/s
    print("z_score is :",round(z,4))
    prob = stats.norm.cdf(z)
    print("Probability of zscore : ", prob);print ("Prob. to score more than 67 is ",round(prob*100,2),"%")

z_score(67,52,16.3)<jupyter_output>z_score is : 0.9202
Probability of zscore :  0.8212777324852483
Prob. to score more than 67 is  82.13 %<jupyter_text>Chi Square test![Chi Square test](Statistics/Chi-square-test.PNG)<jupyter_code>import pandas as pd
from scipy import stats
survey = pd.read_csv("survey.csv")
survey.head(5)
def Chi_square_test(Column1, Column2):
    # Tabulating 2 variables with row & column variables respectively
    survey_tab = pd.crosstab(Column1,Column2,margins=True)
    print(survey_tab)
    # Creating observed table for analysis
    observed = survey_tab.ix[0:4,0:3] 
#     print(observed)
    contg = stats.chi2_contingency(observed= observed)
#     print(contg)
    p_value = round(contg[1],3)
    print ("P-value is: ",p_value)
Chi_square_test(survey.Smoke, survey.Clap)<jupyter_output>Clap   Left  Neither  Right  All
Smoke                           
Heavy     2        2      7   11
Never    29       39    120  188
Occas     3        6     10   19
Regul     4        3     10   17
All      38       50    147  235
P-value is:  0.903<jupyter_text>Anova Model![Anova](Statistics/ANOVA.PNG)<jupyter_code>import pandas as pd
from scipy import stats
fetilizers = pd.read_csv("fetilizers.csv")
one_way_anova = stats.f_oneway(fetilizers["fertilizer1"], fetilizers["fertilizer2"], fetilizers["fertilizer3"])
print ("Statistic :", round(one_way_anova[0],2),", p-value :",round(one_way_anova[1],3))<jupyter_output>Statistic : 4.13 , p-value : 0.037<jupyter_text>Train and Test sample![train and test](Statistics/Trainandtest.PNG)<jupyter_code># Train & Test split
import pandas as pd      
from sklearn.model_selection import train_test_split              
                        
original_data = pd.read_csv("mtcars.csv")     

train_data,test_data = train_test_split(original_data,train_size = 0.7,random_state=42)
train_data.shape,test_data.shape<jupyter_output><empty_output><jupyter_text>Linear and Gradient Boosting  - In the following code, a comparison has been made between applying linear regression in a statistical way and gradient descent in a machine learning way on the same dataset:<jupyter_code>import numpy as np                        
import pandas as pd
train_data = pd.read_csv("mtcars.csv")
print(train_data.shape)
train_data.head(5)
X = np.array(train_data["hp"]); y = np.array(train_data["mpg"]) 
X = X.reshape(32,1); y = y.reshape(32,1)
from sklearn.linear_model import LogisticRegression
model = LinearRegression(fit_intercept=True)
model.fit(X,y)
print("Linear Regression Results")
print("Intercept", model.intercept_[0],"Coefficient", model.coef_[0])
def gradient_descent(x, y,learn_rate, conv_threshold,batch_size,max_iter):    
    converged = False
    iter = 0
    m = batch_size 
    t0 = np.random.random(x.shape[1])
    t1 = np.random.random(x.shape[1])

    MSE = (sum([(t0 + t1*x[i] - y[i])**2 for i in range(m)])/ m)    

    while not converged:        
        grad0 = 1.0/m * sum([(t0 + t1*x[i] - y[i]) for i in range(m)]) 
        grad1 = 1.0/m * sum([(t0 + t1*x[i] - y[i])*x[i] for i in range(m)])

        temp0 = t0 - learn_rate * grad0
        temp1 = t1 - learn_rate * grad1
    
        t0 = temp0
        t1 = temp1

        MSE_New = (sum( [ (t0 + t1*x[i] - y[i])**2 for i in range(m)] ) / m)

        if abs(MSE - MSE_New ) <= conv_threshold:
            print ('Converged, iterations: ', iter)
            converged = True
    
        MSE = MSE_New   
        iter += 1 
    
        if iter == max_iter:
            print ('Max interactions reached')
            converged = True

    return t0,t1

if __name__ == '__main__':
    Inter, Coeff = gradient_descent(x = X,y = y,learn_rate=0.00003 ,conv_threshold=1e-8, batch_size=32,max_iter=2500)
    print ("Gradient Descent Results")
    print (('Intercept = %s Coefficient = %s') %(Inter, Coeff))<jupyter_output>Max interactions reached
Gradient Descent Results
Intercept = [1.29812886] Coefficient = [0.09381787]<jupyter_text>Train Validation Test split<jupyter_code>import pandas as pd      
from sklearn.model_selection import train_test_split              
                        
original_data = pd.read_csv("mtcars.csv")                   
def data_split(dat,trf = 0.5,vlf=0.25,tsf = 0.25):
    nrows = dat.shape[0]    
    trnr = int(nrows*trf)
    vlnr = int(nrows*vlf)    
    
    tr_data,rmng = train_test_split(dat,train_size = trnr,random_state=42)
    vl_data, ts_data = train_test_split(rmng,train_size = vlnr,random_state=45)  
    
    return (tr_data,vl_data,ts_data)
train_data, validation_data, test_data = data_split(original_data,trf=0.5,vlf=0.25,tsf=0.25)<jupyter_output>C:\Users\ashish.patel\Anaconda3\lib\site-packages\sklearn\model_selection\_split.py:2026: FutureWarning: From version 0.21, test_size will always complement train_size unless both are specified.
  FutureWarning)<jupyter_text>Decision Tree<jupyter_code># Grid search on Decision Trees
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split,GridSearchCV
from sklearn.metrics import classification_report,confusion_matrix,accuracy_score
from sklearn.pipeline import Pipeline
input_data = pd.read_csv("ad.csv", header=None)
input_data.head(5)
X_columns = set(input_data.columns.values)
y = input_data[len(input_data.columns.values)-1]
X_columns.remove(len(input_data.columns.values)-1)
X = input_data[list(X_columns)]
X_train, X_test,y_train,y_test = train_test_split(X,y,train_size = 0.7,random_state=33)
pipeline = Pipeline([
    ('clf', DecisionTreeClassifier(criterion='entropy'))
])
parameters = {
    'clf__max_depth': (50,100,150),
    'clf__min_samples_split': (2, 3),
    'clf__min_samples_leaf': (1, 2, 3)
}
grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='accuracy')
grid_search.fit(X_train, y_train)
y_pred = grid_search.predict(X_test)
print ('\n Best score: \n', grid_search.best_score_)
print ('\n Best parameters set: \n')
best_parameters = grid_search.best_estimator_.get_params()
for param_name in sorted(parameters.keys()):
    print ('\t%s: %r' % (param_name, best_parameters[param_name]))
print ("\n Confusion Matrix on Test data \n",confusion_matrix(y_test,y_pred))
print ("\n Test Accuracy \n",accuracy_score(y_test,y_pred))
print ("\nPrecision Recall f1 table \n",classification_report(y_test, y_pred))<jupyter_output>Best score: 
 0.966884531590414

 Best parameters set: 

	clf__max_depth: 100
	clf__min_samples_leaf: 1
	clf__min_samples_split: 2

 Confusion Matrix on Test data 
 [[815  18]
 [ 16 135]]

 Test Accuracy 
 0.9654471544715447

Precision Recall f1 table 
              precision    recall  f1-score   support

          0       0.98      0.98      0.98       833
          1       0.88      0.89      0.89       151

avg / total       0.97      0.97      0.97       984
{'fraction_non_alphanumeric': 0.12369082530372853, 'fraction_numerical': 0.05121491411813992, 'mean_word_length': 3.7248886689757548, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '606888', 'license': 'MIT', 'n_tokens_mistral': 3779, 'n_tokens_neox': 3430, 'n_words': 732, 'path': 'Basic Statistics.ipynb', 'repo_name': 'ashishpatel26/Basic-Statistics', 'size': '9548'}