<jupyter_start><jupyter_text>Basic Data Science and ML Pipeline  OSEMN Pipeline * O - Obtaining our data* S - Scrubbing / Cleaning our data* E - Exploring / Visualizing our data will allow us to find patterns and trends* M - Modeling our data will give us our predictive power as a wizard* N - INterpreting our dataFor reference : https://www.linkedin.com/pulse/life-data-science-osemn-randy-lao/?lipi=urn%3Ali%3Apage%3Ad_flagship3_profile_view_base_post_details%3BmDlg5VsdSBCLBps2R0vRZA%3D%3D  Load the dataset  Basic EDA and statistical analysis<jupyter_code>## gives information about the data types,columns, null value counts, memory usage etc
## you can use pd_dataframe.info function here<jupyter_output><empty_output><jupyter_text>**DataFrame.describe()** method generates descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values. This method tells us a lot of things about a dataset. One important thing is that the describe() method deals only with numeric values. It doesn't work with any categorical values. So if there are any categorical values in a column the describe() method will ignore it and display summary for the other columns unless parameter include="all" is passed.Now, let's understand the statistics that are generated by the describe() method:* count tells us the number of NoN-empty rows in a feature.* mean tells us the mean value of that feature.* std tells us the Standard Deviation Value of that feature.* min tells us the minimum value of that feature.* 25%, 50%, and 75% are the percentile/quartile of each features. This quartile information helps us to detect Outliers.* max tells us the maximum value of that feature.<jupyter_code>## basic statistic details about the data (note only numerical columns would be displayed here unless parameter include="all")
## use pd_dataframe.describe() function to get summary of dataset<jupyter_output><empty_output><jupyter_text>The Question creeping out of this summary Can minimum value of below listed columns be zero (0)?On these columns, a value of zero does not make sense and thus indicates missing value.Following columns or variables have an invalid zero value:1. Glucose2. BloodPressure3. SkinThickness4. Insulin5. BMI  It is better to replace zeros with nan since after that counting them would be easier and zeros need to be replaced with suitable values<jupyter_code># count all nan values in each column for their proper handling<jupyter_output><empty_output><jupyter_text>To fill these Nan values the data distribution needs to be understood<jupyter_code># Plot histograms of values of various features<jupyter_output><empty_output><jupyter_text>Aiming to impute nan values for the columns in accordance with their distribution<jupyter_code># replace NaN values by mean or any other suitable value
# test your code for various values that can replace NaN<jupyter_output><empty_output><jupyter_text>Plotting after Nan removal<jupyter_code># Plot histograms after NaN are removed and observe changes<jupyter_output><empty_output><jupyter_text>SkewnessA ***left-skewed distribution*** has a long left tail. Left-skewed distributions are also called negatively-skewed distributions. That’s because there is a long tail in the negative direction on the number line. The mean is also to the left of the peak.A ***right-skewed distribution*** has a long right tail. Right-skewed distributions are also called positive-skew distributions. That’s because there is a long tail in the positive direction on the number line. The mean is also to the right of the peak.![](https://www.statisticshowto.datasciencecentral.com/wp-content/uploads/2014/02/pearson-mode-skewness.jpg)<jupyter_code>## observe the shape of the data
## data type analysis

## Plot data type vs count of each datatype
## null count analysis
## use msnno (import missingno as msno) for null count analysis
## check the balance of the data by plotting the count of outcomes by their value<jupyter_output><empty_output><jupyter_text>Scatter matrix of uncleaned data<jupyter_code>## plot scatter matrix of uncleaned data for further analysis<jupyter_output><empty_output><jupyter_text>The pairs plot builds on two basic figures, the histogram and the scatter plot. The histogram on the diagonal allows us to see the distribution of a single variable while the scatter plots on the upper and lower triangles show the relationship (or lack thereof) between two variables.  Pair plot for clean data<jupyter_code>## plot scatter matrix of uncleaned data for further analysis<jupyter_output><empty_output><jupyter_text>***Pearson's Correlation Coefficient***: helps you find out the relationship between two quantities. It gives you the measure of the strength of association between two variables. The value of Pearson's Correlation Coefficient can be between -1 to +1. 1 means that they are highly correlated and 0 means no correlation.A heat map is a two-dimensional representation of information with the help of colors. Heat maps can help the user visualize simple or complex information.  Heatmap for unclean data<jupyter_code>## try plotting heatmap for unclean data<jupyter_output><empty_output><jupyter_text>Heatmap for clean data<jupyter_code>## try plotting heatmap for unclean data<jupyter_output><empty_output><jupyter_text>Scaling the data data Z is rescaled such that μ = 0 and 𝛔 = 1, and is done through this formula:![](https://cdn-images-1.medium.com/max/800/0*PXGPVYIxyI_IEHP7.) to learn more about scaling techniqueshttps://medium.com/@rrfd/standardize-or-normalize-examples-in-python-e3f174b65dfc https://machinelearningmastery.com/rescaling-data-for-machine-learning-in-python-with-scikit-learn/<jupyter_code>## use standard scaler to scale the dataset<jupyter_output><empty_output><jupyter_text>Test Train Split and Cross Validation methods***Train Test Split*** : To have unknown datapoints to test the data rather than testing with the same points with which the model was trained. This helps capture the model performance much better.![](https://cdn-images-1.medium.com/max/1600/1*-8_kogvwmL1H6ooN1A1tsQ.png)***Cross Validation***: When model is split into training and testing it can be possible that specific type of data point may go entirely into either training or testing portion. This would lead the model to perform poorly. Hence over-fitting and underfitting problems can be well avoided with cross validation techniques![](https://cdn-images-1.medium.com/max/1600/1*4G__SV580CxFj78o9yUXuQ.png)***About Stratify*** : Stratify parameter makes a split so that the proportion of values in the sample produced will be the same as the proportion of values provided to parameter stratify.For example, if variable y is a binary categorical variable with values 0 and 1 and there are 25% of zeros and 75% of ones, stratify=y will make sure that your random split has 25% of 0's and 75% of 1's.For Reference : https://towardsdatascience.com/train-test-split-and-cross-validation-in-python-80b61beca4b6<jupyter_code># Do train test split and train a Knn Model from sklearn
## get train score
## score that comes from testing on the same datapoints that were used for training
## get test score and optimal value of k
## score that comes from testing on the datapoints that were split in the beginning to be used for testing solely<jupyter_output><empty_output><jupyter_text>Result Visualisation<jupyter_code>## Plot train and test accuracy for various values of k<jupyter_output><empty_output><jupyter_text>The best result is captured at k, then k should be used for the final model<jupyter_code>## Setup a knn classifier with k neighbors
## fit the model and calculate score
## try to plot decision boundary<jupyter_output><empty_output><jupyter_text>Model Performance Analysis  1. Confusion MatrixThe confusion matrix is a technique used for summarizing the performance of a classification algorithm i.e. it has binary outputs.![](https://cdn-images-1.medium.com/max/1600/0*-GAP6jhtJvt7Bqiv.png) ***In the famous cancer example***: Cases in which the doctor predicted YES (they have the disease), and they do have the disease will be termed as TRUE POSITIVES (TP). The doctor has correctly predicted that the patient has the disease. Cases in which the doctor predicted NO (they do not have the disease), and they don’t have the disease will be termed as TRUE NEGATIVES (TN). The doctor has correctly predicted that the patient does not have the disease. Cases in which the doctor predicted YES, and they do not have the disease will be termed as FALSE POSITIVES (FP). Also known as “Type I error”. Cases in which the doctor predicted NO, and they have the disease will be termed as FALSE NEGATIVES (FN). Also known as “Type II error”.![](https://cdn-images-1.medium.com/max/1600/0*9r99oJ2PTRi4gYF_.jpg)For Reference: https://medium.com/@djocz/confusion-matrix-aint-that-confusing-d29e18403327<jupyter_code>## Get confusion matrix<jupyter_output><empty_output><jupyter_text>2. Classification ReportReport which includes Precision, Recall and F1-Score. Precision Score        TP – True Positives        FP – False Positives        Precision – Accuracy of positive predictions.        Precision = TP/(TP + FP)            Recall Score        FN – False Negatives        Recall(sensitivity or true positive rate): Fraction of positives that were correctly identified.        Recall = TP/(TP+FN)         F1 Score        F1 Score (aka F-Score or F-Measure) – A helpful metric for comparing two classifiers.        F1 Score takes into account precision and the recall.         It is created by finding the the harmonic mean of precision and recall.        F1 = 2 x (precision x recall)/(precision + recall)                        > > ***Precision*** - Precision is the ratio of correctly predicted positive observations to the total predicted positive observations. The question that this metric answer is of all passengers that labeled as survived, how many actually survived? High precision relates to the low false positive rate. We have got 0.788 precision which is pretty good.> > > > Precision = TP/TP+FP> > > > ***Recall (Sensitivity)*** - Recall is the ratio of correctly predicted positive observations to the all observations in actual class - yes. The question recall answers is: Of all the passengers that truly survived, how many did we label? A recall greater than 0.5 is good.> > > > Recall = TP/TP+FN> > > > ***F1 score*** - F1 Score is the weighted average of Precision and Recall. Therefore, this score takes both false positives and false negatives into account. Intuitively it is not as easy to understand as accuracy, but F1 is usually more useful than accuracy, especially if you have an uneven class distribution. Accuracy works best if false positives and false negatives have similar cost. If the cost of false positives and false negatives are very different, it’s better to look at both Precision and Recall. > > > > F1 Score = 2*(Recall * Precision) / (Recall + Precision)                For Reference: http://joshlawman.com/metrics-classification-report-breakdown-precision-recall-f1/                        : https://blog.exsilio.com/all/accuracy-precision-recall-f1-score-interpretation-of-performance-measures/<jupyter_code>## find precision recall f1 score and support<jupyter_output><empty_output><jupyter_text>3. ROC - AUCROC (Receiver Operating Characteristic) Curve tells us about how good the model can distinguish between two things (e.g If a patient has a disease or no). Better models can accurately distinguish between the two. Whereas, a poor model will have difficulties in distinguishing between the twoWell Explained in this video: https://www.youtube.com/watch?v=OAl6eAyP-yo<jupyter_code>## use roc_curve under sklearn's metrices to plot roc curve and also roc_auc_score to find area under ROC curve<jupyter_output><empty_output><jupyter_text>Hyper Parameter optimizationGrid search is an approach to hyperparameter tuning that will methodically build and evaluate a model for each combination of algorithm parameters specified in a grid. Let’s consider the following example: Suppose, a machine learning model X takes hyperparameters a1, a2 and a3. In grid searching, you first define the range of values for each of the hyperparameters a1, a2 and a3. You can think of this as an array of values for each of the hyperparameters. Now the grid search technique will construct many versions of X with all the possible combinations of hyperparameter (a1, a2 and a3) values that you defined in the first place. This range of hyperparameter values is referred to as the grid. Suppose, you defined the grid as:a1 = [0,1,2,3,4,5]a2 = [10,20,30,40,5,60]a3 = [105,105,110,115,120,125]Note that, the array of values of that you are defining for the hyperparameters has to be legitimate in a sense that you cannot supply Floating type values to the array if the hyperparameter only takes Integer values.Now, grid search will begin its process of constructing several versions of X with the grid that you just defined.It will start with the combination of [0,10,105], and it will end with [5,60,125]. It will go through all the intermediate combinations between these two which makes grid search computationally very expensive.<jupyter_code>## use GridSearchCV
## In case of classifier like knn the parameter to be tuned is n_neighbors<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06296985485671752, 'fraction_numerical': 0.0148864905098623, 'mean_word_length': 5.008944543828265, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 99, 'https://': 13, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '521146', 'license': 'MIT', 'n_tokens_mistral': 3695, 'n_tokens_neox': 3430, 'n_words': 1727, 'path': 'Diabetes/Template.ipynb', 'repo_name': 'abhisngh/Data-Science', 'size': '13435'}