<jupyter_start><jupyter_text>Data Science Unit 2 Sprint Challenge 3 Logistic Regression and BeyondIn this sprint challenge you will fit a logistic regression modeling the probability of an adult having an income above 50K. The dataset is available at UCI:https://archive.ics.uci.edu/ml/datasets/adultYour goal is to:1. Load, validate, and clean/prepare the data.2. Fit a logistic regression model3. Answer questions based on the results (as well as a few extra questions about the other modules)Don't let the perfect be the enemy of the good! Manage your time, and make sure to get to all parts. If you get stuck wrestling with the data, simplify it (if necessary, drop features or rows) so you're able to move on. If you have time at the end, you can go back and try to fix/improve. HintsIt has a variety of features - some are continuous, but many are categorical. You may find [pandas.get_dummies](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html) (a method to one-hot encode) helpful!The features have dramatically different ranges. You may find [sklearn.preprocessing.minmax_scale](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.minmax_scale.htmlsklearn.preprocessing.minmax_scale) helpful!  Part 1 - Load, validate, and prepare dataThe data is available at: https://archive.ics.uci.edu/ml/datasets/adultLoad it, name the columns, and make sure that you've loaded the data successfully. Note that missing values for categorical variables can essentially be considered another category ("unknown"), and may not need to be dropped.You should also prepare the data for logistic regression - one-hot encode categorical features as appropriate.<jupyter_code># TODO - your work!
import pandas as pd
import numpy as np
from matplotlib.pyplot import scatter
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import MinMaxScaler
from sklearn.linear_model import Ridge
# loading DataFrame
df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data', header=None)
# DataFrame preview
df.head()
# Renaming columns
df.columns = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'income']
# DataFrame preview with renamed columns
df.head()
# Number of rows & columns in DataFrame
df.shape
# Checking for NaNs
df.isna().sum()
# Checking for unique values & what may be null values
for col in df.columns:
    print(col, '\n: ', df[col].unique(),'\n')
# Replacing ' ?' in DataFrame inot NaNs
df = df.replace(' ?', np.nan)
df.isna().sum()
# Dropping all rows with missing values
df = df.dropna()

# Total number of rows with missing values
df.isna().sum()
# Making our target feature ('income') into binary
# <=50K to 0, >50K to 1
df['income'] = df['income'].map({' <=50K':0, ' >50K':1})
df.head()
# Making sure our target feature is 'int' type
df.income.dtypes
#Auto encodes any dataframe column of type category or object.
def dummyEncode(df):
        columnsToEncode = list(df.select_dtypes(include=['category','object']))
        le = LabelEncoder()
        for feature in columnsToEncode:
            try:
                df[feature] = le.fit_transform(df[feature])
            except:
                print('Error encoding '+feature)
        return df
# Encoding features
df1 = dummyEncode(df)
# New df1 with encoded features
df1.head()<jupyter_output><empty_output><jupyter_text>Part 2 - Fit and present a Logistic RegressionYour data should now be in a state to fit a logistic regression. Use scikit-learn, define your `X` (independent variable) and `y`, and fit a model.Then, present results - display coefficients in as interpretible a way as you can (hint - scaling the numeric features will help, as it will at least make coefficients more comparable to each other). If you find it helpful for interpretation, you can also generate predictions for cases (like our 5 year old rich kid on the Titanic) or make visualizations - but the goal is your exploration to be able to answer the question, not any particular plot (i.e. don't worry about polishing it).It is *optional* to use `train_test_split` or validate your model more generally - that is not the core focus for this week. So, it is suggested you focus on fitting a model first, and if you have time at the end you can do further validation.<jupyter_code># Defining x & y variables
X = df1.drop('income', axis=1)
y = df1['income']

# scaling X
scale = MinMaxScaler()
X = scale.fit_transform(X)
# Fitting model to data
lr = LogisticRegression().fit(X, y)
# Calculating coefficients
coef = lr.coef_
# Printing each feature's coefficient
cols = ['age', 'workclass', 'fnlwgt', 'education', 'education-num',
       'marital-status', 'occupation', 'relationship', 'race', 'sex',
       'capital-gain', 'capital-loss', 'hours-per-week', 'native-country']
coef_lst = [print(f'{cols[i]:18}---> { (coef)[0][i]:7.4}\n') for i in range(len(cols))]
print(coef_lst)
# Psudo R^2 score for the model
lr.score(X, y)<jupyter_output><empty_output><jupyter_text>Part 3 - Analysis, Interpretation, and Questions Based on your above model, answer the following questions1. What are 3 features positively correlated with income above 50k?2. What are 3 features negatively correlated with income above 50k?3. Overall, how well does the model explain the data and what insights do you derive from it?*These answers count* - that is, make sure to spend some time on them, connecting to your analysis above. There is no single right answer, but as long as you support your reasoning with evidence you are on the right track.Note - scikit-learn logistic regression does *not* automatically perform a hypothesis test on coefficients. That is OK - if you scale the data they are more comparable in weight. Match the following situation descriptions with the model most appropriate to addressing themIn addition to logistic regression, a number of other approaches were covered this week. Pair them with the situations they are most appropriate for, and briefly explain why.Situations:1. You are given data on academic performance of primary school students, and asked to fit a model to help predict "at-risk" students who are likely to receive the bottom tier of grades.2. You are studying tech companies and their patterns in releasing new products, and would like to be able to model and predict when a new product is likely to be launched.3. You are working on modeling expected plant size and yield with a laboratory that is able to capture fantastically detailed physical data about plants, but only of a few dozen plants at a time.Approaches:1. Ridge Regression2. Quantile Regression3. Survival Analysis  Part 3 Answers:1. What are 3 features positively correlated with income above 50k?> *capital-gain* has the highest positive correlation (18.3 coefficient), then *education-num* (4.9 coefficient),  and then *age* (2.6 cofficient). Here one unit increase in one of these features will cause the income to increase by the coefficient.2. What are 3 features negatively correlated with income above 50k?> *marital status* (-1.3 coefficient), *relationship* (-0.6 coefficient ), *workclass* (-0.7 coefficient). Here one unit increase in one of the features will cause income to decrese.3. Overall, how well does the model explain the data and what insights do you derive from it?> *The score for our model is of 0.819*, meaning the model seems to be pretty good. In other words, *the model gives us a good explanation of the features that are correlated to having an income above 50k*. Thus we can have a good confidence in running predictions.  Other Answers:1. You are given data on academic performance of primary school students, and asked to fit a model to help predict "at-risk" students who are likely to receive the bottom tier of grades.> In this scinario it may be best to use a *Quantile Regression* model. Here were looking to predict which students would be below average and near some low grade score level. Quantile Regression gives us the freedom to fit our model according to the quantile mark of our choice and not just based on the mean of our independent variable. Thus in this case, by calculating the distribution of recorded grade scores among students we can find the quantile mark in the distribution for the grade trashold that defines "at-risk" students. Then fitting our model will allow us to find features that correlate to low performing students and make predictions.2. You are studying tech companies and their patterns in releasing new products, and would like to be able to model and predict when a new product is likely to be launched.> Using *Survival Analysis* in this case may be best. Survival Analysis can be described as a *time-to-event-analysis*. It's a way to model a case with a finite duration where we predict the length of time between two events. Commonly the begining of the time period is defined as "birth", while the event of interest that ends our observation is "death." We should note that we may not be able to calculate the release of products infinitely, thus for some observations one will not find the time of release for that product, yet with Survival Analysis we are still allowed to incorporate these in our predictions of time.3. You are working on modeling expected plant size and yield with a laboratory that is able to capture fantastically detailed physical data about plants, but only of a few dozen plants at a time.> *Rigde Regression* should be an ideal model for this case. Since the laboratory offers much detail (probably tons of features) on a few dozen plants at a time applying a Linear Regression model will most likely produce a model that overfits, meaning that a Linear Regression will fail to generalize thus underminding prediction and explanatory power. So, *Rigdge Regression* would work best because it allows us to add bias to our model. Adding bias will cause variance to decrease and reduce overfitting.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06959241520867951, 'fraction_numerical': 0.008014856807741179, 'mean_word_length': 4.640573318632856, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 22, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '106015', 'license': 'MIT', 'n_tokens_mistral': 2697, 'n_tokens_neox': 2555, 'n_words': 1425, 'path': 'DS_Unit_2_Sprint_Challenge_3.ipynb', 'repo_name': 'carlos-gutier/DS-Unit-2-Sprint-3-Advanced-Regression', 'size': '10231'}