<jupyter_start><jupyter_text>Data Science Unit 2 Sprint Challenge 3 Logistic Regression and BeyondIn this sprint challenge you will fit a logistic regression modeling the probability of an adult having an income above 50K. The dataset is available at UCI:https://archive.ics.uci.edu/ml/datasets/adultYour goal is to:1. Load, validate, and clean/prepare the data.2. Fit a logistic regression model3. Answer questions based on the results (as well as a few extra questions about the other modules)Don't let the perfect be the enemy of the good! Manage your time, and make sure to get to all parts. If you get stuck wrestling with the data, simplify it (if necessary, drop features or rows) so you're able to move on. If you have time at the end, you can go back and try to fix/improve. HintsIt has a variety of features - some are continuous, but many are categorical. You may find [pandas.get_dummies](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html) (a method to one-hot encode) helpful!The features have dramatically different ranges. You may find [sklearn.preprocessing.minmax_scale](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.minmax_scale.htmlsklearn.preprocessing.minmax_scale) helpful!  Part 1 - Load, validate, and prepare dataThe data is available at: https://archive.ics.uci.edu/ml/datasets/adultLoad it, name the columns, and make sure that you've loaded the data successfully. Note that missing values for categorical variables can essentially be considered another category ("unknown"), and may not need to be dropped.You should also prepare the data for logistic regression - one-hot encode categorical features as appropriate.<jupyter_code># TODO - your work!
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt


url_data = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'
url_test = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test'

col_names = ('age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 
            'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 
            'hours-per-week', 'native-country', 'income')
df_data = pd.read_csv(url_data, header=None, names=col_names)
df_data.head()
df_data.income.value_counts()
df_data.loc[:, 'income'].replace(regex=True, to_replace="<=", value="under_", inplace=True)
df_data.loc[:, 'income'].replace(regex=True, to_replace=">", value="over_", inplace=True)
df_data.income.value_counts()
df_data.describe()
df_data.shape
df_data.info()
df.replace('?', np.NaN)
df_data.isna().sum()
df_data.sex.value_counts()
# not working, gah
# encode sex from categorical to numeric
#df_data['sex'] = [1 if each == "Male" else 0 for each in df_data['sex']]
#df_data.head()

# Just figured out this didnt work because there seems to be a space before Male
# So I spent waaaaay too much time trying to find another way to fix the data. *sigh*
# more one hot tries
one_hot_sex = pd.get_dummies(df_data.sex)
one_hot_sex


# Drop column 'sex' as it is now encoded
#df_data = df_data.drop('sex',axis = 1)
# Join the encoded df
df_data = df_data.join(one_hot_sex)
df_data.head() 

df_data.head()
df_data.describe()
# more one hot tries
one_hot_income2 = pd.get_dummies(df_data.income)
one_hot_income2
df_data = df_data.join(one_hot_income2)
df_data_add.head() 
df_data.describe()
df_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 32561 entries, 0 to 32560
Data columns (total 19 columns):
age               32561 non-null int64
workclass         32561 non-null object
fnlwgt            32561 non-null int64
education         32561 non-null object
education-num     32561 non-null int64
marital-status    32561 non-null object
occupation        32561 non-null object
relationship      32561 non-null object
race              32561 non-null object
sex               32561 non-null object
capital-gain      32561 non-null int64
capital-loss      32561 non-null int64
hours-per-week    32561 non-null int64
native-country    32561 non-null object
income            32561 non-null object
 Female           32561 non-null uint8
 Male             32561 non-null uint8
 over_50K         32561 non-null uint8
 under_50K        32561 non-null uint8
dtypes: int64(6), object(9), uint8(4)
memory usage: 3.9+ MB<jupyter_text>Part 2 - Fit and present a Logistic RegressionYour data should now be in a state to fit a logistic regression. Use scikit-learn, define your `X` (independent variable) and `y`, and fit a model.Then, present results - display coefficients in as interpretible a way as you can (hint - scaling the numeric features will help, as it will at least make coefficients more comparable to each other). If you find it helpful for interpretation, you can also generate predictions for cases (like our 5 year old rich kid on the Titanic) or make visualizations - but the goal is your exploration to be able to answer the question, not any particular plot (i.e. don't worry about polishing it).It is *optional* to use `train_test_split` or validate your model more generally - that is not the core focus for this week. So, it is suggested you focus on fitting a model first, and if you have time at the end you can do further validation.<jupyter_code># TODO - your work!
from sklearn.linear_model import LinearRegression

X = df_data[['age', 'education-num', 'hours-per-week', ' Female', ' Male']]
y = df_data[' over_50K']

#X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
#X_scaled = X_std * (max - min) + min

linear_reg = LinearRegression().fit(X, y)
linear_reg.score(X, y)
linear_reg.predict(df_data[['age', 'education-num', 'hours-per-week', ' Female', ' Male']])
linear_reg.coef_
test_case = np.array([[47, 16, 40, 0, 1]])  # 47 year old male with a 4 year degree and works 40 hours a week
linear_reg.predict(test_case)
from sklearn.linear_model import LogisticRegression

log_reg = LogisticRegression().fit(X, y)
log_reg.score(X, y)
np.set_printoptions(threshold=np.nan)
log_reg.predict(df_data[['age', 'education-num', 'hours-per-week', ' Female', ' Male']])
log_reg.predict(test_case)[0]
log_reg.predict_proba(test_case)[0]
# What's the math?
log_reg.coef_
log_reg.intercept_
# The logistic sigmoid "squishing" function, implemented to accept numpy arrays
def sigmoid(x):
  return 1 / (1 + np.e**(-x))
sigmoid(log_reg.intercept_ + np.dot(log_reg.coef_, np.transpose(test_case)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10243376463339494, 'fraction_numerical': 0.026802218114602587, 'mean_word_length': 4.4563025210084035, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '102547', 'license': 'MIT', 'n_tokens_mistral': 2201, 'n_tokens_neox': 2016, 'n_words': 722, 'path': 'DS_Unit_2_Sprint_Challenge_3.ipynb', 'repo_name': 'BrianBehnke/DS-Unit-2-Sprint-3-Advanced-Regression', 'size': '6492'}