<jupyter_start><jupyter_text>Out of these 15 features 'Salary' is the target feature and rest all are covariates.<jupyter_code>#let see if any cloumn has missing values
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 32561 entries, 0 to 32560
Data columns (total 15 columns):
age               32561 non-null int64
workclass         32561 non-null object
fnlwgt            32561 non-null int64
education         32561 non-null object
education.num     32561 non-null int64
marital.status    32561 non-null object
occupation        32561 non-null object
relationship      32561 non-null object
race              32561 non-null object
sex               32561 non-null object
capital.gain      32561 non-null int64
capital.loss      32561 non-null int64
hours.per.week    32561 non-null int64
native.country    32561 non-null object
income            32561 non-null object
dtypes: int64(6), object(9)
memory usage: 3.7+ MB<jupyter_text>It is clear that none on the column has any Nan values. Since number of samples in each column = total number of samples = 32561Let's classify features into numerical and categorical:Numerical features:- age- fnlwgt- education-num- capital-gain- capital-loss- hours-per-weekCategorical features:- workclass- education- marital-status- occupation- relationship- race- sex- native-country- salary Let's see the classes of the categorical variables<jupyter_code>print('workclass\n',set(df.workclass))
print('\neducation\n',set(df.education))
print('\nmarital.status\n',set(df['marital.status']))
print('\noccupation\n',set(df.occupation))
print('\nrace\n',set(df.race))
print('\nrelationship',set(df.relationship))
print('\nsex',set(df.sex))
print('\nnative.country',set(df['native.country']))
print('\nsalary',set(df.income))<jupyter_output>workclass
 {' Self-emp-not-inc', ' Private', ' Federal-gov', ' State-gov', ' Without-pay', ' Local-gov', ' Never-worked', ' ?', ' Self-emp-inc'}

education
 {' Prof-school', ' Assoc-acdm', ' 9th', ' 11th', ' Some-college', ' Bachelors', ' Assoc-voc', ' 5th-6th', ' 10th', ' 7th-8th', ' 1st-4th', ' HS-grad', ' 12th', ' Preschool', ' Doctorate', ' Masters'}

marital.status
 {' Never-married', ' Married-spouse-absent', ' Separated', ' Divorced', ' Married-civ-spouse', ' Widowed', ' Married-AF-spouse'}

occupation
 {' Adm-clerical', ' Other-service', ' Handlers-cleaners', ' Craft-repair', ' Farming-fishing', ' Transport-moving', ' Machine-op-inspct', ' Tech-support', ' Protective-serv', ' Sales', ' ?', ' Priv-house-serv', ' Exec-managerial', ' Prof-specialty', ' Armed-Forces'}

race
 {' Amer-Indian-Eskimo', ' White', ' Other', ' Black', ' Asian-Pac-Islander'}

relationship {' Other-relative', ' Own-child', ' Wife', ' Husband', ' Not-in-family', ' Unmarried'}

sex {' Male', ' Female'}

nativ[...]<jupyter_text>As we can see that in workclass and occupation we have unknown class '?'.Now let's represent categorical data in terms of numerical dummy variables<jupyter_code>df.workclass = df.workclass.map({ ' Self-emp-inc':0, ' Never-worked':1, ' Local-gov':2, ' Self-emp-not-inc':3, ' ?':4, ' Private':5, ' Federal-gov':6, ' Without-pay':7, ' State-gov':8})

df.income = np.where(df.income == ' >50K',1,0)

df.occupation = df.occupation.map({' Prof-specialty':0, ' Handlers-cleaners':1, ' Craft-repair':2, ' Tech-support':3, ' Protective-serv':4, ' Sales':5, ' ?':6, ' Priv-house-serv':7, ' Transport-moving':8, ' Exec-managerial':9, ' Machine-op-inspct':10, ' Adm-clerical':11, ' Other-service':12, ' Armed-Forces':13, ' Farming-fishing':14})

df['marital.status'] = df['marital.status'].map({' Divorced':0, ' Married-civ-spouse':1, ' Widowed':2, ' Married-spouse-absent':3, ' Never-married':4, ' Separated':5, ' Married-AF-spouse':6})

df.race = df.race.map({' Other':0, ' White':1, ' Asian-Pac-Islander':2, ' Black':3, ' Amer-Indian-Eskimo':4})

df.sex = np.where(df.sex == 'Male',1,0)

df.relationship = df.relationship.map({' Own-child':0, ' Husband':1, ' Other-relative':2, ' Not-in-family':3, ' Unmarried':4, ' Wife':5})

df['native.country'] = df['native.country'].map({' England':0, ' France':1, ' India':2, ' Italy':3, ' Portugal':4, ' Cuba':5, ' Japan':6, ' Laos':7, ' Hungary':8, ' Scotland':9, ' Holand-Netherlands':10, ' Greece':11, ' Poland':12, ' Philippines':13, ' Hong':14, ' Ecuador':15, ' Honduras':16, ' Outlying-US(Guam-USVI-etc)':17, ' Nicaragua':18, ' Jamaica':19, ' Peru':20, ' South':21, ' Columbia':22, ' Iran':23, ' United-States':24, ' Cambodia':25, ' Haiti':26, ' China':27, ' El-Salvador':28, ' Yugoslavia':29, ' Mexico':30, ' Dominican-Republic':31, ' Guatemala':32, ' Vietnam':33, ' Thailand':34, ' ?':35, ' Puerto-Rico':36, ' Ireland':37, ' Germany':38, ' Canada':39, ' Trinadad&Tobago':40, ' Taiwan':41})

df.education = df.education.map({' 9th':0, ' 7th-8th':1, ' Preschool':2, ' 10th':3, ' Doctorate':4, ' 11th':5, ' Prof-school':6, ' 5th-6th':7, ' Bachelors':8, ' Assoc-acdm':9, ' 1st-4th':10, ' Masters':11, ' HS-grad':12, ' Some-college':13, ' Assoc-voc':14, ' 12th':15})

df.head()<jupyter_output><empty_output><jupyter_text>Feature engineering usinf Recursive Feature Elimination and Feature importance rankingThe Recursive Feature Elimination (RFE) method is a feature selection approach. It works by recursively removing attributes and building a model on those attributes that remain. It uses the model accuracy to identify which attributes (and combination of attributes) contribute the most to predicting the target attribute.<jupyter_code>features = df.drop('income',axis=1)
target = df.income
set(df.income)
from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
#define a classifier
model = LogisticRegression()

#create RFE model to return top 3 attributes
rfe = RFE(model,3)
rfe = rfe.fit(features,target)

#summarise the selection of attributes
print(rfe.support_)
print(rfe.ranking_)
print(features.columns.values)<jupyter_output>[False False False False  True  True False  True False False False False
 False False]
[ 5  6 11  8  1  1  3  1  2 12 10  9  4  7]
['age' 'workclass' 'fnlwgt' 'education' 'education.num' 'marital.status'
 'occupation' 'relationship' 'race' 'sex' 'capital.gain' 'capital.loss'
 'hours.per.week' 'native.country']<jupyter_text>So the top three features as per RFE are- educational-num- marital status- relationships- raceLet's use Feature Importance to identify the top 3 featuresMethods that use ensembles of decision trees (like Random Forest or Extra Trees) can also compute the relative importance of each attribute. These importance values can be used to inform a feature selection process.This recipe shows the construction of an Extra Trees ensemble of the dataset and the display of the relative feature importance.<jupyter_code>from sklearn import metrics
from sklearn.ensemble import ExtraTreesClassifier

#define and fit a ExtraTreeClassifier to the data
model = ExtraTreesClassifier()
model.fit(features,target)

#display the feature importance
print(model.feature_importances_)
print('\n',features.columns.values)
#bar plot of feature importance
values = model.feature_importances_
pos = np.arange(14) + 0.02
plt.barh(pos,values,align = 'center')
plt.title('Feature importance plot')
plt.xlabel('feature importance ')
plt.ylabel('features')
plt.yticks(np.arange(14),('age', 'workclass', 'fnlwgt', 'education', 'education.num', 'marital.status','occupation' ,'relationship', 'race' ,'sex', 'capital-gain', 'capital.loss','hours.per.week', 'native.country'))
plt.grid(True)<jupyter_output><empty_output><jupyter_text>So according to feature importance method, top 3 features are fnlwgt,age,hours-per-week.Let's create training and testing datasets<jupyter_code>#updating features: combining best features from both RFE and feature importance
features = features[['education.num','marital.status','relationship','race','age','fnlwgt']]

#here we have consider best features from both RFE and feature importance results

#spliting data into train and test data
X_train,X_test,y_train,y_test = train_test_split(features,target,random_state = 12)

from sklearn.neighbors import KNeighborsClassifier

k_values = np.arange(1,26)
scores = []

for i in k_values:
    clf = KNeighborsClassifier(n_neighbors=i)
    clf.fit(X_train,y_train)
    y_predict = clf.predict(X_test)
    scores.append(metrics.accuracy_score(y_test,y_predict))

print("Accuracy for {} is {}".format(np.argmax(scores),max(scores)))

plt.plot(np.arange(1,26),scores)
plt.title('Varition of accuracy with K value,with best features from RFE and feature importance')
plt.xlabel('K values')
plt.ylabel('Accuracy')
#Let's update features with the results of RFE and evaluate how
#accuracy varies

features1 = features[['education.num','marital.status','relationship','race']]

X_train,X_test,y_train,y_test = train_test_split(features1,target,random_state = 12)

from sklearn.neighbors import KNeighborsClassifier

k_values = np.arange(1,26)
scores = []

for i in k_values:
    clf = KNeighborsClassifier(n_neighbors=i)
    clf.fit(X_train,y_train)
    y_predict = clf.predict(X_test)
    scores.append(metrics.accuracy_score(y_test,y_predict))

print("Accuracy for {} is {}".format(np.argmax(scores),max(scores)))

plt.plot(np.arange(1,26),scores)
plt.title('Varition of accuracy with K value,with best features from RFE ')
plt.xlabel('K values')
plt.ylabel('Accuracy')
#Let's update features with the results of feature importance and evaluate how
#accuracy varies

features2 = features[['age','fnlwgt']]

X_train,X_test,y_train,y_test = train_test_split(features2,target,random_state = 12)

from sklearn.neighbors import KNeighborsClassifier

k_values = np.arange(1,26)
scores = []

for i in k_values:
    clf = KNeighborsClassifier(n_neighbors=i)
    clf.fit(X_train,y_train)
    y_predict = clf.predict(X_test)
    scores.append(metrics.accuracy_score(y_test,y_predict))

print("Accuracy for {} is {}".format(np.argmax(scores),max(scores)))

plt.plot(np.arange(1,26),scores)
plt.title('Varition of accuracy with K value,with best features from feature importance')
plt.xlabel('K values')
plt.ylabel('Accuracy')<jupyter_output>Accuracy for 21 is 0.7555582852229455
{'fraction_non_alphanumeric': 0.15138697619516034, 'fraction_numerical': 0.03580562659846547, 'mean_word_length': 5.43888537048765, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '139488', 'license': 'MIT', 'n_tokens_mistral': 3731, 'n_tokens_neox': 3392, 'n_words': 835, 'path': 'Source.ipynb', 'repo_name': 'satishjasthi/K-Nearest-Neighbors', 'size': '10166'}