<jupyter_start><jupyter_text>Question 1  1.1<jupyter_code>train = np.loadtxt('pa1train.txt', encoding = 'ascii')
test = np.loadtxt('pa1test.txt', encoding = 'ascii')
validate = np.loadtxt('pa1validate.txt', encoding = 'ascii')
print(train.shape)
print(test.shape)
print(validate.shape)
# splite the data set into X and y: X as feature, y as target.
X_train = train[:, :-1]
y_train = train[:, -1]
X_test = test[:,:-1]
y_test = test[:,-1]
X_valid = validate[:, :-1]
y_valid = validate[:,-1]
def knn_classifier(X_train, y_train, X_test, y_test, k):
    '''get the accuracy error rate with training set and testing set
    '''
    distance_mat = cdist(X_train, X_test) #Get the distance of two pair of matrix
    ind = np.argpartition(distance_mat, k, axis=0)[:k] #Get kth largest neighbors index in vector form
    prediction = np.concatenate(mode(y_train[ind].astype(int), axis = 0)[0])  #Get prediction from y_train
    return 1 - (prediction == y_test).sum()/len(y_test)  #Calculate error rate
#KDTree VERSION
# def knn_classifier(X_train, y_train, X_test, y_test, n):
#     '''get the accuracy error rate with training set and testing set
#     '''
#     tree = KDTree(X_train) # initialize a kdtree with training set
#     ind = tree.query(X_test, k = n)[1] # get the index of the nearest k points' indecies
#     prediction = np.concatenate(mode(y_train[ind].astype(int), axis = 1)[0]) 
#     #get prediction set by taking the most frequent element
#     return 1 - (prediction == y_test).sum()/len(y_test) # calculating training error
knn_classifier(X_train, y_train, X_train, y_train, 3) # with 3nn classfier training error is
train_err = []
valid_err = []
for i in [1,5,9,15]:
    train_err += [knn_classifier(X_train, y_train,X_train, y_train, i)]
    valid_err += [knn_classifier(X_train, y_train,X_valid, y_valid, i)]
output1 = pd.DataFrame(
        {
            'Training Error': train_err,
            'Validation Error': valid_err
        }, index = pd.Series([1,5,9,15], name = 'k')
)
display(output1)<jupyter_output><empty_output><jupyter_text>According to the table above, we can see k = 1 works the best classifier of the training data and validation error<jupyter_code># computing test error for k = 1
print('Test Error for the k = 1 knn classifier is {}'.format(knn_classifier(X_train, y_train, X_test, y_test, 1)))<jupyter_output>Test Error for the k = 1 knn classifier is 0.09399999999999997<jupyter_text>1.2<jupyter_code>projection = np.loadtxt('projection.txt', encoding = 'ascii')
print(projection.shape)
X_train_proj = X_train.dot(projection)
X_test_proj = X_test.dot(projection)
X_valid_proj = X_valid.dot(projection)
train_err = []
valid_err = []
for i in [1,5,9,15]:
    train_err += [knn_classifier(X_train_proj, y_train,X_train_proj, y_train, i)]
    valid_err += [knn_classifier(X_train_proj, y_train,X_valid_proj, y_valid, i)]
output2 = pd.DataFrame(
        {
            'Training Error': train_err,
            'Validation Error': valid_err
        }, index = pd.Series([1,5,9,15], name = 'k')
)
display(output2)<jupyter_output><empty_output><jupyter_text>Based on the validation error, the best classifier is 15-NN classifier.<jupyter_code>print('Test Error for the 15-knn classifier is {}'.format(knn_classifier(X_train_proj, \
                                                                         y_train, X_test_proj, y_test, 15)))
print('After we done project the original data, the accuracy of the prediction was siginicantly reduced.')
print('With full dimentioan dataset, the best validation error is {}'.format(output1['Validation Error'].min()))
print('Wihile, with 20 dimentional projection dataset, the best validation error is {}'.format(output2['Validation Error'].min()))
print('the time cost for training and predicting a 15-nn classfier with full dimention training dataset')
%timeit knn_classifier(X_train, y_train, X_test, y_test, 15)
print('the time cost for training and predicting a 15-nn classfier with 20 projected dimention training dataset')
%timeit knn_classifier(X_train_proj, y_train, X_test_proj, y_test, 15)<jupyter_output>the time cost for training and predicting a 15-nn classfier with full dimention training dataset
1.99 s ± 11.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
the time cost for training and predicting a 15-nn classfier with 20 projected dimention training dataset
82.7 ms ± 437 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
{'fraction_non_alphanumeric': 0.11712931618144888, 'fraction_numerical': 0.025727826675693975, 'mean_word_length': 4.418092909535452, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '450594', 'license': 'MIT', 'n_tokens_mistral': 1568, 'n_tokens_neox': 1469, 'n_words': 456, 'path': 'Knn implementation/.ipynb_checkpoints/Knn-checkpoint.ipynb', 'repo_name': 'syeehyn/Machine-Learning-in-Practice', 'size': '4431'}