<jupyter_start><jupyter_text>Homework 3:Eigenvector DecompositionFor this homework you will use the Communities and Crime Data Set-  http://archive.ics.uci.edu/ml/datasets/communities+and+crime- In the class we talked about how to reduce data dimensionality by extracting new set of featuresusing PCA, LDA and other methods.  The basis of these methods is the eigenvector decompositionof the data matrix.- Load the crime dataset and store it as a matrix (The data is already normalized so you should not need to normalize the data further.)- Report a table with the top 20 eigenvalues, is there a clear point where you could cut the dimensions?<jupyter_code>## load the data in pandas
crime_data = pd.read_csv('communities.data.txt', sep=",", header=None)
crime_data.head
print(crime_data.describe())<jupyter_output>0            4            5            6            7    \
count  1994.000000  1994.000000  1994.000000  1994.000000  1994.000000   
mean     28.683551     5.493982     0.057593     0.463395     0.179629   
std      16.397553     2.873694     0.126906     0.163717     0.253442   
min       1.000000     1.000000     0.000000     0.000000     0.000000   
25%      12.000000     3.000000     0.010000     0.350000     0.020000   
50%      34.000000     5.000000     0.020000     0.440000     0.060000   
75%      42.000000     8.000000     0.050000     0.540000     0.230000   
max      56.000000    10.000000     1.000000     1.000000     1.000000   

               8            9            10           11           12   \
count  1994.000000  1994.000000  1994.000000  1994.000000  1994.000000   
mean      0.753716     0.153681     0.144022     0.424218     0.493867   
std       0.244039     0.208877     0.232492     0.155196     0.143564   
min       0.000000     0.000000     0[...]<jupyter_text>Imputing the dataWe can find that the data is not clean where there are lot of entries missing with '?' value. Hence we plan to imputing the missing data.<jupyter_code>def isnumber(x):
    try:
        float(x)
        return True
    except:
        return False

## change all the non-number instances to nan
crime_data = crime_data[crime_data.applymap(isnumber)]

## convert as matrix and change the datatype to float
crime_matrix = crime_data.values.astype('float')
crime_matrix
from sklearn.impute import SimpleImputer
imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_mean.fit(crime_matrix)
crime_data_matrix_p = imp_mean.transform(crime_matrix)
crime_data_matrix_p.shape
crime_data_matrix_p
from numpy import linalg as LA<jupyter_output><empty_output><jupyter_text>Calculate the eigenvalues and eigenvectors Instead of performing eigenvalue decomposition directly on the covariance matrix $R = A^TA$, I use the singular value decomposition on the crime data.In this case, the singular values are the root square of the eigenvalue of covariance matrix and the $V^T$ contains the eigenvectors.<jupyter_code>u, s, vt = np.linalg.svd(crime_data_matrix_p, full_matrices=True)<jupyter_output><empty_output><jupyter_text>- The eigenvectors:<jupyter_code>print("The eigenvectos of covariance matrix are:", vt)<jupyter_output>The eigenvectos of covariance matrix are: [[-5.53123394e-04 -1.19968821e-03 -9.99999118e-01 ... -1.81501123e-06
  -3.75976360e-06 -4.52118901e-06]
 [-5.05036105e-02 -9.98714061e-01  1.22656965e-03 ... -2.77429497e-04
  -8.01371377e-05 -3.01512856e-04]
 [-9.96714151e-01  5.06316063e-02  4.98884478e-04 ...  2.71377291e-04
  -1.54157976e-03  1.79559232e-04]
 ...
 [ 1.25904828e-05 -1.26924180e-06 -6.69745864e-09 ... -1.28522827e-03
   2.50731577e-02  8.97904379e-04]
 [ 2.61194455e-06 -3.59124771e-07 -2.81955695e-09 ... -7.87703668e-04
   1.41537014e-03 -7.33994697e-04]
 [ 2.39320222e-07 -2.01118625e-08  4.04408331e-10 ...  1.89826373e-05
  -4.42752682e-05 -4.21173134e-05]]<jupyter_text>- The eigenvalues:<jupyter_code>squarer = lambda t: t ** 2
eigs = np.array([squarer(xi) for xi in s])
print("The eigenvalues are:", eigs)
## Check on the shape of the singular values 
## to confirm that there is no zero eigenvalues
eigs.shape<jupyter_output><empty_output><jupyter_text>List the top 20 eigenvalues<jupyter_code>print("The top 20 eigenvalues are \n", eigs[0:20])<jupyter_output>The top 20 eigenvalues are 
 [4.77623528e+12 1.31476439e+07 6.86061468e+05 2.24667102e+04
 2.93218333e+03 2.00404191e+03 1.07771021e+03 6.52550070e+02
 5.52199932e+02 3.65353700e+02 3.12976961e+02 2.33596986e+02
 2.16949841e+02 1.74577407e+02 1.48709631e+02 1.12553687e+02
 1.08318653e+02 1.00416719e+02 8.79745696e+01 8.10405157e+01]
{'fraction_non_alphanumeric': 0.0962671905697446, 'fraction_numerical': 0.236847849814451, 'mean_word_length': 3.1168014375561546, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '451063', 'license': 'MIT', 'n_tokens_mistral': 2345, 'n_tokens_neox': 1659, 'n_words': 424, 'path': 'ECE657 - HW3 Communities and Crime Data.ipynb', 'repo_name': 'ChenSunMac/DKMA_2019', 'size': '4581'}