<jupyter_start><jupyter_text>PCA and AutoencodersWhen high dimensional data is near a linear manifold; if we can find this linearmanifold then we can project data with N dims onto the manifold reducing the data to M dimsIn directions orthogonal to the data there is not much variation in the dataTry data reduction on rossman starting w/variable importance dimsCan do this efficiently w/PCA algorithm, inefficiently with 1 hidden layer NNAdvantage w/NN is we can generalize to DNN to deal with curved manifoldsPCA -- start w/N dim dataReconstrct using mean vlue oer all the data on the M_N directions not representedhttps://stats.stackexchange.com/questions/229092/how-to-reverse-pca-and-reconstruct-original-variables-from-several-principal-comIf we view the projecion of N onto M as a MSE distance we can use a MSE loss functionin an sutoencoder to mimic the PCA decomposition. This won't be exactly the same but theweights will give us a reduced dimensionlaity<jupyter_code>Adding an encoder and decoder to reproduce the input as closely as possible with M hidden nodes in 1 hidden layer. 

Example of linear AE and PCA.




<h6>PCA w/Deep AE</h6>
Deep Autoencoders HInton: after pretraining layers of autoencoders he was able to demonstrate better percormance 
from Deep AE than PCA. The learning time is linear or better than the number of training cases. The final model
is compact and fast. Very difficult to optimize AE using BP alone. If you use small initial weights, the BP gradient
vanished or died. You can use unsupervised layer by layer pretraining or initilaiE weights carefully 
as in echo state networks
MNIST 2006 Salakhutdinov was first successful deep autoencoder. 
<h6>Deep AE demo with MNIST</h6>
Start with MNIST in 784 -> 1000->500->250 in 3 layers in 30 linear units. Then decoded back to 784. Used stack of
RBMS to initialize the weights for encoding. Then took the transposes of these weights to init the decoder for them. 
After RBMs are trained; then apply backprop to minimize reconstruction error. Use cross entropy error. Once started bp, 
the weights for encoding and decoding diverged. 
<img src="hintonpca2.png">
Can see from the image above the 30-D DNN AE is better than the original bc the 8 does not have the missing edge
artifact. It was able to reconstruct the edge. The 30-D Deep AE is visibly better than the 30-D PCA. 

<h6>LSA - Latent Semantic Analysis</h6>
Using Deep AE for document retrieval. Deep AE should work better than LSI. Using database of documents 
showed 10 components extracted from Deep AE was better
than 50 components extracted with a linear method like Latent Semantic Analysis. 
<p>Convert each document to Bag of words remove stop words and count occurrence of words</p>
<p></p><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.03818443804034582, 'fraction_numerical': 0.017651296829971182, 'mean_word_length': 4.531872509960159, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 19, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '328952', 'license': 'Apache-2.0', 'n_tokens_mistral': 804, 'n_tokens_neox': 717, 'n_words': 423, 'path': 'hinton/pca_autoencoders.ipynb', 'repo_name': 'dougc333/DeepLearning', 'size': '2776'}