<jupyter_start><jupyter_text>Build Resnet from Scratch Model Configuration**Net Architecture:** The same as decribed in the homework.   - dropout layer: p=0.2  - max pooling layer: stride = 2, kenerl_size=2**Batch Size:** 500**Optimizer:**   - sgd(lr=0.05, momentum=0.9, weight_decay=5e-4)  - for every 30 epoches, `lr /= 2`**Data Augmentation:**     - RandomCrop  - RandomHorizontalFlip Experiment Results**For 50 epochs, the highest testing accuracy I obtained during the training is 66.06% at the 32-th epoch.** After that the testing accuracy drops a bit and oscillate around 64%.Part of the log is provided below. **The final test accuracy is 64.48%**.```2018-10-05 18:53:04,475 - [INFO] - => Epoch: [32/50] | Training Loss:[0.47846364974975586] | Training Accuracy: [0.8605199999999997]2018-10-05 18:53:11,658 - [INFO] - => Epoch: [32/50] | Testing Loss:[1.299492597579956] | Testing Accuracy: [0.6606000000000001]2018-10-05 18:53:11,869 - [INFO] - => Best parameters are updated2018-10-05 18:54:26,938 - [INFO] - => Epoch: [33/50] | Training Loss:[0.45494773983955383] | Training Accuracy: [0.8647199999999999]2018-10-05 18:55:49,112 - [INFO] - => Epoch: [34/50] | Training Loss:[0.4044838547706604] | Training Accuracy: [0.8736999999999999]2018-10-05 18:55:56,082 - [INFO] - => Epoch: [34/50] | Testing Loss:[1.3297039270401] | Testing Accuracy: [0.6481999999999999]2018-10-05 18:57:11,305 - [INFO] - => Epoch: [35/50] | Training Loss:[0.510037899017334] | Training Accuracy: [0.88004]2018-10-05 18:58:33,666 - [INFO] - => Epoch: [36/50] | Training Loss:[0.3825763761997223] | Training Accuracy: [0.88046]2018-10-05 18:58:40,782 - [INFO] - => Epoch: [36/50] | Testing Loss:[1.3726511001586914] | Testing Accuracy: [0.6528]2018-10-05 18:59:55,949 - [INFO] - => Epoch: [37/50] | Training Loss:[0.4261672794818878] | Training Accuracy: [0.88594]2018-10-05 19:00:03,426 - [INFO] - => Best parameters are updated2018-10-05 19:01:18,523 - [INFO] - => Epoch: [38/50] | Training Loss:[0.3609049618244171] | Training Accuracy: [0.8888199999999999]2018-10-05 19:01:25,734 - [INFO] - => Epoch: [38/50] | Testing Loss:[1.576693058013916] | Testing Accuracy: [0.6409999999999999]2018-10-05 19:02:40,913 - [INFO] - => Epoch: [39/50] | Training Loss:[0.4049341082572937] | Training Accuracy: [0.8927200000000001]2018-10-05 19:04:03,075 - [INFO] - => Epoch: [40/50] | Training Loss:[0.3678820729255676] | Training Accuracy: [0.8916000000000001]2018-10-05 19:04:10,092 - [INFO] - => Epoch: [40/50] | Testing Loss:[1.48447585105896] | Testing Accuracy: [0.6491]2018-10-05 19:05:25,303 - [INFO] - => Epoch: [41/50] | Training Loss:[0.3971269726753235] | Training Accuracy: [0.8979600000000001]2018-10-05 19:06:47,696 - [INFO] - => Epoch: [42/50] | Training Loss:[0.3653419613838196] | Training Accuracy: [0.9029600000000002]2018-10-05 19:06:54,902 - [INFO] - => Epoch: [42/50] | Testing Loss:[1.3690619468688965] | Testing Accuracy: [0.6395999999999998]2018-10-05 19:08:10,098 - [INFO] - => Epoch: [43/50] | Training Loss:[0.3626648187637329] | Training Accuracy: [0.9033599999999999]2018-10-05 19:09:32,318 - [INFO] - => Epoch: [44/50] | Training Loss:[0.33587783575057983] | Training Accuracy: [0.9051200000000001]2018-10-05 19:09:39,526 - [INFO] - => Epoch: [44/50] | Testing Loss:[1.5433708429336548] | Testing Accuracy: [0.6497999999999999]2018-10-05 19:10:54,710 - [INFO] - => Epoch: [45/50] | Training Loss:[0.25377246737480164] | Training Accuracy: [0.9077000000000001]2018-10-05 19:12:17,097 - [INFO] - => Epoch: [46/50] | Training Loss:[0.333034873008728] | Training Accuracy: [0.9115400000000001]2018-10-05 19:12:24,290 - [INFO] - => Epoch: [46/50] | Testing Loss:[1.3026373386383057] | Testing Accuracy: [0.6446000000000001]2018-10-05 19:13:39,492 - [INFO] - => Epoch: [47/50] | Training Loss:[0.30075380206108093] | Training Accuracy: [0.9142800000000001]2018-10-05 19:15:01,872 - [INFO] - => Epoch: [48/50] | Training Loss:[0.2824198007583618] | Training Accuracy: [0.9175399999999999]2018-10-05 19:15:08,851 - [INFO] - => Epoch: [48/50] | Testing Loss:[1.137478232383728] | Testing Accuracy: [0.6436]2018-10-05 19:16:24,017 - [INFO] - => Epoch: [49/50] | Training Loss:[0.36052873730659485] | Training Accuracy: [0.91734]2018-10-05 19:17:46,417 - [INFO] - => Epoch: [50/50] | Training Loss:[0.2664807438850403] | Training Accuracy: [0.9181600000000003]2018-10-05 19:17:53,402 - [INFO] - => Epoch: [50/50] | Testing Loss:[1.2844302654266357] | Testing Accuracy: [0.6448]2018-10-05 19:17:53,529 - [INFO] - Trained on [50] epoch, with test accuracy [0.6448].  => During the training stages, historical best test accuracy is  [0.6618]```Some simple visualization on- training accuracy v.s. testing accuracy- training loss v.s. testing lossare provided below.<jupyter_code>import torch
import matplotlib.pyplot as plt
checkpoint = torch.load('../myresnet_checkpoint.pth.tar',map_location='cpu')
testing_accuracy_seq = checkpoint['testing_accuracy_seq']
training_accuracy_seq = checkpoint['training_accuracy_seq']
plt.plot( training_accuracy_seq, '-',  color='blue', linewidth=2, label="train acc")
plt.plot( testing_accuracy_seq, '-*', color='red', linewidth=2, label="test acc")
plt.legend()
plt.show()
testing_loss_seq = checkpoint['testing_loss_seq']
training_loss_seq = checkpoint['training_loss_seq']
plt.plot( training_loss_seq, '-',  color='blue', linewidth=2, label="train loss")
plt.plot( testing_loss_seq, '-*', color='red', linewidth=2, label="test loss")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Build Resnet from Pretrained Resnet18 Model Configuration**Net Architecture:** Change the last fully connected layer with `out_features=100`.  - dropout layer: p=0.2  - max pooling layer: stride = 2, kenerl_size=2**Batch Size:** 100 (avoid the memory deficiency issue.)**Optimizer:**   - sgd(lr=0.05, momentum=0.9, weight_decay=5e-4)  - for every 30 epoches, `lr /= 2`**Data Augmentation:**   - Resize the figure to 224 * 224  - RandomCrop  - RandomHorizontalFlip Experiment ResultsTraining Strategy: Instead of only train the last fully-connected layer, I train on the whole net work to get higher testing accuracy, which is of course much more computational intensive,I set wall time to 2 hours, but each epoch takes around 9 minutes to train, so I cannot finished the planned 30 epochs. Within 2 hours limitation, I managed to finish 13 epochs of training with around **73% testing accuracy.** The highest testing accuracy I obtained during the training is 76.18% at the 10-th epoch.** After that the testing accuracy drops a bit and oscillate around 73%.The log is provided below. **The final test accuracy is 73.97%**.```2018-10-05 22:05:34,010 - [INFO] - torch version: 0.3.02018-10-05 22:05:34,011 - [INFO] - Data Preparation...2018-10-05 22:05:39,298 - [INFO] - Loading Data...2018-10-05 22:05:39,298 - [INFO] - Model setting...2018-10-05 22:05:41,102 - [INFO] - Resume from the checkpoint...2018-10-05 22:05:41,103 - [INFO] - => no checkpoint found at './tf_checkpoint.pth.tar'2018-10-05 22:05:41,103 - [INFO] - => Training based on the resnet-18 from scratch...2018-10-05 22:05:41,103 - [INFO] - Model Training...2018-10-05 22:14:06,773 - [INFO] - => Epoch: [1/30] | Training Loss:[1.6555067300796509] | Training Accuracy: [0.45396000000000003]2018-10-05 22:14:52,561 - [INFO] - => Best parameters are updated2018-10-05 22:23:18,275 - [INFO] - => Epoch: [2/30] | Training Loss:[1.0985631942749023] | Training Accuracy: [0.63412]2018-10-05 22:24:00,918 - [INFO] - => Epoch: [2/30] | Testing Loss:[1.2507389783859253] | Testing Accuracy: [0.5905]2018-10-05 22:24:01,436 - [INFO] - => Best parameters are updated2018-10-05 22:32:23,823 - [INFO] - => Epoch: [3/30] | Training Loss:[0.8547025918960571] | Training Accuracy: [0.6952200000000001]2018-10-05 22:33:06,877 - [INFO] - => Best parameters are updated2018-10-05 22:41:31,053 - [INFO] - => Epoch: [4/30] | Training Loss:[0.8175657391548157] | Training Accuracy: [0.7298199999999999]2018-10-05 22:42:13,646 - [INFO] - => Epoch: [4/30] | Testing Loss:[1.4175124168395996] | Testing Accuracy: [0.6254000000000001]2018-10-05 22:42:14,149 - [INFO] - => Best parameters are updated2018-10-05 22:50:41,597 - [INFO] - => Epoch: [5/30] | Training Loss:[0.7954866886138916] | Training Accuracy: [0.75752]2018-10-05 22:51:25,901 - [INFO] - => Best parameters are updated2018-10-05 22:59:51,816 - [INFO] - => Epoch: [6/30] | Training Loss:[0.9352169632911682] | Training Accuracy: [0.7698400000000001]2018-10-05 23:00:34,434 - [INFO] - => Epoch: [6/30] | Testing Loss:[1.1475493907928467] | Testing Accuracy: [0.6557]2018-10-05 23:09:02,586 - [INFO] - => Epoch: [7/30] | Training Loss:[1.0427197217941284] | Training Accuracy: [0.7843600000000001]2018-10-05 23:18:08,401 - [INFO] - => Epoch: [8/30] | Training Loss:[0.8770634531974792] | Training Accuracy: [0.795]2018-10-05 23:18:50,968 - [INFO] - => Epoch: [8/30] | Testing Loss:[1.046979546546936] | Testing Accuracy: [0.6552999999999999]2018-10-05 23:27:19,654 - [INFO] - => Epoch: [9/30] | Training Loss:[0.7507113814353943] | Training Accuracy: [0.8043800000000001]2018-10-05 23:28:02,417 - [INFO] - => Learning rate is updated!2018-10-05 23:36:26,854 - [INFO] - => Epoch: [10/30] | Training Loss:[0.24965983629226685] | Training Accuracy: [0.9040199999999999]2018-10-05 23:37:09,390 - [INFO] - => Epoch: [10/30] | Testing Loss:[0.8782387375831604] | Testing Accuracy: [0.7618]2018-10-05 23:37:09,905 - [INFO] - => Best parameters are updated2018-10-05 23:45:37,206 - [INFO] - => Epoch: [11/30] | Training Loss:[0.23527362942695618] | Training Accuracy: [0.94082]2018-10-05 23:54:39,736 - [INFO] - => Epoch: [12/30] | Training Loss:[0.31548258662223816] | Training Accuracy: [0.9526399999999999]2018-10-05 23:55:22,430 - [INFO] - => Epoch: [12/30] | Testing Loss:[0.9379260540008545] | Testing Accuracy: [0.7397]```Some simple visualization on- training accuracy v.s. testing accuracy- training loss v.s. testing lossare provided below.<jupyter_code>checkpoint = torch.load('../tf_checkpoint.pth.tar',map_location='cpu')
testing_accuracy_seq = checkpoint['testing_accuracy_seq']
training_accuracy_seq = checkpoint['training_accuracy_seq']
plt.plot( training_accuracy_seq, '-',  color='blue', linewidth=2, label="train acc")
plt.plot( testing_accuracy_seq, '-*', color='red', linewidth=2, label="test acc")
plt.legend()
plt.show()
testing_loss_seq = checkpoint['testing_loss_seq']
training_loss_seq = checkpoint['training_loss_seq']
plt.plot( training_loss_seq, '-',  color='blue', linewidth=2, label="train loss")
plt.plot( testing_loss_seq, '-*', color='red', linewidth=2, label="test loss")
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17182966775854, 'fraction_numerical': 0.26167524567150213, 'mean_word_length': 5.088888888888889, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 68, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 31, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '267939', 'license': 'MIT', 'n_tokens_mistral': 5808, 'n_tokens_neox': 3949, 'n_words': 888, 'path': 'Python_IE534/hw4/submitted/.ipynb_checkpoints/report-checkpoint.ipynb', 'repo_name': 'Rothdyt/codes-for-courses', 'size': '10685'}