<commit_before>---
layout: post
title:  "Udacity Deep Learning P1"
desc: "Describe the Udacity study of DL , assignment 3"
keywords: "DL"
date: 2016-11-30
categories: [Life]
tags: [blog]
icon: fa-bookmark-o
---

After I installed my new laptop, I continued to study my Udacity deep learning
[course ](https://classroom.udacity.com/courses/ud730) assignment 3.

The code is based on [original](https://github.com/Arn-O/udacity-deep-learning)
with [modification](https://github.com/rdcsung/udacity-deep-learning)

It is about deep neural network.
The technology of regularization, drop-out are implemented in the assignment.

## Normal logistic classification with regularization

The image size is 28 * 28 = 784.
Learning steps 20001.
I have tried different batch size, it looks like 128 is a good batch size.
It is also clear with logistic classification, the accuracy is never reach 90%

Batch size 128

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_1.png)

Batch size 64

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_2.png)

Batch size 32

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_3.png)

Batch size 16

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_4.png)


## Find best regularization
With batch size 128
I tried different regularization parameter. 1e-4, 1e-3.9, 1e-3.8,.....   1e-2.
(step by 0.1 in exponent index)

From the figure below, it shows the best regu parameter is around 1e-3

![regu](https://rdcsung.github.io/static/img/blog/udacity/regu-1.png)

## Neural network implementation
In this example, one hidden layer is built.
The hidden layer has 1024 neuron.
The test accuracy, reaches 94.2%

Batch size 128

![LC](https://rdcsung.github.io/static/img/blog/udacity/lc-nn-1.png)<commit_msg>add nn overfit<commit_after>---
layout: post
title:  "Udacity Deep Learning Part 1"
desc: "Describe the Udacity study of DL , assignment 3"
keywords: "DL"
date: 2016-11-30
categories: [Life]
tags: [blog]
icon: fa-bookmark-o
---

After I installed my new laptop, I continued to study my Udacity deep learning
[course ](https://classroom.udacity.com/courses/ud730) assignment 3.

The code is based on [original](https://github.com/Arn-O/udacity-deep-learning)
with [modification](https://github.com/rdcsung/udacity-deep-learning)

It is about deep neural network.
The technology of regularization, drop-out are implemented in the assignment.

## Normal logistic classification with regularization

The image size is 28 * 28 = 784.
Learning steps 20001.
I have tried different batch size, it looks like 128 is a good batch size.
It is also clear with logistic classification, the accuracy is never reach 90%

Batch size 128

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_1.png)

Batch size 64

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_2.png)

Batch size 32

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_3.png)

Batch size 16

![LC](https://rdcsung.github.io/static/img/blog/udacity/learning_curve_4.png)


## Find best regularization
With batch size 128
I tried different regularization parameter. 1e-4, 1e-3.9, 1e-3.8,.....   1e-2.
(step by 0.1 in exponent index)

From the figure below, it shows the best regu parameter is around 1e-3

![regu](https://rdcsung.github.io/static/img/blog/udacity/regu-1.png)

## Neural network implementation
In this example, one hidden layer is built.
The hidden layer has 1024 neuron.
The test accuracy, reaches 94.2%

Batch size 128

![LC](https://rdcsung.github.io/static/img/blog/udacity/lc-nn-1.png)

## Neural network, overfit
Set the batch size to 3.
The hidden layer has 1024 neuron.
The training data set is too small for the complex network.

Batch size = 3

![LC](https://rdcsung.github.io/static/img/blog/udacity/lc-nn-overfit-1.png)
{'fraction_non_alphanumeric': 0.11737331954498449, 'fraction_numerical': 0.039555325749741466, 'mean_word_length': 4.292749658002736, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 19, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2210151', 'n_tokens_mistral': 1395, 'n_tokens_neox': 1248, 'n_words': 412}