<commit_before>---
layout:     post
title:      Understanding Convolutions in Text
date:       2016-11-27 12:31:19
summary:    Understanding Convolutions in for text classification systems
categories: nlp deep learning word-embeddings
---

Hey, we will build a convolutional neural network, but we will understand it through images. CNNs are really powerful, and there is a huge body of research that have used CNNs for a variety of tasks. We will understand CNNs through images.

Let's start with inputs:


![png](/assets/images/conv.001.png)

So the green boxes represent the words or the characters. And the corresponding blue rows represent the representation of the words or the characters. In case of character convolutions, since the number of characters was around 70, including punctuations, numbers and alphabets, this is generally the one-hot representation. In case of words, the blue boxes generally represent dense vectors. These dense vectors can be pre-trained word embeddings or word vectors trained during training.

<!-- <img src="conv/conv.002.png",width=550,height=550> -->

![png](/assets/images/conv.002.png)

Now as you can see, filters or kernels can be of any length. Here is something interesting. In case of images the width and lenght of the kernel can be any size. But the width of the kernel in case of character and word representations is the dimension of the entire word embedding or the entire character representation. This makes sense, why would we want to convolve with the partial representation of the word? :P

Now the filters need to convolve with the input and produce the output. This part is slightly tricky to understand and varies based on things like stride (How much the filter moves every stage?) and the length of the filter. The output of the convolution operation is directly dependent on these two aspects.

<!-- <img src="conv/conv.003.png",width=300,height=300>
<img src="conv/conv.004.png",width=300,height=300>
<img src="conv/conv.005.png",width=300,height=300>
<img src="conv/conv.006.png",width=300,height=300> -->

![png](/assets/images/conv.003.png)
![png](/assets/images/conv.004.png)
![png](/assets/images/conv.005.png)
![png](/assets/images/conv.006.png)

Now, this is where all the interesting bit happens! The convolution is just the multiplication of the weights in the filters and the corresponding representation of the words or characters. Each of the output of the multiplication is then just summed up and it produces one output, shown with the arrow. Thus if the filter would have moved with a stride of 2, then the number of filter outputs would have been different. If the filter length was different, the convolved output would be different. Convince yourself that the filter of the length 4, when convolved, wil just produce 2 outputs.

Now the next key part is just like there can be multiple filter lenghts, there can be multiple filters, so there can be a 100 filters of length 2, a hundred filters of length 4 and so on.
<commit_msg>fixed references<commit_after>---
layout:     post
title:      Understanding Convolutions in Text
date:       2016-11-27 12:31:19
summary:    Understanding Convolutions in for text classification systems
categories: nlp deep learning word-embeddings
---

Hey, we will build a convolutional neural network, but we will understand it through images. CNNs are really powerful, and there is a huge body of research that have used CNNs for a variety of tasks. We will understand CNNs through images.

This post relies on a lot of fantastic work that has already been done on convolutions in text,
and here is a list of papers that delves into the details of most of the content I have described here.

* [Natural Language Processing (almost) from Scratch](https://arxiv.org/abs/1103.0398)
* [Character-level Convolutional Networks for Text
Classification](https://papers.nips.cc/paper/5782-character-level-convolutional-networks-for-text-classification.pdf)
* [Convolutional Neural Networks for Sentence Classification](http://www.aclweb.org/anthology/D14-1181)  


Let's start with inputs:


![png](/assets/images/conv.001.png)

So the green boxes represent the words or the characters. And the corresponding blue rows represent the representation of the words or the characters. In case of character convolutions, since the number of characters was around 70, including punctuations, numbers and alphabets, this is generally the one-hot representation. In case of words, the blue boxes generally represent dense vectors. These dense vectors can be pre-trained word embeddings or word vectors trained during training.

<!-- <img src="conv/conv.002.png",width=550,height=550> -->

![png](/assets/images/conv.002.png)

Now as you can see, filters or kernels can be of any length. Here is something interesting. In case of images the width and lenght of the kernel can be any size. But the width of the kernel in case of character and word representations is the dimension of the entire word embedding or the entire character representation. This makes sense, why would we want to convolve with the partial representation of the word? :P

Now the filters need to convolve with the input and produce the output. This part is slightly tricky to understand and varies based on things like stride (How much the filter moves every stage?) and the length of the filter. The output of the convolution operation is directly dependent on these two aspects.

<!-- <img src="conv/conv.003.png",width=300,height=300>
<img src="conv/conv.004.png",width=300,height=300>
<img src="conv/conv.005.png",width=300,height=300>
<img src="conv/conv.006.png",width=300,height=300> -->

![png](/assets/images/conv.003.png)
![png](/assets/images/conv.004.png)
![png](/assets/images/conv.005.png)
![png](/assets/images/conv.006.png)

Now, this is where all the interesting bit happens! The convolution is just the multiplication of the weights in the filters and the corresponding representation of the words or characters. Each of the output of the multiplication is then just summed up and it produces one output, shown with the arrow. Thus if the filter would have moved with a stride of 2, then the number of filter outputs would have been different. If the filter length was different, the convolved output would be different. Convince yourself that the filter of the length 4, when convolved, wil just produce 2 outputs.

Now the next key part is just like there can be multiple filter lenghts, there can be multiple filters, so there can be a 100 filters of length 2, a hundred filters of length 4 and so on.

{'fraction_non_alphanumeric': 0.06911185207638679, 'fraction_numerical': 0.029099727190057594, 'mean_word_length': 4.545378151260504, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '527149', 'n_tokens_mistral': 1823, 'n_tokens_neox': 1632, 'n_words': 910}