<commit_before><commit_msg>+counting emojis<commit_after>---
layout: post
author: csiu
date: 2017-05-07
title: "Day72: Counting emojis"
categories: update
tags:
  - 100daysofcode
excerpt:
---
<style>
table th, table td {
  padding-bottom: 0.6em;
}
tbody {
  font-family: Menlo, Monaco, Consolas, monospace; font-size:0.6em; padding-bottom: 1em;
}
pre.highlight.whitebackground {
  background-color: #FFF;
}
</style>

Now that we can recognize emojis, I can count them using this example:

<center>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Another test: Counting the number of emojis (w/ duplicates) from tweets <br> --.--<br> 🚦<br> 🚗🚗🚌🚕🚒🚕🚕🚚🚚🚗🚑🚗</p>&mdash; <NAME> (@celiassiu) <a href="https://twitter.com/celiassiu/status/861443380076699648">May 8, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</center>

The R Markdown document for this project is found [here](https://gist.github.com/csiu/a20433adb05618bbbd9f89b8f75acbdf/3438f2c028670cb07b1735037197bf30e3ddea3a).

## Counting emojis

The [original code](https://prismoji.com/2017/02/06/emoji-data-science-in-r-tutorial/#part2) appears to be in base R (which is hard to read). From what I can see, `sapply(emojis$rencoding, regexpr, tweets$text, ignore.case = T, useBytes = T)` is the key line used to count emojis from tweets. This code essentially tells R to do pattern matching for each pattern in "emojis$rencoding" to text "tweets$text" with options "ignore.case = T" and "useBytes = T".

**Attempt 1: Using base R**

Following the example, I tried to use `regexpr` to count the number of times an emoji appears in a tweet. The problem, however, is that emojis are only counted once per tweet (irregardless of duplicates). This underrepresentation is not what I want.

**Attempt 2: Word tokenization**

Next I tried tokenizing the tweets with `RWeka::WordTokenizer` as done in [Twimoji: Identifying Emoji in Tweets (<NAME>, 2015)](http://miningthedetails.com/blog/r/IdentifyEmojiInTweets/). The problem, however, is that I am unable to install RWeka and thus am unable to tokenize my tweets.

**Attempt 3: stringr**

Finally the `stringr::str_count` function by <NAME> allow users to "count the number of matches in a string" -- which is what I want to do.

## Counting emoijs the tidy way

I first define a helper function to count the number of times an emoji appears in a tweet (using `stringr::str_count`) and then apply this function (using `purrr::map`) for all emojis.

```r
# Helper function to count number of times pattern occur in string
count_emojis <- function(e){
  counts <- str_count(df_tidy$text, e)
  data.frame(
    counts,
    tweet_id = 1:length(counts)
  )
}
# Do the counting of emojis for each tweet
emoji_counts <-
  emoticons %>%
  select(description, r_encoding) %>%
  mutate(
    counts = purrr::map(r_encoding, ~count_emojis(.x))
  ) %>%
  tidyr::unnest(counts)
```

<div class="highlighter-rouge">
<pre class="highlight whitebackground">
<code>## # A tibble: 2,526 × 4
##       description               r_encoding counts tweet_id
##             &lt;chr&gt;                    &lt;chr&gt;  &lt;int&gt;    &lt;int&gt;
## 1  aerial tramway &lt;ed&gt;&lt;a0&gt;&lt;bd&gt;&lt;ed&gt;&lt;ba&gt;&lt;a1&gt;      0        1
## 2  aerial tramway &lt;ed&gt;&lt;a0&gt;&lt;bd&gt;&lt;ed&gt;&lt;ba&gt;&lt;a1&gt;      0        2
## 3  aerial tramway &lt;ed&gt;&lt;a0&gt;&lt;bd&gt;&lt;ed&gt;&lt;ba&gt;&lt;a1&gt;      0        3
## 4        airplane             &lt;e2&gt;&lt;9c&gt;&lt;88&gt;      0        1
## 5        airplane             &lt;e2&gt;&lt;9c&gt;&lt;88&gt;      0        2
## 6        airplane             &lt;e2&gt;&lt;9c&gt;&lt;88&gt;      0        3
## 7     alarm clock             &lt;e2&gt;&lt;8f&gt;&lt;b0&gt;      0        1
## 8     alarm clock             &lt;e2&gt;&lt;8f&gt;&lt;b0&gt;      0        2
## 9     alarm clock             &lt;e2&gt;&lt;8f&gt;&lt;b0&gt;      0        3
## 10  alien monster &lt;ed&gt;&lt;a0&gt;&lt;bd&gt;&lt;ed&gt;&lt;b1&gt;&lt;be&gt;      0        1
## # ... with 2,516 more rows
</code>
</pre>
</div>

After the count of emojis per tweet is computed, we can next look at total emoiji counts.

```r
emoji_counts %>%
  # Ignore emojis with no counts
  filter(counts != 0) %>%

  # Here I want to only consider the latest tweet
  filter(tweet_id == 1) %>%
  select(-tweet_id) %>%

  # Summarize the counts per emoji
  group_by(description) %>%
  summarise(count = sum(counts)) %>%
  arrange(desc(count))
```

<div class="highlighter-rouge">
<pre class="highlight whitebackground">
<code>## Source: local data frame [7 x 3]
## Groups: description [7]
##
##              description count
##                    &lt;chr&gt; &lt;int&gt;
## 1             automobile     4
## 2                   taxi     3
## 3         delivery truck     2
## 4              ambulance     1
## 5                    bus     1
## 6            fire engine     1
## 7 vertical traffic light     1
</code>
</pre>
</div>

The results check out!

<center>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">Another test: Counting the number of emojis (w/ duplicates) from tweets <br> --.--<br> 🚦<br> 🚗🚗🚌🚕🚒🚕🚕🚚🚚🚗🚑🚗</p>&mdash; <NAME> (@celiassiu) <a href="https://twitter.com/celiassiu/status/861443380076699648">May 8, 2017</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</center>
{'fraction_non_alphanumeric': 0.1500092030185901, 'fraction_numerical': 0.03938891956561752, 'mean_word_length': 3.329880478087649, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 56, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2641369', 'n_tokens_mistral': 2184, 'n_tokens_neox': 1987, 'n_words': 554}