<commit_before>---
layout: post
title:  Automated exploratory data analysis in R
date:   2017-12-12

---

You want to listen to what your data tells you. But that can be hard when you have a lot of things (variables) you're keeping track of. If you measure 10 variables and are interested in what's happening in your dataset, you need to produce at least 10 choose 2 = 45 plots. That'll take some time. If there are 20 variables, creating 190 plots is in your future. But your outlook is improving with the code I'm sharing below.

The goal of the `eda` function is to allow you to quickly get a sense of potential patterns in your dataset that you can then follow up on with a more thorough analysis. 

## what it does

The `eda` function computes some statistic of interest between every pair of columns in your dataframe. The statistic it computes depends on the nature of the variables:

- We calculate the [_correlation coefficient_](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) for two continuous variables. A correlation, or _r_, ranges from -1 to 1. Values near zero are rarely interesting.

- [_Goodman and Kruskal’s tau_](https://cran.r-project.org/web/packages/GoodmanKruskal/vignettes/GoodmanKruskal.html) is calculated if the two variables are both categorical. It can range from 0 (not interesting) to 1 (interesting). 

> Interestingly, tau is not symmetric: tau(x,y) is not necessarily the same as tau(y,x). Why? Think of the relationship between country and continent, two categorical variables. If I know the country, I know with 100% certainty what continent it belongs to. So tau(country, continent) is 1. But if I'm only given the continent and had to predict the country, I'd have a harder time (except for Australia or Antarctica). So tau(continent, country) < 1, for the ability for continent to predict country is less than perfect.

- Finally, for the relationship between a categorical and a continuous variable, we calculate the intraclass correlation coefficient, or [ICC](https://en.wikipedia.org/wiki/Intraclass_correlation). It ranges from 0 (not interesting) to 1 (interesting). 

The `eda` function figures out the appropriate test to use for each pair of variables.

## the input

The input to the `eda` (which stands for 'exploratory data analysis', by the way) is a dataframe. The function assumes that each thing you care about, i.e., each variable, is in its own column. If not, you'll need to `gather` or `spread` with tools from the `tidyr` package. It also assumes that categorical variables are coded as factors in R. You can check this with the `is.factor` function. Character vectors are simply ignored.

## the output 

The output is also a dataframe. The dataframe has six columns:

- var1: the first variable
- var2: the second variable
- statistic: what type of statistic was calculated (_r_, the ICC, or tau).
- value: the value of the statistic
- p_value: p-value for the relationship. Based on permutation tests with only 99 permutations for tau and ICC for speed reasons. Take with a grain of salt or a whole box of Morton's. As explained elsewhere, the purpose of this function is to give you a sense of interesting or strong relationships in your data, not to give you a final result.
- n: the number of observations. This might vary due to missingness in some columns. By default, missing values are removed.

## let's try it out
<commit_msg>add explicit link to function code<commit_after>---
layout: post
title:  Automated exploratory data analysis in R
date:   2017-12-12

---

You want to listen to what your data tells you. But that can be hard when you have a lot of things (variables) you're keeping track of. If you measure 10 variables and are interested in what's happening in your dataset, you need to produce at least 10 choose 2 = 45 plots. That'll take some time. If there are 20 variables, creating 190 plots is in your future. But your outlook is improving with the code I'm sharing below.

The goal of the `eda` function is to allow you to quickly get a sense of potential patterns in your dataset that you can then follow up on with a more thorough analysis. 

## what it does

The `eda` function, the code for which you can find [here](https://github.com/lukereding/random_scripts/blob/master/eda.R), computes some statistic of interest between every pair of columns in your dataframe. The statistic it computes depends on the nature of the variables:

- We calculate the [_correlation coefficient_](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) for two continuous variables. A correlation, or _r_, ranges from -1 to 1. Values near zero are rarely interesting.

- [_Goodman and Kruskal’s tau_](https://cran.r-project.org/web/packages/GoodmanKruskal/vignettes/GoodmanKruskal.html) is calculated if the two variables are both categorical. It can range from 0 (not interesting) to 1 (interesting). 

> Interestingly, tau is not symmetric: tau(x,y) is not necessarily the same as tau(y,x). Why? Think of the relationship between country and continent, two categorical variables. If I know the country, I know with 100% certainty what continent it belongs to. So tau(country, continent) is 1. But if I'm only given the continent and had to predict the country, I'd have a harder time (except for Australia or Antarctica). So tau(continent, country) < 1, for the ability for continent to predict country is less than perfect.

- Finally, for the relationship between a categorical and a continuous variable, we calculate the intraclass correlation coefficient, or [ICC](https://en.wikipedia.org/wiki/Intraclass_correlation). It ranges from 0 (not interesting) to 1 (interesting). 

The `eda` function figures out the appropriate test to use for each pair of variables.

## the input

The input to the `eda` (which stands for 'exploratory data analysis', by the way) is a dataframe. The function assumes that each thing you care about, i.e., each variable, is in its own column. If not, you'll need to `gather` or `spread` with tools from the `tidyr` package. It also assumes that categorical variables are coded as factors in R. You can check this with the `is.factor` function. Character vectors are simply ignored.

## the output 

The output is also a dataframe. The dataframe has six columns:

- var1: the first variable
- var2: the second variable
- statistic: what type of statistic was calculated (_r_, the ICC, or tau).
- value: the value of the statistic
- p_value: p-value for the relationship. Based on permutation tests with only 99 permutations for tau and ICC for speed reasons. Take with a grain of salt or a whole box of Morton's. As explained elsewhere, the purpose of this function is to give you a sense of interesting or strong relationships in your data, not to give you a final result.
- n: the number of observations. This might vary due to missingness in some columns. By default, missing values are removed.

## let's try it out

{'fraction_non_alphanumeric': 0.06064961195745904, 'fraction_numerical': 0.01006036217303823, 'mean_word_length': 4.267978803936412, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1843708', 'n_tokens_mistral': 1886, 'n_tokens_neox': 1700, 'n_words': 1067}