<commit_before>---
layout: post
title: "One and Two-Way Anova"
date: 2017-4-21 00:00:00
excerpt_separator: <!--more-->
---


In the last blog post, I went over different variations of the t-test to
compare means between two groups. When we're trying to compare more than
two groups however, we move on to an analyis of variance test, also
known as ANOVA. ANOVA tests follow the same general procedures as
t-tests, but allows for the user to test for differences in the means of
more than two groups.

<!--more-->

### The Data ###

The example we're going to use is pretty arbitrary, but is easy to
understand. We're going to use ANOVA tests to find out if there is a
significant difference in the mean height of heavy usage players in each
Eastern Conference division. Let's start by using our data scraping
skills to get information from the stats.nba website (remember, this
site has its data in JSON format!).

We'll first bring up the `jsonlite` library and reading the JSON data
into R (I've provided the request url here so you don't have to go
searching for it). se is the Southeast division, cen the Central, and
atl the Atlantic.
```r
library(jsonlite)

se<- fromJSON(readLines("http://stats.nba.com/stats/leaguedashplayerbiostats?
College=&Conference=&Country=&DateFrom=&DateTo=&Division=Southeast&DraftPick=
&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location
=&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PerMode=PerGame&Period=0&Player
Experience=&PlayerPosition=&Season=2016-17&SeasonSegment=&SeasonType=Regular+
Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
&Weight="))

cen<- fromJSON(readLines("http://stats.nba.com/stats/leaguedashplayerbiostats?
College=&Conference=&Country=&DateFrom=&DateTo=&Division=Central&DraftPick=&
DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=
&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PerMode=PerGame&Period=0&Player
Experience=&PlayerPosition=&Season=2016-17&SeasonSegment=&SeasonType=Regular+
Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
&Weight="))

atl<- fromJSON(readLines("http://stats.nba.com/stats/leaguedashplayerbiostats?
College=&Conference=&Country=&DateFrom=&DateTo=&Division=Atlantic&DraftPick=
&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=
&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PerMode=PerGame&Period=0&Player
Experience=&PlayerPosition=&Season=2016-17&SeasonSegment=&SeasonType=Regular+
Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
&Weight="))
```
Next we're going to use `dplyr` to format the data into a dataframe.
We'll be looking at the top 20 highest usage players in each division,
setting a games played bar of at least 58 games. Let's look at the
Southeast division first. I've included notes over each line of the code
block to explain what's happening. Note that I've used the same names as
the list where the JSON data was stored; I only did this for simplicity
and because we won't need the JSON info anymore.
```r
library(dplyr)

se<-
  #storing the data in a dataframe
  data.frame(se$resultSets$rowSet) %>%
  #using the setnames command from the stats package to set headers as the 
  #column names
  stats::setNames(se$resultSets$headers[[1]]) %>%
  #selecting specific columns: name, team, player height, games played, 
  #and usage percentage
  select(PLAYER_NAME, TEAM_ABBREVIATION, PLAYER_HEIGHT_INCHES, GP, 
         USG_PCT) %>%
  #making a factor called division that shows the division
  mutate(DIVISION = factor("SE")) %>%
  #changing player height, games played, and usage percentage to numeric from 
  #a factor
  mutate_each(funs(as.numeric(as.character(.))), c(PLAYER_HEIGHT_INCHES, GP, 
              USG_PCT)) %>%
  #ordering players by usage percentage in descending order
  arrange(desc(USG_PCT)) %>%
  #removing players who've played less than 58 games
  filter(GP>=58) %>%
  #taking the top 20 players based on usage percentage 
  top_n(20, USG_PCT)
```
With an idea of what each line of code means, we can do the same thing
for the Central and Atlantic divisions.
```r
cen<-
  data.frame(cen$resultSets$rowSet) %>%
  stats::setNames(cen$resultSets$headers[[1]]) %>%
  dplyr::select(PLAYER_NAME, TEAM_ABBREVIATION, PLAYER_HEIGHT_INCHES, GP, USG_PCT) %>%
  mutate(DIVISION = factor("CEN")) %>%
  mutate_each(funs(as.numeric(as.character(.))), c(PLAYER_HEIGHT_INCHES, GP, USG_PCT)) %>%
  arrange(desc(USG_PCT)) %>%
  filter(GP>=58) %>%
  top_n(20, USG_PCT)

atl<-
  data.frame(atl$resultSets$rowSet) %>%
  stats::setNames(atl$resultSets$headers[[1]]) %>%
  dplyr::select(PLAYER_NAME, TEAM_ABBREVIATION, PLAYER_HEIGHT_INCHES, GP, USG_PCT) %>%
  mutate(DIVISION = factor("ATL")) %>%
  mutate_each(funs(as.numeric(as.character(.))), c(PLAYER_HEIGHT_INCHES, GP, USG_PCT)) %>%
  arrange(desc(USG_PCT)) %>%
  filter(GP>=58) %>%
  top_n(20, USG_PCT)
```
### One-way ANOVA ###

One-way ANOVA tests look at the difference between groups of one factor.
For example, in the t-test blog post we looked at the difference in game
scores between two players. The one factor was the player, and the
groups were the two players.

The assumptions of normality and homogeneity of variance from t-tests
are again used for ANOVA tests. Let's check out the normality
assumptions with shapiro tests for each division.
```r
shapiro.test(se$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  se$PLAYER_HEIGHT_INCHES
## W = 0.94671, p-value = 0.3199

shapiro.test(atl$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  atl$PLAYER_HEIGHT_INCHES
## W = 0.94906, p-value = 0.353

shapiro.test(cen$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  cen$PLAYER_HEIGHT_INCHES
## W = 0.81839, p-value = 0.001271
```
According to the Shapiro test, we should reject the assumption of
normality for the Central division. Let's check out the normality
assumption graphically with the QQ plot.
```r
qqnorm(se$PLAYER_HEIGHT_INCHES, main = "Southeast")
qqline(se$PLAYER_HEIGHT_INCHES)

qqnorm(cen$PLAYER_HEIGHT_INCHES, main = "Central")
qqline(cen$PLAYER_HEIGHT_INCHES)
```
![](2017-4-21-anova_files/figure-markdown_strict/unnamed-chunk-5-1.png)

The Central division definitely strays a bit from the QQ line,
indicating non-normality. ANOVA is fairly robust against normality, and
because it's only an issue with one group, we are going to move forward
with the example, although we have to take the results with a grain of
salt.

Before we test the homogeneity of variance, we're going to bind each
dataframe into one called east.
```r
east<-
  atl %>%
  bind_rows(cen, se) %>%
  mutate(DIVISION = factor(DIVISION))
```
With a combined dataframe, we're going to use the Bartlett test to test
for homogeneity of variance. This test works similarly as the `var.test`
command in the t-test post, but accepts more than two groups.
```r
bartlett.test(east$PLAYER_HEIGHT_INCHES, east$DIVISION)

## 
##  Bartlett test of homogeneity of variances
## 
## data:  east$PLAYER_HEIGHT_INCHES and east$DIVISION
## Bartlett's K-squared = 1.4408, df = 2, p-value = 0.4865
```
Because the p-value for the Barlett test is above .05, we fail to reject
the null hypothesis and assume that variances are essentially equal.

Let's compare the distribution of heights between divisions graphically
via a boxplot. I'm using the `ggplot2` library for this graph.
```r
library(ggplot2)

ggplot(data=east, aes(x=DIVISION, y=PLAYER_HEIGHT_INCHES)) +
  geom_boxplot(fill="red", alpha=.4)
```
![](2017-4-21-anova_files/figure-markdown_strict/unnamed-chunk-8-1.png)

It looks like there isn't a huge difference between each division, but
let's either confirm or disprove this using the actual ANOVA test.

After going over assumptions, we can move forward with building an ANOVA
model. the R command we'll use to make the model is `aov()`. We'll also
be using R's formula interface. The formula interface looks like `y ~ x`
where y is the dependent and x is the independent variable.

In our example, `PLAYER_HEIGHT_INCHES` is our y and `DIVISION` is our x.
Knowing this, let's make an ANOVA model and store it in `div.anova`.
```r
div.anova<- aov(east$PLAYER_HEIGHT_INCHES ~ east$DIVISION)
```
Now we can use the `anova()` command to run an F-test to tell whether
there is a significant difference in means.
```r
anova(div.anova)

## Analysis of Variance Table
## 
## Response: east$PLAYER_HEIGHT_INCHES
##               Df Sum Sq Mean Sq F value Pr(>F)
## east$DIVISION  2  11.27  5.6327  0.3663 0.6949
## Residuals     58 891.82 15.3761
```
Since the p-value is above .05, we can reject the null hypothesis and
say that there is no significant difference in high-usage player heights
between divisions.

Unfortunately, this just gives us info on whether there is a significant
difference in the means or not; we don't know how the groups differ
between each other. For this, we'd need a multiple comparison test, one
of the most popular of which is the Tukey Honest Significance Test.

To run the Tukey HSD test, we simply run `TukeyHSD()` on our ANOVA
model.
```r
TukeyHSD(div.anova)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = east$PLAYER_HEIGHT_INCHES ~ east$DIVISION)
## 
## $`east$DIVISION`
##               diff       lwr      upr     p adj
## CEN-ATL  0.1166667 -2.830215 3.063549 0.9950136
## SE-ATL  -0.8500000 -3.832603 2.132603 0.7728654
## SE-CEN  -0.9666667 -3.913549 1.980215 0.7112298
```
We can see that there are differences among the means of all divisions,
most notably a 0.97 inch difference between the Central and Southeastern
divisions, but the lower and upper bounds of all confidence intervals
include 0, so none are significant.

### Two-way ANOVA ###

A two-way ANOVA is pretty similar to a one-way ANOVA, it just adds
another grouping factor we're looking at a difference for. We'll
continue with our previous example of differences in mean height between
Eastern Conference divisions, but we'll add in another grouping
variable. We'll add a binary variable indicating whether the player's
team made the playoffs this year. Now we're not only looking for a
difference in average height by division, but also by playoff and
non-playoff teams.

We'll start by creating a vector called `p.teams`. This vector will have
the team abbreviations of all Eastern conference playoff teams. From
there, we'll create a new column in our `east` dataframe by using an
`if_else` command to see whether each row's `TEAM_ABBREVIATION` is in
the `p.teams` vector.
```r
p.teams<- c("BOS", "CLE", "TOR", "WAS", "ATL", "MIL", "IND", "CHI")

east<-
  east %>%
  mutate(PLAYOFF = factor(if_else(TEAM_ABBREVIATION %in% p.teams, "Y", "N")))
```
Next, we should check our assumptions for both the Y and N playoff
groups. First normality (note that for the Shapiro test, we have to
create a subset for each group):
```r
shapiro.test(subset(east, east$PLAYOFF=="Y")$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  subset(east, east$PLAYOFF == "Y")$PLAYER_HEIGHT_INCHES
## W = 0.91307, p-value = 0.0178

shapiro.test(subset(east, east$PLAYOFF=="N")$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## <commit_msg>Update 2017-4-21-anova.md<commit_after>---
layout: post
title: "One and Two-Way Anova"
date: 2017-4-21 00:00:00
excerpt_separator: <!--more-->
---


In the last blog post, I went over different variations of the t-test to
compare means between two groups. When we're trying to compare more than
two groups however, we move on to an analyis of variance test, also
known as ANOVA. ANOVA tests follow the same general procedures as
t-tests, but allows for the user to test for differences in the means of
more than two groups.

<!--more-->

### The Data ###

The example we're going to use is pretty arbitrary, but is easy to
understand. We're going to use ANOVA tests to find out if there is a
significant difference in the mean height of heavy usage players in each
Eastern Conference division. Let's start by using our data scraping
skills to get information from the stats.nba website (remember, this
site has its data in JSON format!).

We'll first bring up the `jsonlite` library and reading the JSON data
into R (I've provided the request url here so you don't have to go
searching for it). se is the Southeast division, cen the Central, and
atl the Atlantic.
```r
library(jsonlite)

se<- fromJSON(readLines("http://stats.nba.com/stats/leaguedashplayerbiostats?
College=&Conference=&Country=&DateFrom=&DateTo=&Division=Southeast&DraftPick=
&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location
=&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PerMode=PerGame&Period=0&Player
Experience=&PlayerPosition=&Season=2016-17&SeasonSegment=&SeasonType=Regular+
Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
&Weight="))

cen<- fromJSON(readLines("http://stats.nba.com/stats/leaguedashplayerbiostats?
College=&Conference=&Country=&DateFrom=&DateTo=&Division=Central&DraftPick=&
DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=
&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PerMode=PerGame&Period=0&Player
Experience=&PlayerPosition=&Season=2016-17&SeasonSegment=&SeasonType=Regular+
Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
&Weight="))

atl<- fromJSON(readLines("http://stats.nba.com/stats/leaguedashplayerbiostats?
College=&Conference=&Country=&DateFrom=&DateTo=&Division=Atlantic&DraftPick=
&DraftYear=&GameScope=&GameSegment=&Height=&LastNGames=0&LeagueID=00&Location=
&Month=0&OpponentTeamID=0&Outcome=&PORound=0&PerMode=PerGame&Period=0&Player
Experience=&PlayerPosition=&Season=2016-17&SeasonSegment=&SeasonType=Regular+
Season&ShotClockRange=&StarterBench=&TeamID=0&VsConference=&VsDivision=
&Weight="))
```
Next we're going to use `dplyr` to format the data into a dataframe.
We'll be looking at the top 20 highest usage players in each division,
setting a games played bar of at least 58 games. Let's look at the
Southeast division first. I've included notes over each line of the code
block to explain what's happening. Note that I've used the same names as
the list where the JSON data was stored; I only did this for simplicity
and because we won't need the JSON info anymore.
```r
library(dplyr)

se<-
  #storing the data in a dataframe
  data.frame(se$resultSets$rowSet) %>%
  #using the setnames command from the stats package to set headers as the 
  #column names
  stats::setNames(se$resultSets$headers[[1]]) %>%
  #selecting specific columns: name, team, player height, games played, 
  #and usage percentage
  select(PLAYER_NAME, TEAM_ABBREVIATION, PLAYER_HEIGHT_INCHES, GP, 
         USG_PCT) %>%
  #making a factor called division that shows the division
  mutate(DIVISION = factor("SE")) %>%
  #changing player height, games played, and usage percentage to numeric from 
  #a factor
  mutate_each(funs(as.numeric(as.character(.))), c(PLAYER_HEIGHT_INCHES, GP, 
              USG_PCT)) %>%
  #ordering players by usage percentage in descending order
  arrange(desc(USG_PCT)) %>%
  #removing players who've played less than 58 games
  filter(GP>=58) %>%
  #taking the top 20 players based on usage percentage 
  top_n(20, USG_PCT)
```
With an idea of what each line of code means, we can do the same thing
for the Central and Atlantic divisions.
```r
cen<-
  data.frame(cen$resultSets$rowSet) %>%
  stats::setNames(cen$resultSets$headers[[1]]) %>%
  select(PLAYER_NAME, TEAM_ABBREVIATION, PLAYER_HEIGHT_INCHES, GP, 
         USG_PCT) %>%
  mutate(DIVISION = factor("CEN")) %>%
  mutate_each(funs(as.numeric(as.character(.))), c(PLAYER_HEIGHT_INCHES, GP, 
              USG_PCT)) %>%
  arrange(desc(USG_PCT)) %>%
  filter(GP>=58) %>%
  top_n(20, USG_PCT)

atl<-
  data.frame(atl$resultSets$rowSet) %>%
  stats::setNames(atl$resultSets$headers[[1]]) %>%
  select(PLAYER_NAME, TEAM_ABBREVIATION, PLAYER_HEIGHT_INCHES, GP, 
         USG_PCT) %>%
  mutate(DIVISION = factor("ATL")) %>%
  mutate_each(funs(as.numeric(as.character(.))), c(PLAYER_HEIGHT_INCHES, GP, 
                   USG_PCT)) %>%
  arrange(desc(USG_PCT)) %>%
  filter(GP>=58) %>%
  top_n(20, USG_PCT)
```
### One-way ANOVA ###

One-way ANOVA tests look at the difference between groups of one factor.
For example, in the t-test blog post we looked at the difference in game
scores between two players. The one factor was the player, and the
groups were the two players.

The assumptions of normality and homogeneity of variance from t-tests
are again used for ANOVA tests. Let's check out the normality
assumptions with shapiro tests for each division.
```r
shapiro.test(se$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  se$PLAYER_HEIGHT_INCHES
## W = 0.94671, p-value = 0.3199

shapiro.test(atl$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  atl$PLAYER_HEIGHT_INCHES
## W = 0.94906, p-value = 0.353

shapiro.test(cen$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  cen$PLAYER_HEIGHT_INCHES
## W = 0.81839, p-value = 0.001271
```
According to the Shapiro test, we should reject the assumption of
normality for the Central division. Let's check out the normality
assumption graphically with the QQ plot.
```r
qqnorm(se$PLAYER_HEIGHT_INCHES, main = "Southeast")
qqline(se$PLAYER_HEIGHT_INCHES)

qqnorm(cen$PLAYER_HEIGHT_INCHES, main = "Central")
qqline(cen$PLAYER_HEIGHT_INCHES)
```
![](2017-4-21-anova_files/figure-markdown_strict/unnamed-chunk-5-1.png)

The Central division definitely strays a bit from the QQ line,
indicating non-normality. ANOVA is fairly robust against normality, and
because it's only an issue with one group, we are going to move forward
with the example, although we have to take the results with a grain of
salt.

Before we test the homogeneity of variance, we're going to bind each
dataframe into one called east.
```r
east<-
  atl %>%
  bind_rows(cen, se) %>%
  mutate(DIVISION = factor(DIVISION))
```
With a combined dataframe, we're going to use the Bartlett test to test
for homogeneity of variance. This test works similarly as the `var.test`
command in the t-test post, but accepts more than two groups.
```r
bartlett.test(east$PLAYER_HEIGHT_INCHES, east$DIVISION)

## 
##  Bartlett test of homogeneity of variances
## 
## data:  east$PLAYER_HEIGHT_INCHES and east$DIVISION
## Bartlett's K-squared = 1.4408, df = 2, p-value = 0.4865
```
Because the p-value for the Barlett test is above .05, we fail to reject
the null hypothesis and assume that variances are essentially equal.

Let's compare the distribution of heights between divisions graphically
via a boxplot. I'm using the `ggplot2` library for this graph.
```r
library(ggplot2)

ggplot(data=east, aes(x=DIVISION, y=PLAYER_HEIGHT_INCHES)) +
  geom_boxplot(fill="red", alpha=.4)
```
![](2017-4-21-anova_files/figure-markdown_strict/unnamed-chunk-8-1.png)

It looks like there isn't a huge difference between each division, but
let's either confirm or disprove this using the actual ANOVA test.

After going over assumptions, we can move forward with building an ANOVA
model. the R command we'll use to make the model is `aov()`. We'll also
be using R's formula interface. The formula interface looks like `y ~ x`
where y is the dependent and x is the independent variable.

In our example, `PLAYER_HEIGHT_INCHES` is our y and `DIVISION` is our x.
Knowing this, let's make an ANOVA model and store it in `div.anova`.
```r
div.anova<- aov(east$PLAYER_HEIGHT_INCHES ~ east$DIVISION)
```
Now we can use the `anova()` command to run an F-test to tell whether
there is a significant difference in means.
```r
anova(div.anova)

## Analysis of Variance Table
## 
## Response: east$PLAYER_HEIGHT_INCHES
##               Df Sum Sq Mean Sq F value Pr(>F)
## east$DIVISION  2  11.27  5.6327  0.3663 0.6949
## Residuals     58 891.82 15.3761
```
Since the p-value is above .05, we can reject the null hypothesis and
say that there is no significant difference in high-usage player heights
between divisions.

Unfortunately, this just gives us info on whether there is a significant
difference in the means or not; we don't know how the groups differ
between each other. For this, we'd need a multiple comparison test, one
of the most popular of which is the Tukey Honest Significance Test.

To run the Tukey HSD test, we simply run `TukeyHSD()` on our ANOVA
model.
```r
TukeyHSD(div.anova)

##   Tukey multiple comparisons of means
##     95% family-wise confidence level
## 
## Fit: aov(formula = east$PLAYER_HEIGHT_INCHES ~ east$DIVISION)
## 
## $`east$DIVISION`
##               diff       lwr      upr     p adj
## CEN-ATL  0.1166667 -2.830215 3.063549 0.9950136
## SE-ATL  -0.8500000 -3.832603 2.132603 0.7728654
## SE-CEN  -0.9666667 -3.913549 1.980215 0.7112298
```
We can see that there are differences among the means of all divisions,
most notably a 0.97 inch difference between the Central and Southeastern
divisions, but the lower and upper bounds of all confidence intervals
include 0, so none are significant.

### Two-way ANOVA ###

A two-way ANOVA is pretty similar to a one-way ANOVA, it just adds
another grouping factor we're looking at a difference for. We'll
continue with our previous example of differences in mean height between
Eastern Conference divisions, but we'll add in another grouping
variable. We'll add a binary variable indicating whether the player's
team made the playoffs this year. Now we're not only looking for a
difference in average height by division, but also by playoff and
non-playoff teams.

We'll start by creating a vector called `p.teams`. This vector will have
the team abbreviations of all Eastern conference playoff teams. From
there, we'll create a new column in our `east` dataframe by using an
`if_else` command to see whether each row's `TEAM_ABBREVIATION` is in
the `p.teams` vector.
```r
p.teams<- c("BOS", "CLE", "TOR", "WAS", "ATL", "MIL", "IND", "CHI")

east<-
  east %>%
  mutate(PLAYOFF = factor(if_else(TEAM_ABBREVIATION %in% p.teams, 
                                  "Y", "N")))
```
Next, we should check our assumptions for both the Y and N playoff
groups. First normality (note that for the Shapiro test, we have to
create a subset for each group):
```r
shapiro.test(subset(east, east$PLAYOFF=="Y")$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
## data:  subset(east, east$PLAYOFF == "Y")$PLAYER_HEIGHT_INCHES
## W = 0.91307, p-value = 0.0178

shapiro.test(subset(east, east$PLAYOFF=="N")$PLAYER_HEIGHT_INCHES)

## 
##  Shapiro-Wilk normality test
## 
{'fraction_non_alphanumeric': 0.10652193098579119, 'fraction_numerical': 0.025725884740976084, 'mean_word_length': 4.957676130389064, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3710708', 'n_tokens_mistral': 8299, 'n_tokens_neox': 7225, 'n_words': 2591}