<jupyter_start><jupyter_text>Descriptive Statistics`R` has plenty of functions to describe data quantitatively and visually. Before we continue, let's set the seed since we are sampling.<jupyter_code>set.seed(37)<jupyter_output><empty_output><jupyter_text>Summarization  Summarization for data structuresThe `summary` function may be used for `vectors`, `factors`, `matrices` and `data frames`.<jupyter_code>x <- rnorm(1000, mean=10, sd=2)

print(summary(x))
x <- factor(sample(c('water', 'soda', 'tea', 'coffee'), 1000, replace=TRUE))

print(summary(x))
A <- matrix(rnorm(1000), ncol=2)

print(summary(A))
df <- data.frame(
    V1 <- rnorm(500),
    V2 <- rnorm(500)
)

print(summary(df))<jupyter_output>V1....rnorm.500.  V2....rnorm.500.  
 Min.   :-2.7513   Min.   :-2.80647  
 1st Qu.:-0.6653   1st Qu.:-0.64585  
 Median : 0.1058   Median : 0.03351  
 Mean   : 0.0221   Mean   : 0.09724  
 3rd Qu.: 0.6595   3rd Qu.: 0.83527  
 Max.   : 3.4937   Max.   : 3.32944<jupyter_text>Scalar value summarizationThe following functions may be used to produce a scalar value summarization for `vectors`.* `min` returns the smallest number* `max` returns the largest number* `length` returns the length of the vector (number of elements)* `mean` returns the average of the elements in the vector* `sd` returns the standard deviation of the elements in the vector* `var` returns the variance of the elements in the vector* `mad` returns the mean absolute deviation of the elements in the vector<jupyter_code>library(purrr)

x <- rnorm(1000, mean=10, sd=2)

xMin <- min(x)
xMax <- max(x)
xLen <- length(x)
xSum <- sum(x)
xMean <- mean(x)
xMed <- median(x)
xSd <- sd(x)
xVar <- var(x)
xMad <- mad(x)

v1 <- c('min', 'max', 'length', 'sum', 'mean', 'median', 'standard deviation', 'variance', 'mean absolute deviation')
v2 <- c(xMin, xMax, xLen, xSum, xMean, xMed, xSd, xVar, xMax)

for (item in map2(v1, v2, function(x, y) paste(x, y))) {
    print(item)
}<jupyter_output>[1] "min 3.72279465941585"
[1] "max 16.3291609049194"
[1] "length 1000"
[1] "sum 10013.9733547688"
[1] "mean 10.0139733547688"
[1] "median 9.93479341285127"
[1] "standard deviation 2.06130538657392"
[1] "variance 4.24897989671867"
[1] "mean absolute deviation 16.3291609049194"<jupyter_text>Some of these functions may also be applied to data frames.<jupyter_code>df <- data.frame(
    V1 <- rnorm(500),
    V2 <- rnorm(500)
)

xMin <- min(df)
xMax <- max(df)
xLen <- length(df)
xSum <- sum(df)

v1 <- c('min', 'max', 'length', 'sum')
v2 <- c(xMin, xMax, xLen, xSum)

for (item in map2(v1, v2, function(x, y) paste(x, y))) {
    print(item)
}<jupyter_output>[1] "min -2.85010220013358"
[1] "max 3.83156099100195"
[1] "length 2"
[1] "sum 56.1376545619577"<jupyter_text>Summarization with multiple resultsThe `quantile` and `fivenum` functions will return summary with multiple results.<jupyter_code>x <- rnorm(100, mean=15, sd=3)

xQuant <- quantile(x)
print(xQuant)
xFive <- fivenum(x)
print(xFive)<jupyter_output>[1]  6.51585 12.82177 14.39959 16.67352 26.07596<jupyter_text>Row and column summarization There are row and column summarization functions for matrices and data frames.<jupyter_code>df <- data.frame(
    V1 <- c(1, 2, 3, 4, 5),
    V2 <- c(6, 7, 8, 9, 10)
)

print(rowMeans(df))
print(rowSums(df))
print(colMeans(df))
print(colSums(df))<jupyter_output>[1] 3.5 4.5 5.5 6.5 7.5
[1]  7  9 11 13 15
 V1....c.1..2..3..4..5. V2....c.6..7..8..9..10. 
                      3                       8 
 V1....c.1..2..3..4..5. V2....c.6..7..8..9..10. 
                     15                      40<jupyter_text>You may also use the `apply` method to produce summaries. The second parameter is either `1` or `2` for rows or columns, respectively.<jupyter_code>print(apply(df, 1, sd))
print(apply(df, 2, sd))<jupyter_output>[1] 3.535534 3.535534 3.535534 3.535534 3.535534
 V1....c.1..2..3..4..5. V2....c.6..7..8..9..10. 
               1.581139                1.581139<jupyter_text>CummulativeYou may apply the following functions to find the cummulative value* `cumsum` computes the cummulative sum* `cummax` computes the cummulative max* `cummin` computes the commulative minimum* `cummax` computes the commulative maximum<jupyter_code>x <- seq(1, 5)

print(cumsum(x))
x <- seq(1, 5)

print(cummax(x))
x <- seq(1, 5)

print(cummin(x))
x <- seq(1, 5)

print(cumprod(x))<jupyter_output>[1]   1   2   6  24 120<jupyter_text>TablesThe `table` command builds a contingency table. When `table` is used on a vector of numbers, a sort frequency distribution appears.<jupyter_code>x <- c(5, 4, 4, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 1)
y <- table(x)
print(y)<jupyter_output>x
1 2 3 4 5 
5 4 3 2 1<jupyter_text>The `table` function may also be used on a vector of characters.<jupyter_code>x <- c('hi', 'hi', 'hi', 'bye', 'bye')
y <- table(x)
print(y)<jupyter_output>x
bye  hi 
  2   3<jupyter_text>Here's the `table` command used on a matrix.<jupyter_code>s <- sample(seq(1:10), 100, replace=TRUE)
m <- matrix(s, ncol=5)
print(m)
y <- table(m)
print(y)<jupyter_output>m
 1  2  3  4  5  6  7  8  9 10 
10 11  9  6  8 12 10  7 13 14<jupyter_text>You may subset a matrix and then apply `table`.<jupyter_code>y <- table(m[,1], m[,2], dnn=c('V1', 'V2'))
print(y)<jupyter_output>V2
V1   1 2 3 5 6 7 8 9 10
  1  0 0 0 0 0 0 0 0  1
  3  1 0 0 0 0 0 0 0  1
  5  0 0 0 0 1 0 0 1  0
  6  2 0 0 1 0 0 0 0  1
  7  0 0 1 0 0 0 0 0  0
  8  0 0 0 1 0 0 0 0  0
  9  1 1 2 0 1 1 0 0  0
  10 1 0 0 0 0 1 1 0  0<jupyter_text>The following shows using `table` on a table with numeric and character data.<jupyter_code>df <- data.frame(
    x <- c(7, 7, 7, 7, 7, 8, 8, 8, 8),
    y <- c('w', 'w', 'w', 'l', 'l', 'l', 'l', 'l', 'w')
)

y <- table(df, dnn=c('x', 'y'))
print(y)<jupyter_output>y
x   l w
  7 2 3
  8 3 1<jupyter_text>The following shows using `table` on a table with numeric data.<jupyter_code>df <- data.frame(
    x <- c(7, 7, 7, 7, 8, 8, 8, 8),
    y <- c(1, 1, 2, 3, 1, 2, 3, 4)
)

y <- table(df, dnn=c('x', 'y'))
print(y)<jupyter_output>y
x   1 2 3 4
  7 2 1 1 0
  8 1 1 1 1<jupyter_text>Use the `with` command to avoid using `dnn` with the `table` function.<jupyter_code>with(df, table(x, y))<jupyter_output><empty_output><jupyter_text>Stem<jupyter_code>x <- sample(seq(1:10), 100, replace=TRUE)
s <- stem(x)
print(s)<jupyter_output>The decimal point is at the |

   1 | 0000
   2 | 00000000000000
   3 | 0000
   4 | 00000000000
   5 | 000000000000
   6 | 00000000000
   7 | 0000000000000
   8 | 0000000000000
   9 | 000000000
  10 | 000000000

NULL<jupyter_text>Compare the `stem` to the `table` function.<jupyter_code>t <- table(x)
print(t)<jupyter_output>x
 1  2  3  4  5  6  7  8  9 10 
 4 14  4 11 12 11 13 13  9  9<jupyter_text>You may control the `scaling` of the `stem` function.  Histogram<jupyter_code>x <- sample(seq(1:10), 100, replace=TRUE)

options(repr.plot.width=4, repr.plot.height=4)

hist(x, col='gray75')<jupyter_output><empty_output><jupyter_text>Density<jupyter_code>x <- sample(seq(1:10), 100, replace=TRUE)
d <- density(x, bw='nrd0', kernel='gaussian', na.rm=FALSE)
print(d)
options(repr.plot.width=4, repr.plot.height=4)

plot(d, main='Density')
options(repr.plot.width=4, repr.plot.height=4)

hist(x, freq=F, col='gray85')
lines(density(x), lty=2)
lines(density(x, k='rectangular'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15718008503634617, 'fraction_numerical': 0.11973666163763544, 'mean_word_length': 3.621039290240811, 'pattern_counts': {'":': 0, '<': 121, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 8, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '489839', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3648, 'n_tokens_neox': 2986, 'n_words': 889, 'path': 'sphinx/r-intro/source/descriptive.ipynb', 'repo_name': 'oneoffcoder/books', 'size': '7291'}