<jupyter_start><jupyter_text>Asymptotics===========Asymptotic theory is concerned about the behavior of statistics when thesample size is arbitrarily large. It is a useful approximation techniqueto simplify complicated finite-sample analysis.Modes of Convergence--------------------Convergence of a deterministic sequence $z_1, z_2,\ldots$ means that for any$\varepsilon>0$, there exists an $N\left(\varepsilon\right)$ such thatfor all $n>N\left(\varepsilon\right)$, we have$\left|z_{n}-z\right|<\varepsilon$. We say $z$ is the limit of $z_{n}$,and write as $z_{n}\to z$.In contrast to the convergence of a deterministic sequence, we areinterested in the convergence of random variables. Since a randomvariable has a probability measure associated to any event, we must be clear what *convergence* means.Several modes of convergence are often encountered.-   Convergence almost surely\*-   Convergence in probability:    $\lim_{n\to\infty}P\left(\omega:\left|Z_{n}\left(\omega\right)-z\right|<\varepsilon\right)=1$    for any $\varepsilon>0$.-   Squared-mean convergence:    $\lim_{n\to\infty}E\left[\left(z_{n}-z\right)^{2}\right]=0.$**Example**: $(z_{n})$ is a sequence of binary random variables: $z_{n}=\sqrt{n}$ with probability$1/n$, and $z_{n}=0$ with probability $1-1/n$. Then$z_{n}\stackrel{p}{\to}0$ but $z_{n}\stackrel{m.s.}{\nrightarrow}0$.Convergence in probability does not count what happens on a subset inthe sample space of small probability. Squared-mean convergence dealswith the average over the entire probability space. If a random variablecan take a wild value, even with small probability, it may blow away thesquared-mean convergence. On the contrary, such irregularity does notundermine convergence in probability.-   Convergence in distribution: $x_{n}\stackrel{d}{\to}x$ if    $F\left(x_{n}\right)\to F\left(x\right)$ for each $x$ on which    $F\left(x\right)$ is continuous.Convergence in distribution is about *pointwise* convergence of CDF, notthe random variables themselves.**Example**: Let $x\sim N\left(0,1\right)$. If $z_{n}=x+1/n$, then$z_{n}\stackrel{p}{\to}x$ and of course $z_{n}\stackrel{d}{\to}x$.However, if $z_{n}=-x+1/n$, or $z_{n}=y+1/n$ where$y\sim N\left(0,1\right)$ is independent of $x$, then$z_{n}\stackrel{d}{\to}x$ but $z_{n}\stackrel{p}{\nrightarrow}x$.Squared-mean convergence implies convergence in probability. Convergence in probability implies convergence in distribution.*Cramér-Wold device* handles convergence in distribution for randomvectors. We say a sequence of $K$-dimensional random vectors.$\left(X_{n}\right)$ converge in distribution to $X$ if$\lambda'X_{n}\stackrel{d}{\to}\lambda'X$ for any$\lambda\in\mathbb{R}^{K}$. Law of Large Numbers------------------------(Weak) law of large numbers (LLN) is a collection of statements aboutconvergence in probability of the sample average to its populationcounterpart. The basic form of LLN is:$$\frac{1}{n}\sum_{i=1}^{n} (z_{i}-E[z_i])\stackrel{p}{\to}0$$as $n\to\infty$. Various versions of LLN work under differentassumptions about the distributions and dependence of the randomvariables.-   Chebyshev LLN: if $\left(z_{1},\ldots,z_{n}\right)$ is a sample of    i.i.d. observations, $E\left[z_{1}\right]=\mu$ , and    $\sigma^{2}=\mathrm{var}\left[x_{1}\right]<\infty$ exists, then    $\frac{1}{n}\sum_{i=1}^{n}z_{i}-\mu\stackrel{p}{\to}0.$Chebyshev LLN utilizes Chebyshev inequality.-   *Chebyshev inequality*: for any random variable $x$ , we have    $P\left(\left|x\right|>\varepsilon\right)\leq E\left[x^{2}\right]/\varepsilon^{2}$    for any $\varepsilon>0$, if $E\left[x^{2}\right]<\infty$.Chebyshev inequality is a special case of Markov inequality.-   *Markov inequality*:    $P\left(\left|x\right|>\varepsilon\right)\leq E\left[\left|x\right|^{r}\right]/\varepsilon^{r}$    for $r\geq1$ and any $\varepsilon>0$, if    $E\left[\left|x\right|^{r}\right]<\infty$.It is easy to verify Markov inequality. $$\begin{aligned}E\left[\left|x\right|^{r}\right] & =\int_{\left|x\right|>\varepsilon}\left|x\right|^{r}dF_{X}+\int_{\left|x\right|\leq\varepsilon}\left|x\right|^{r}dF_{X}\\ & \geq\int_{\left|x\right|>\varepsilon}\left|x\right|^{r}dF_{X}\geq\varepsilon^{r}\int_{\left|x\right|>\varepsilon}dF_{X}=\varepsilon^{r}P\left(\left|x\right|>\varepsilon\right).\end{aligned}$$Consider a partial sum $S_{n}=\sum_{i=1}^{n}x_{i}$, where$\mu_{i}=E\left[x_{i}\right]$ and$\sigma_{i}^{2}=\mathrm{var}\left[x_{i}\right]$. We apply the Chebyshevinequality to the sample mean$\overline{x}-\bar{\mu}=n^{-1}\left(S_{n}-E\left[S_{n}\right]\right)$.$$\begin{aligned}P\left(\left|\bar{x}-\bar{\mu}\right|\geq\varepsilon\right) & =P\left(\left|S_{n}-E\left[S_{n}\right]\right|\geq n\varepsilon\right)\\ & \leq\left(n\varepsilon\right)^{-2}E\left[\sum_{i=1}^{n}\left(x_{i}-\mu_{i}\right)^{2}\right]\\ & =\left(n\varepsilon\right)^{-2}\mathrm{var}\left(\sum_{i=1}^{n}x_{i}\right)\\ & =\left(n\varepsilon\right)^{-2}\left[\sum_{i=1}^{n}\mathrm{var}\left(x_{i}\right)+\sum_{i=1}^{n}\sum_{j\neq i}\mathrm{cov}\left(x_{i},x_{j}\right)\right].\end{aligned}$$From the above derivation, convergence in probability holds as long asthe right-hand side shrinks to 0 as $n\to\infty$. Actually, theconvergence can be maintained under much more general conditions thanjust under the i.i.d. assumption. The random variables in the sample donot have to be identically distributed, and they do not have to beindependent either.Another useful LLN is *Kolmogorov LLN*. Since its derivation requiresadvanced knowledge of probability theory, we state the result without proof.-   Kolmogorov LLN: if $\left(z_{1},\ldots,z_{n}\right)$ is a sample of    i.i.d. observations and $E\left[z_{1}\right]=\mu$ exists, then    $\frac{1}{n}\sum_{i=1}^{n}z_{i}-\mu\stackrel{p}{\to}0.$Compared to Chebyshev LLN, Kolmogorov LLN only requires the existence ofthe population mean, but not any higher moment. On the other hand,i.i.d. is essential for Kolmogorov LLN. **Simulation Example**We demonstrate the law of large numbers (LLN) along with the underlying assumptions.Write a function to generate the sample mean given the sample size $n$ and the distribution.We allow three distributions, namely, $N(0,1)$, $t(2)$ and Cauchy.<jupyter_code>sample_mean = function( n, distribution ){
  if (distribution == "normal"){ y = rnorm( n ) } 
  else if (distribution == "t2") {y = rt(n, 2) }
  else if (distribution == "cauchy") {y = rcauchy(n) }
  return( mean(y) )
}
# This function plots the sample mean over the path of geometrically increasing sample size.
library(repr)
options(repr.plot.width=6, repr.plot.height=5)


LLN_plot = function(distribution){
  
  y_bar = rep(0, length(NN) )
  
  for ( i in 1:length(NN)){
    n = NN[i]
    y_bar[i] = sample_mean(n, distribution)
  }
  
  plot(y_bar, type = "l", col = "red", ylab = "mean", xlab = "", lwd = 2, main = distribution)
  abline(h = 0, lty = 2)
  return(y_bar)
}<jupyter_output><empty_output><jupyter_text>The sample size is chosen as $2^x$, where $x=1:20$. Notice that $2^{20}=1.05\times  10^6$ is quite a large sample for economic datasets.  We have the following observations.* When the distribution is $N(0,1)$, the Chebyshev LLN works. The sample mean converges fast.* When the distribution is $t(2)$, which has zero mean but infinite variance, the Kolmogorov LLN works. The sample mean still converges, though more slowly than the $N(0,1)$ case.* The Cauchy distribution has no moment at any order. The sample mean does not converge no matter how large is the sample size.<jupyter_code>NN = 2^(1:20)
set.seed(888) # can remove this line

l1 = LLN_plot("normal")
l2 = LLN_plot("t2")
l3 = LLN_plot("cauchy")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1714509394572025, 'fraction_numerical': 0.012265135699373696, 'mean_word_length': 4.964980544747082, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '182216', 'license': 'MIT', 'n_tokens_mistral': 2557, 'n_tokens_neox': 2425, 'n_words': 725, 'path': 'lec_notes_ipynb/lecture4.ipynb', 'repo_name': 'fatjudy/Econ5121A', 'size': '7664'}