<commit_before>---
layout: post
title: White Noise is Pretty Weird
tags: [White Noise, Probability, Random Variables, Stochastic Process]
---
<script> 
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-82391879-1', 'auto');
  ga('send', 'pageview');

</script>


<!-- Useful book: http://ft-sipil.unila.ac.id/dbooks/AN%20INTRODUCTION%20TO%20STOCHASTIC%20DIFFERENTIAL%20EQUATIONS%20VERSION%201.2.pdf
 -->
I recently went off on a tangent trying to figure out how white noise works, and I found that there is a lot of strangeness to it that may not be apparent at a first glance.

TLDR: We can't just define white noise as an $$\mathbb{R}$$-indexed collection of uncorrelated standard normal random variables because such a collection does not exist. 

## The Problem With White Noise

Let's start with a few simple definitions. In the following we will assume we are working over the well-behaved probability space $$\mathcal{P} = ([0,1], \mathcal{B}, \mu)$$, where $$\mu$$ is the Lebesgue measure on the Borel $$\sigma$$-algebra $$\mathcal{B}$$. Now, a **real-valued stochastic process** $$X$$ is a random-variable valued function such that $$X_t$$ is a real-valued random variable, or a measurable function from $$\mathcal{P}$$ to $$\mathbb{R}$$. We can think of $$t$$ as representing time, but this does not need to be the case.

A **stationary** random process is a stochastic process whose unconditional joint probability distribution does not change when shifted in $$t$$. That is, for any $$$\tau \in \mathbb{R}$$ and $$t_1, ..., t_n \in \mathbb{R}$$ we have that the joint distributions of the sets of random variables $$(X_{t_1}, ..., X_{t_n})$$ and $$(X_{t_1 + \tau}, ..., X_{t_n + \tau})$$ are the same. 

White noise is often defined as a stationary stochastic process where the mean of all $$X_t$$ is $$0$$, the variance of all $$X_t$$ is $$\sigma^2$$, and for all $$\tau$$ we have that $$E[X(t)]E[X(t+\tau)]$$ is $$\sigma^2$$ when $$\tau=0$$ and $$0$$ otherwise.

That is, for all $$t_1,t_2$$, the random variables $$X_{t_1}$$ and $$X_{t_2}$$ are uncorrelated. Some authors strengthen this definition to say that the random variables $$X_{t_1}$$ and $$X_{t_2}$$ are independent. For simplicity, we will assume that $$X_{t} = \mathcal{N}(0,1)$$, and that our white noise process is then a $$\mathbb{R}$$-indexed collection of uncorrelated standard normal random variables.

However, such a collection can not exist! To see this, let's define the collection of random variables $$Y_t = X_t * 1_{\|X_t\| \leq 1}$$. Then we have that $$Y_t$$ is square integrable, and therefore in $$L^2([0,1], \mu)$$. However, $$L^2([0,1], \mu)$$ is separable, and can therefore only countain countably many mutually orthogonal elements. This implies that not all $$X_t$$ can be mutually orthogonal.

## Working around the Problem

To resolve this, we need to use some pretty beefy mathematical machinery. Basically, while we can't define white noise to be a random variable-valued function over $$t$$, we can define it as a random variable-valued generalized function

To start, let's define a *Brownian Motion Process* $$\mathcal{B}$$ to be a stochastic process that satisfies:
<commit_msg>new post<commit_after>---
layout: post
title: White Noise is Pretty Weird
tags: [White Noise, Probability, Random Variables, Stochastic Process]
---
<script> 
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-82391879-1', 'auto');
  ga('send', 'pageview');

</script>


<!-- Useful book: http://ft-sipil.unila.ac.id/dbooks/AN%20INTRODUCTION%20TO%20STOCHASTIC%20DIFFERENTIAL%20EQUATIONS%20VERSION%201.2.pdf
 -->
I recently went off on a tangent trying to figure out how white noise works, and I found that there is a lot of strangeness to it that may not be apparent at a first glance.

**TLDR**: We can't just define white noise as an $$\mathbb{R}$$-indexed collection of uncorrelated standard normal random variables because such a collection does not exist.

## The Problem With White Noise

Let's start with a few simple definitions. In the following we will assume we are working over the well-behaved probability space $$\mathcal{P} = ([0,1], \mathcal{B}, \mu)$$, where $$\mu$$ is the Lebesgue measure on the Borel $$\sigma$$-algebra $$\mathcal{B}$$. Now, a **real-valued stochastic process** $$X$$ is a random-variable valued function such that $$X_t$$ is a real-valued random variable, or a measurable function from $$\mathcal{P}$$ to $$\mathbb{R}$$. We can think of $$t$$ as representing time, but this does not need to be the case.

A **stationary** random process is a stochastic process whose unconditional joint probability distribution does not change when shifted in $$t$$. That is, for any $$$\tau \in \mathbb{R}$$ and $$t_1, ..., t_n \in \mathbb{R}$$ we have that the joint distributions of the sets of random variables $$(X_{t_1}, ..., X_{t_n})$$ and $$(X_{t_1 + \tau}, ..., X_{t_n + \tau})$$ are the same. 

White noise is often defined as a stationary stochastic process where the mean of all $$X_t$$ is $$0$$, the variance of all $$X_t$$ is $$\sigma^2$$, and for all $$\tau$$ we have that $$E[X(t)]E[X(t+\tau)]$$ is $$\sigma^2$$ when $$\tau=0$$ and $$0$$ otherwise.

That is, for all $$t_1,t_2$$, the random variables $$X_{t_1}$$ and $$X_{t_2}$$ are uncorrelated. Some authors strengthen this definition to say that the random variables $$X_{t_1}$$ and $$X_{t_2}$$ are independent. For simplicity, we will assume that $$X_{t} = \mathcal{N}(0,1)$$, and that our white noise process is then a $$\mathbb{R}$$-indexed collection of uncorrelated standard normal random variables.

However, such a collection can not exist! To see this, let's define the collection of random variables $$Y_t = X_t * 1_{\|X_t\| \leq 1}$$. Then we have that $$Y_t$$ is square integrable, and therefore in $$L^2([0,1], \mu)$$. However, $$L^2([0,1], \mu)$$ is separable, and can therefore only countain countably many mutually orthogonal elements. This implies that not all $$X_t$$ can be mutually orthogonal.

## Working around the Problem

To resolve this, we need to use some pretty beefy mathematical machinery. Basically, while we can't define white noise to be a random variable-valued function over $$t$$, we can define it as a random variable-valued generalized function

To start, let's define a *Brownian Motion Process* $$\mathcal{B}$$ to be a stochastic process that satisfies:

{'fraction_non_alphanumeric': 0.1511060040218328, 'fraction_numerical': 0.015512783682849756, 'mean_word_length': 4.895850973751059, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2340726', 'n_tokens_mistral': 2248, 'n_tokens_neox': 2082, 'n_words': 885}