<commit_before>---
layout: post
title: On Approximating Functions or Fitting Models
date: 2016-12-19
---
## On Approximating Functions or Fitting Models

### A short essay on the topic with exercises at the bottom

Here's the situation: you have a dataset of pairs $(x_i,y_i)$ where $i$ is an indexing variable, assumed to be an integer and $0\le i\le n$ and where  $n$ is the total number of datapoints.

In math, there are two ways to define a function $g$. The first way is to write an expression for $g(x)$ and the second is to write down the set total set of pairs $(x,y)$ such that $y=g(x)$. Because the set of points for real-valued functions $\text{(}$functions defined on the real numbers $\text{)}$ would be impossible to write down because there are an uncountably infinite number of such points, we prefer to define functions with the former option.

Now when we have a dataset of points, we can imagine that there is a function $F(x)$ and these points come from the set of pairs $(x,y)$ such that $y=F(x)$. However, knowing just the datapoints is useless to us if we want to know the value of the function for some $x$ not included in the data. Our challenge is to develop a method for figuring out what $F$ is.

Let's take a moment to think about how we'll solve this problem.  We need to reconstruct the rule version of the function $F$ from its datapoints.

To start, we make an informed assumption about the symbolic structure of $F$ – whether it is linear, polynomial, etc. I.e., you choose the structure of the function you think best describes the data. Some functions will be of the format $f(x)=Ce^x$, others of the form $f(x) = Ax^2 + Bx + C$, and yet others of the form $f(x) = m\ln x + b$.

This symbolic structure will be formally defined as a model:



**Definition 1.1**

> A **model** is a function $f$ with parameters $w_1, ..., w_n$ and independent variable $x$ declared as $$y = f(x,w_1, ..., w_n)$$


Once we have made an educated guess as to the structure of our model, we will want to find the best set of parameters to our model. We will call this endeavor "*fitting a model*"

**Definition 1.2**

> We say we are trying to **fit** a model  $f$ on a set of data points of some unknown function $F$ $\{(x_i, y_i)\}$,  if our goal is to best approximate $F$ with $f$

That is, we want to know what the best parameters are so that $f$ is as close as possible to $F$. But we are still groping around in the dark because our definition of "approximate" is ambiguous


**Definition 1.3**

> We say we are trying to approximate a function $F$ with a model $f$ if we are trying to find the set of parameters $\{w\}$ to $f$ such that for any $x_i$ in our dataset the value $\hat{y_i}=f(x_i, w_1, ..., w_n) $ is as close as possible  to the actual value of the function at $x_i$ which is $F(x) = y_i$.

What do we mean by making a model "as close as possible" to a function.

We define some function which takes in the actual value of the function and our guess and spits our a number representing how bad our guess is. Let us say that this is called a distance function defined as $d(\hat{y}, y)$. Formally, it takes as input an actual value of the function at some $x$ value, $F(x)=y$ and a value of the function predicted by our model $f(x)=\hat{y}$, and spits out a number. This number will be how far  $\hat{y}$ is from $y$.

Our goal will be to find the set of parameters to our model so that we get our distance $e$ as low as possible across the entire dataset.

The most naive way to approximate $F$ is to just keep on making random guesses for the set of parameters to the model $f$. So if we have $q$ number of parameters, we just keep on find $q$ random numbers and feed them as parameters to $f$ and choose the set which minimizes

Of course this method can work on the first try, but it is exceedingly unlikely it actually will. We really need an algorithm for finding the set of parameters to our model.

It will have the following flavor:

1. Start with some set of parameters
2. Determine how much error there is with these parameters
3. Change each parameter an amount proportional to how much they contribute to the error
4. Keep on doing this until the error converges on some minimum.

## Getting Specific

Here's where we are now:

1. We have a symbolic model which we think will be a good approximator to our function
2. We need to **fit** this model by making its guesses as good as possible.

We shall say that getting our guesses to be as good as possible means finding the parameters which make $e$ as low as possible over the entire dataset. For our purposes, we will define an error function which takes the values of $D$ for all actual values of the function and predicted values of the function and spits out a number representing the error of our model. We will need to get this error as little as possible.

If we take $D$ to be defined as the square of the distance between the actual value of the function and that value predicted by our model

$$
D(\hat{y}, y) = \left(y-\hat{y}\right)^2
$$
Then we can define the  error function called $E$. We need to get the value of $E$ as low as possible. Taking

$$E = \frac{1}{n}\sum^m_{j=0}d(\hat{y}_j, y_j)$$

Which becomes, when we expand $D$ and remember that
 $\hat{y_i} = f(x_i, w_1, ..., w_n)$

$$
E=\frac{1}{m}\sum_{j=0}^{j=m}{ \left(y_j-\hat{y_j} \right)^2  }<commit_msg>updates to regression<commit_after>---
layout: post
title: On Approximating Functions or Fitting Models
date: 2016-12-19
---
## On Approximating Functions or Fitting Models

### A short essay on the topic with exercises at the bottom

Here's the situation: you have a dataset of pairs $(x_i,y_i)$ where $i$ is an indexing variable, assumed to be an integer and $0\le i\le n$ and where  $n$ is the total number of datapoints.

In math, there are two ways to define a function $g$. The first way is to write an expression for $g(x)$ and the second is to write down the set total set of pairs $(x,y)$ such that $y=g(x)$. Because the set of points for real-valued functions $\text{(}$functions defined on the real numbers $\text{)}$ would be impossible to write down because there are an uncountably infinite number of such points, we prefer to define functions with the former option.

Now when we have a dataset of points, we can imagine that there is a function $F(x)$ and these points come from the set of pairs $(x,y)$ such that $y=F(x)$. However, knowing just the datapoints is useless to us if we want to know the value of the function for some $x$ not included in the data. Our challenge is to develop a method for figuring out what $F$ is.

Let's take a moment to think about how we'll solve this problem.  We need to reconstruct the rule version of the function $F$ from its datapoints.

To start, we make an informed assumption about the symbolic structure of $F$ – whether it is linear, polynomial, etc. I.e., you choose the structure of the function you think best describes the data. Some functions will be of the format $f(x)=Ce^x$, others of the form $f(x) = Ax^2 + Bx + C$, and yet others of the form $f(x) = m\ln x + b$.

This symbolic structure will be formally defined as a model:



**Definition 1.1**

> A **model** is a function $f$ with parameters $w_1, ..., w_n$ and independent variable $x$ declared as $$y = f(x,w_1, ..., w_n)$$


Once we have made an educated guess as to the structure of our model, we will want to find the best set of parameters to our model. We will call this endeavor "*fitting a model*"

**Definition 1.2**

> We say we are trying to **fit** a model  $f$ on a set of data points of some unknown function $F$ $\{(x_i, y_i)\}$,  if our goal is to best approximate $F$ with $f$

That is, we want to know what the best parameters are so that $f$ is as close as possible to $F$. But we are still groping around in the dark because our definition of "approximate" is ambiguous


**Definition 1.3**

> We say we are trying to approximate a function $F$ with a model $f$ if we are trying to find the set of parameters $\{w\}$ to $f$ such that for any $x_i$ in our dataset the value $\hat{y_i}=f(x_i, w_1, ..., w_n) $ is as close as possible  to the actual value of the function at $x_i$ which is $F(x) = y_i$.

What do we mean by making a model "as close as possible" to a function.

We define some function which takes in the actual value of the function and our guess and spits our a number representing how bad our guess is. Let us say that this is called a distance function defined as $D(\hat{y}, y)$. Formally, it takes as input an actual value of the function at some $x$ value, $F(x)=y$ and a value of the function predicted by our model $f(x)=\hat{y}$, and spits out a number. This number will be how far  $\hat{y}$ is from $y$.

Our goal will be to find the set of parameters to our model so that we get our distance $e$ as low as possible across the entire dataset.

The most naive way to approximate $F$ is to just keep on making random guesses for the set of parameters to the model $f$. So if we have $q$ number of parameters, we just keep on find $q$ random numbers and feed them as parameters to $f$ and choose the set which minimizes

Of course this method can work on the first try, but it is exceedingly unlikely it actually will. We really need an algorithm for finding the set of parameters to our model.

It will have the following flavor:

1. Start with some set of parameters
2. Determine how much error there is with these parameters
3. Change each parameter an amount proportional to how much they contribute to the error
4. Keep on doing this until the error converges on some minimum.

## Getting Specific

Here's where we are now:

1. We have a symbolic model which we think will be a good approximator to our function
2. We need to **fit** this model by making its guesses as good as possible.

We shall say that getting our guesses to be as good as possible means finding the parameters which make the distance between the actual value of the function and the one our model spits out as low as possible over the entire dataset. We shall denote this distance by the function $D$. For our purposes, we will define an error function which takes the values of $D$ for all actual values of the function and predicted values of the function and spits out a number representing the error of our model. We will need to get this error as little as possible.

If we take $D$ to be defined as the square of the distance between the actual value of the function and that value predicted by our model

$$
D(\hat{y}, y) = \left(y-\hat{y}\right)^2
$$

Then we can define the  error function called $E$. We need to get the value of $E$ as low as possible. Taking

$$E = \frac{1}{n}\sum^m_{j=0}d(\hat{y}_j, y_j)$$

Which becomes, when we expand $D$ and remember that
 $\hat{y_i} = f(x_i, w_1, ..., w_n)$

$$
E=\frac{1}{m}\sum_{j=0}^{j=m}{ \left(y_j-\hat{y_j} \right)^2  }
{'fraction_non_alphanumeric': 0.07877842993396919, 'fraction_numerical': 0.00586940572267058, 'mean_word_length': 3.8748323647742513, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7062985', 'n_tokens_mistral': 3103, 'n_tokens_neox': 3012, 'n_words': 1914}