From Wikipedia I have the compliment of the CDF parameterized for fat-tails distributions.

$$
\Pr[X>x] \sim x^{- \alpha}\text{ as }x \to \infty,\qquad \alpha > 0.\,
$$

Here $\alpha$ is the fatness parameter. According to Taleb. $\alpha \leq 2.5$ is forecastable, but $\alpha > 2.5$ is not.

I would like to fit $\alpha$ given my data so I can mark it as forecastable or not.

I thought I would start by trying to fit my data to a linear model.

```
set.seed(42)
df_tails <- tibble(y = 1- seq(0.01,1, 0.01),
               norm = sort(rnorm(n = 100, 0,1)), 
               cauchy = sort(rcauchy(n = 100, 0,1)))
lm(log(y) ~ norm - 1, data = df_tails)
lm(log(y) ~ cauchy - 1, data = df_tails)

```

The problem is that I end up with many `NAs` so I think I am coding something wrong.

### Try 2

```
library(tidyverse)
set.seed(42)

df_tails_raw <- tibble(y = log(1- seq(0.01,1, 0.01)),
               norm = log(sort(rnorm(n = 100, 0,1))), 
               cauchy = log(sort(rcauchy(n = 100, 0,1))))
df_tails <- na.omit(df_tails_raw)

df_tails |>
  ggplot() +
  geom_point(aes(x = norm, y=y), color = 'tomato', size = 2, stroke = 2, shape = 1) +
  geom_point(aes(x = cauchy, y = y), color = 'grey50', size = 2, stroke = 2, shape = 1) +
  theme_classic() +
  labs('Red is normal and Grey is Cauchy')

lm(y ~ norm, data = df_tails)
lm(y ~ cauchy - 1, data = df_tails)

```

[![enter image description here](https://i.stack.imgur.com/1WjzT.png)](https://i.stack.imgur.com/1WjzT.png)

My error is

> 
> Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) :
> NA/NaN/Inf in 'y'
> 
> 
>
['https://stats.stackexchange.com/questions/597528', 'https://stats.stackexchange.com', 'https://stats.stackexchange.com/users/142914/']
There are several issues with this question.

### The error message

The simplest, is the issue about the error message which is the explicit question in the text.

> 
> Error in lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) :
> NA/NaN/Inf in 'y'
> 
> 
> 

The error says that the dependent variable in the linear model is not right and contains NA/NaN/Inf. The reason is because your $y$ variable contains a zero and when you take the logarithm of this then you get an NA value. Then, when you pass this to the `lm` function you get the error. (Because you pass log(y) nested inside the lm function this is not so clear, but the 'y' of the lm function is your 'log(y)' value and not your 'y' value)

Sidenote: to fit a powerlaw with linearisation you should use $\log(y) = a + b \cdot \log(x)$. In your code you use $\log(y) = bx$ and you miss the intercept as well as taking the logarithm of $x$.

### The fitting of the power law

Distributions that have power law behaviour are often only having this behaviour for a limited range. In your fitting method you should only fit the part of the distribution that follows the power law.

In the log-log plot below you see that you don't have a straight line over the entire range, in addition, the points in the tail are the ones with a large scatter and error. If you plot the points along with the known underlying distribution you see that the error is not just the scatter but also the error is correlated and the entire curve can have an error.

[![example of behaviour on log log scale](https://i.stack.imgur.com/HIXsD.png)](https://i.stack.imgur.com/HIXsD.png)

On the plot I also have added a log-normal distribution. It shows that curves that are not really straight can appear to be straight. Just fitting a straight line does not tell that you also actually have a straight line.

The article mentioned by Sycorax on the comments, [Power-law distributions in empirical data](https://arxiv.org/abs/0706.1062), discusses this issue on more detail.
#### You might want to use the `tailplot` function in the `utilities` package

The standard way of examining tail behaviour of data is through a tail-plot or a Hill plot (or variations of these). The tailplot shows the tails of a dataset against the empirical tail probability, each exhibited on a logarithmic scale. The plot can be generated in `R` by using the [`tailplot` function](https://search.r-project.org/CRAN/refmans/utilities/html/tailplot.html) in the [`utilities` package](https://CRAN.R-project.org/package=utilities). This function allows you to plot the data in one or both tails using a chosen proportion of the dataset (by default the plot will show 5% of the data in each tail) and it will compare this with a specified power-rate of decay (by default it is compared with cubic decay, which determines finiteness of the variance.

In the code below I give an example of a tailplot for a set of $n=1000$ datapoints generated from a standard normal distribution. The plot shows that the tails of the distributin decays substantially faster than cubic decay, which is sufficient to give finite variance. You can compare your data with an alternate rate-of-decay if you prefer.

```
#Set some mock data
set.seed(1)
DATA <- rnorm(1000)

#Show the tail plots
library(utilities)
tailplot(DATA)

```

[![enter image description here](https://i.stack.imgur.com/4yIFi.jpg)](https://i.stack.imgur.com/4yIFi.jpg)

Note that the `tailplot` function also allows you to include a Hill-plot and/or De Sousa-Michailidis plot used for estimating the rate-of-decay of the tails. To include these plots, just set `hill.plot = TRUE` and/or `dsm.plot = TRUE`.