<jupyter_start><jupyter_text>"How to smoothen noisy data and find peaks and dips in a line plot using Python"> "In this small tutorial we will use the U.S. COVID-19 inoculation data to demonstrate the effect of the Savitzky-Golay filter and find the most prominent peaks and dips in daily vaccinations."- toc: false- branch: master- badges: true- comments: true- categories: [python, jupyter]- image: images/Savitzky-Golay-Filter.png- hide: false- search_exclude: false Presenting peaks and dips in a noisy line plot can be a bit of a challenge, as there is a lot of unnecessary visual information. Savitzky-Golay filter is a function that can be applied to such data in order to clarify the points with minimal distortion and precision loss. It was formulated for the exact purpose of finding maxima and minima in curve data by Savitzky themselves {% fn 1 %}. In this small tutorial we will use the U.S. COVID-19 inoculation data to demonstrate the effect of the filter and find the most prominent peaks and dips in daily vaccinations. We will use an interactive widget to tweak the optimal parameters for the filter (click one of the above links for an interactive version).  Step 1: Install the following Python packages<jupyter_code>!pip install widgetsnbextension ipywidgets jupyter-js-widgets-nbextension ipympl<jupyter_output><empty_output><jupyter_text>Step 2: Enable widget support in your Jupyter environment<jupyter_code>!jupyter nbextension enable --py widgetsnbextension --sys-prefix<jupyter_output>Enabling notebook extension jupyter-js-widgets/extension...
      - Validating: [32mOK[0m<jupyter_text>Step 3: Importing the dependenciesWe will use Pandas to read and manipulate the `.csv` file, Matplotlib for plotting the data, `signal` method from the Scipy package to apply the filter, Numpy and `argrelextrema` function to find the "extreme" values in the data, and finally `interactive` to build the necessary sliders.<jupyter_code>from ipywidgets import interactive

import pandas as pd
import matplotlib.pyplot as plt
from scipy import signal
import numpy as np<jupyter_output><empty_output><jupyter_text>Step 4: Import and filter the data by location; we will use a CSV file from Our World in Data.<jupyter_code>df_raw = pd.read_csv("https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv", usecols=["location", "date", "daily_vaccinations"], parse_dates=["date"])

df = df_raw[df_raw["location"] == "United States"]

df.set_index("location", inplace=True, drop=True)

df<jupyter_output><empty_output><jupyter_text>Step 5: Build the function and plot the dataFirst, we assign our X and Y values. We feed the Y values, or the first 444 consecutive days of `daily_vaccinations`, into `signal.savgol_filter()` method. This function requires 2 parameters: `windows_size` and `polyorder`. According to the documentation, `windows_size` is always a positive *odd* integer and `polyorder` is *any* positive integer that is less than `window_length` {% fn 2 %}. Unfortunately my understanding of the concept is very limited, but the general goal is to "[keep] the important features and getting rid of the meaningless fluctuations" {% fn 3 %}. These variables control the smoothness of the curve: too low and the curve will lose the detail, too high -- it will become distorted; the rule of thumb is to start low and build up from that. Because the exact values vary with data, they cannot be known beforehand. Controlling them visually will help us find the optimal curve for our purpose, hence the need for a slider to set the inputs more intuitively.<jupyter_code>def make_iplot(window_size, polyorder):
  data_x = df["date"].values
  data_y = df["daily_vaccinations"].values # original
  data_y_filtered = signal.savgol_filter(data_y, window_size, polyorder) # smoothed

  # Find peaks (np.greater)
  peak_indexes = signal.argrelextrema(data_y_filtered, np.greater)
  peak_indexes = peak_indexes[0]

  # Find valleys (np.less)
  valley_indexes = signal.argrelextrema(data_y_filtered, np.less)
  valley_indexes = valley_indexes[0]

  # Matplotlib plot
  plt.figure(figsize=(20, 5))
  plt.plot(data_x, data_y, color="grey") # line plot for the original data
  plt.plot(data_x, data_y_filtered, color="black") # line plot for the filtered data
  plt.plot(data_x[valley_indexes], data_y_filtered[valley_indexes], "o", label="dip", color="r")
  plt.plot(data_x[peak_indexes], data_y_filtered[peak_indexes], "o", label="peak", color="g")
  plt.show()<jupyter_output><empty_output><jupyter_text>With that said, we arbitrarily set the range of 1 to 100 for the `windows_size` slider, and 1 to 10 for the `polyorder`.<jupyter_code># this line of code makes the figure appear in the output below
%matplotlib inline

iplot = interactive(
  make_iplot,
  window_size=(1,100,2),
  polyorder=(1,10,1)
)

iplot<jupyter_output><empty_output><jupyter_text>![](images/1.png) Setting the initial values of 3 and 1 (`polyorder` < `windows_size`) as starting points gives us a result that isn't dissimilar to the original data (red dots represent dips and green ones -- peaks).<jupyter_code># this line of code makes the figure appear in the output below
%matplotlib inline

iplot = interactive(
  make_iplot,
  window_size=(1,100,2),
  polyorder=(1,10,1)
)

iplot<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08278703877130549, 'fraction_numerical': 0.01011425360554411, 'mean_word_length': 5.4963503649635035, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '479792', 'license': 'Apache-2.0', 'n_tokens_mistral': 1675, 'n_tokens_neox': 1565, 'n_words': 637, 'path': '_notebooks/Savitzky_Golay_Filter.ipynb', 'repo_name': 'sergeyyurkov1/blog', 'size': '5339'}