<jupyter_start><jupyter_text>Comparison of pivot building methodsMethods include the following: * `make_pivots` used in the original diffusion frequencies module; this function uses `np.linspace`  which requires the number of data points to create ahead of time instead of spacing between data points. Notably, `linspace` includes the last value in the range by default. Does not allow specification of start and end dates to override the given date observations. * `get_pivots` used in the new modular augur interface (but not provided in the frequency estimators module); this function uses `np.arange` which requires the step size between points defined as a floating point value. `arange` does not include the final value in the range unless there is a floating point overflow. When the step size is not an integer, [this function is known to produce inconsistent results due to floating point overflows](https://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.htmlnumpy.arange). For example, the final value in the output range can be greater than the given stop value. `linspace` is recommended instead. Allows either a start or an end date to override the given date observations. * `calculate_pivots` used in the KDE frequencies class (but only available as a class method); this function uses `pd.date_range` which requires either the frequency of points in the requested range or the number of points, combining the functionality of `arange` and `linspace` into one date-aware function. `date_range` also includes the final value by default and supports closed intervals on the left or right. The `calculate_pivots` method requires both a start and an end date to override the given date observations<jupyter_code>import numpy as np
import pandas as pd

np.random.seed(1)

def float_to_datestring(time):
    """Convert a floating point date to a date string

    >>> float_to_datestring(2010.75)
    '2010-10-01'
    >>> float_to_datestring(2011.25)
    '2011-04-01'
    >>> float
    """
    year = int(time)
    month = int(np.rint(((time - year) * 12) + 1))
    day = 1
    return "%s-%02d-%02d" % (year, month, day)

def timestamp_to_float(time):
    """Convert a pandas timestamp to a floating point date.

    >>> import datetime
    >>> time = datetime.date(2010, 10, 1)
    >>> timestamp_to_float(time)
    2010.75
    >>> time = datetime.date(2011, 4, 1)
    >>> timestamp_to_float(time)
    2011.25
    """
    return time.year + ((time.month - 1) / 12.0)<jupyter_output><empty_output><jupyter_text>Generate some example date observations in floating point space like augur would provide.<jupyter_code>n_samples = 15
observations = np.array(sorted(np.random.choice([2010.0, 2011.0], size=n_samples) +
                               np.random.random(n_samples) * 0.75))
observations[:10]<jupyter_output><empty_output><jupyter_text>Compare calculation of monthly pivots with different methods described above.Start with `linspace` approach from `make_pivots`.<jupyter_code># Determine the range of the given observations.
dt = np.max(observations) - np.min(observations)

# Define number of pivots to match the arange and date_range examples below.
n_pivots = 21

# Calculate pivots with linear spacing between start and end with 1% of the total range
# added to each side of the range.
np.linspace(np.min(observations) - 0.01 * dt, np.max(observations) + 0.01 * dt, n_pivots)<jupyter_output><empty_output><jupyter_text>Next, use `arange` approach from `get_pivots` method.<jupyter_code>min_date = None
max_date = None
pivots_per_year = 12
dt = 1.0 / pivots_per_year

first_pivot = min_date if min_date else np.floor(np.min(observations) / dt) * dt
last_pivot = max_date if max_date else np.ceil(np.max(observations) / dt) * dt
np.arange(first_pivot, last_pivot, dt)
# Floating point step size leads to overflow in arange call above.
dt
first_pivot
last_pivot<jupyter_output><empty_output><jupyter_text>Depending on which observations are included above, the last pivot may or may not be included in the output from `arange` because the step size of `1 / 12.0` leads to a buffer flow.<jupyter_code>min_date = None
max_date = None
pivots_per_year = 4
dt = 1.0 / pivots_per_year

first_pivot = min_date if min_date else np.floor(np.min(observations) / dt) * dt
last_pivot = max_date if max_date else np.ceil(np.max(observations) / dt) * dt
np.arange(first_pivot, last_pivot, dt)
last_pivot<jupyter_output><empty_output><jupyter_text>Now the last pivot value is not included in the output from `arange` because the step size of `1 / 4.0` does not lead to a buffer overflow.Next, try to replicate the above examples with the pandas `date_range` function.<jupyter_code># Use same step size as first arange example above of 1 month between pivots.
pivot_frequency = 1
dt = 1.0 / 12

first_pivot = float_to_datestring(min_date if min_date else np.floor(np.min(observations) / dt) * dt)
last_pivot = float_to_datestring(max_date if max_date else np.ceil(np.max(observations) / dt) * dt)
pd.date_range(first_pivot, last_pivot, freq="%sMS" % pivot_frequency)
first_pivot
last_pivot
# Convert datetime values to floats
np.array([
    timestamp_to_float(timestamp)
    for timestamp in pd.date_range(first_pivot, last_pivot, freq="%sMS" % pivot_frequency)
])
# Use same step size as second arange example above of 3 months between pivots (4 pivots per year).
pivot_frequency = 3
dt = 1.0 / 4

first_pivot = float_to_datestring(min_date if min_date else np.floor(np.min(observations) / dt) * dt)
last_pivot = float_to_datestring(max_date if max_date else np.ceil(np.max(observations) / dt) * dt)
pd.date_range(first_pivot, last_pivot, freq="%sMS" % pivot_frequency)
# Convert datetime values to floats
np.array([
    timestamp_to_float(timestamp)
    for timestamp in pd.date_range(first_pivot, last_pivot, freq="%sMS" % pivot_frequency)
])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0864910790144435, 'fraction_numerical': 0.020050977060322855, 'mean_word_length': 4.787610619469026, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 49, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '473426', 'license': 'MIT', 'n_tokens_mistral': 1938, 'n_tokens_neox': 1825, 'n_words': 661, 'path': 'jupyter/2018-12-17-comparison-of-pivot-building-methods.ipynb', 'repo_name': 'huddlej/notebooks', 'size': '5885'}