<jupyter_start><jupyter_text>Testing CPI with arrays of dates 10-22-18<jupyter_code>import cpi
import pandas as pd
import numpy as np

from datetime import date<jupyter_output><empty_output><jupyter_text>Using `pandas.util.testing` we can create 2 dataframes with datetime indexes and using `numpy.random.randint()` we can create a set of corresponding _income_ values.We use `freq = 'W'` so we have 100 different dates but ~25 different months as an example of data that could be found _in the wild_.<jupyter_code>first_date = pd.date_range(end = '2018-01-01', periods = 3000, freq = 'W')

second_date = pd.date_range(start = '1930-01-01', periods = 3000, freq = 'W')

incomes = np.random.randint(low = 1500, high = 200_000, size = 3000)<jupyter_output><empty_output><jupyter_text>From there we can construct our working dataframe<jupyter_code>df = pd.DataFrame(columns=['date_from', 'date_to', 'incomes'])
df['date_from'] = second_date
df['date_to'] = first_date
df['incomes'] = incomes

df.head()<jupyter_output><empty_output><jupyter_text>***`CPI` works in a simple fashion:1. Look for `year_or_month` for when the values are from and retrieve its ___source_index___.2. Look for `to` for the ___target_index___ to inflate the values _to_.3. `return (value * target_index) / float(source_index)`The simplicity of the conversion lends `CPI` to be useful when `value` is a pandas series or numpy array. The goal is to be able to provide a pandas series or numpy array with dates as well.Here's an example of a work-around:<jupyter_code># source_index
cpi_values_source = {}

dates_from = df['date_from'].astype(str).str[:7] # 1234-56

for item in dates_from.unique():
    # retrieve all values and store them in a dict()
    y_m = str(item).split("-")
    y_m = [int(y_m[0]), int(y_m[1])] # year and month
    target_date = date(y_m[0], y_m[1], 1)
    cpi_values_source[item] = cpi.get(target_date)
    
# Map those values to another series
source_index = dates_from.map(cpi_values_source)
# targe_index
cpi_values_target = {}

dates_to = df['date_to'].astype(str).str[:7]

for item in dates_to.unique():
    # retrieve all values and store them in a dict()
    y_m = str(item).split("-")
    y_m = [int(y_m[0]), int(y_m[1])] # year and month
    target_date = date(y_m[0], y_m[1], 1)
    cpi_values_target[item] = cpi.get(target_date)
    
# Map those values to another series
target_index = dates_to.map(cpi_values_target)
df['inflated'] = df['incomes'] * target_index / source_index

df.head()<jupyter_output><empty_output><jupyter_text>***The value here is that even if you have 6000 different weekly observations you only have around ($6000 / 4=$) 1500 different months and so you should only call `cpi.get()` 1500 times, not 6000. A much more common case would be to have a set of weekly observations and to _inflate_ them all to the most up-to-date index.<jupyter_code>cpi.update()
cpi.LATEST_MONTH
date_from = pd.date_range(end = '2018-01-01', periods = 3000, freq = 'W')

incomes = np.random.randint(low = 1500, high = 200_000, size = 3000)

df = pd.DataFrame(columns=['date_from', 'date_to', 'incomes'])

df['date_from'] = date_from
df['date_to'] = "2018-09-01"
df['incomes'] = incomes

df.head()<jupyter_output><empty_output><jupyter_text>Currently, the example from `CPI` documentation for working with pandas uses the `.apply()` method.<jupyter_code>df['YEAR'] = df['date_from'].dt.year # prepping for CPI README example
%%timeit
df['ADJUSTED'] = df.apply(lambda x: cpi.inflate(x['incomes'], x['YEAR']), axis=1)
df[['YEAR', 'incomes', 'ADJUSTED']].head()<jupyter_output><empty_output><jupyter_text>Using this _new_ method, not only can we do this for each month (not just year) but it is also a bit faster.<jupyter_code>%%timeit
# source_index
cpi_values_source = {}

dates_from = df['date_from'].astype(str).str[:7] # 1234-56

for item in dates_from.unique():
    # retrieve all values and store them in a dict()
    y_m = str(item).split("-")
    y_m = [int(y_m[0]), int(y_m[1])] # year and month
    target_date = date(y_m[0], y_m[1], 1)
    cpi_values_source[item] = cpi.get(target_date)
    
# Map those values to another series
source_index = dates_from.map(cpi_values_source)
target_index = cpi.get(date(2018,9,1))

df['ADJUSTED_2'] = df['incomes'] * target_index / source_index
df[['date_from', 'incomes', 'ADJUSTED_2']].head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13290705640557204, 'fraction_numerical': 0.03356930806120119, 'mean_word_length': 5.32034632034632, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '39215', 'license': 'MIT', 'n_tokens_mistral': 1671, 'n_tokens_neox': 1556, 'n_words': 443, 'path': 'example.ipynb', 'repo_name': 'chekos/cpi', 'size': '4379'}