<jupyter_start><jupyter_text>Converting Timestamps to Periods(and Back) **to_period()** method for Series<jupyter_code>rng = pd.date_range('2000-01-01', periods=3, freq='M') # monthly frequency with 3 periods
ts = pd.Series(np.random.randn(3), index=rng)
pts = ts.to_period() # can also be used for DataFrames
pts
ts<jupyter_output><empty_output><jupyter_text>Frequency of timestamp is bydefault inherited by **PeriodIndex**<jupyter_code>rng = pd.date_range('1/28/2020', periods=6, freq='D')
ts2 = pd.Series(np.random.randn(len(rng)), index=rng)
ts2.to_period('M')
ts2
# converting back to timestamps
pts = ts.to_period()
pts
pts.to_timestamp(how='end')<jupyter_output><empty_output><jupyter_text>Converting a PeriodIndex from Arrays<jupyter_code>! wget https://resources.oreilly.com/examples/0636920023784/raw/master/pydata-book-master/ch07/macrodata.csv
! mkdir examples
! mv macrodata.csv examples/macrodata.csv
data = pd.read_csv('examples/macrodata.csv')
data.head()
data.year.unique()
data.quarter.unique()<jupyter_output><empty_output><jupyter_text>We can create index of the DataFrame by passing these arrays to PeriodIndex<jupyter_code>index = pd.PeriodIndex(year = data.year, quarter = data.quarter, freq='Q-DEC')
index
data.index = index
data.infl<jupyter_output><empty_output><jupyter_text>Resampling and Frequency Conversion Method - **resample()****Resampling** - Conversion of one frequency to anotherHigher frequency to lower frequency - **Downsampling**Lower frequency to higher frequency - **Upsampling**<jupyter_code>rng = pd.date_range('2020-01-01', periods=100, freq='D')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts.resample('M').mean() # resample is similar to groupby
ts.resample('M', kind='period').mean()<jupyter_output><empty_output><jupyter_text>Downsampling<jupyter_code>rng = pd.date_range('2000-01-01', periods=12, freq='T')
ts = pd.Series(np.arange(12), index=rng)
ts
# aggregating into five-minute chunks / bars
ts.resample('5min', closed='right').sum()
ts.resample('5min', closed='left').sum()
# default labels are from the left side as in previous cell we have 1999-12-31 23:55:00 as first timestamp
# we can switch to the right edge label using lable option
ts.resample('5min', closed='right', label='right').sum()<jupyter_output><empty_output><jupyter_text>Shifting result index by some **offset** using loffset option<jupyter_code># subtracting 1s from right edge
ts.resample('5min', closed='right', label='right', loffset='-1s').sum()<jupyter_output><empty_output><jupyter_text>Shifting result index by some **offset** using **shift()** method<jupyter_code>ts.resample('5min', closed='right', label='right').sum().shift(-1, freq='S')<jupyter_output><empty_output><jupyter_text>Open-High-Low-Close (OHLC) resampling Aggregate a time series - open, close, high and low using **ohlc()** function<jupyter_code>ts
ts.resample('5min').ohlc()<jupyter_output><empty_output><jupyter_text>Upsampling and Interpolation No aggregation is required when converting from low to high frequency<jupyter_code>frame = pd.DataFrame(np.random.randn(2, 4),\
                     index=pd.date_range('1/1/2020', periods=2, freq='W-WED'),\
                     columns=['Colorado', 'Texas', 'New York', 'Ohio'])
frame<jupyter_output><empty_output><jupyter_text>Missing values will result in gaps (from 2020-01-02 to 2020-01-07)<jupyter_code>df_daily = frame.resample('D').sum()
df_daily<jupyter_output><empty_output><jupyter_text>Fill Forward values on Non-Wednesdays<jupyter_code>frame.resample('D').ffill()<jupyter_output><empty_output><jupyter_text>Fill Forward to certain number of periods<jupyter_code>frame.resample('D').ffill(limit=2)<jupyter_output><empty_output><jupyter_text>New data index need not overlap with the old one<jupyter_code>frame
frame.resample('W-THU').ffill()<jupyter_output><empty_output><jupyter_text>Resampling with Periods<jupyter_code>frame = pd.DataFrame(np.random.randn(24, 4), index = pd.period_range('1-2000', '12-2001', freq='M'),
                     columns=['Colorado', 'Texas', 'New York', 'Ohio'])

frame[:5]
annual_frame = frame.resample('A-DEC').mean()
annual_frame<jupyter_output><empty_output><jupyter_text>Unsampling can be achieved by **convention** argument: **start**, **end**<jupyter_code>#Q-DEC: Quarterly, year ending in December
annual_frame.resample('Q-DEC').ffill()
annual_frame.resample('Q-DEC', convention='start').ffill()
annual_frame.resample('Q-DEC', convention='end').ffill()<jupyter_output><empty_output><jupyter_text>While performing upsampling or downsampling Quarterly, annual and weekly frequencies should be used with appropriate sub period or super period, else it will raise exeption<jupyter_code># since annual frame is defined by A-DEC, we can apply quarterly downsampling easily
annual_frame.resample('Q-MAR').ffill()<jupyter_output><empty_output><jupyter_text>Moving Window Functions<jupyter_code>! wget https://raw.githubusercontent.com/wesm/pydata-book/2nd-edition/examples/stock_px_2.csv
! mv stock_px_2.csv examples/<jupyter_output>--2020-07-04 11:03:58--  https://raw.githubusercontent.com/wesm/pydata-book/2nd-edition/examples/stock_px_2.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 192.168.127.12, 172.16.31.10, 172.16.17.32, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|192.168.127.12|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 101402 (99K) [text/plain]
Saving to: ‘stock_px_2.csv’


stock_px_2.csv        0%[                    ]       0  --.-KB/s               
stock_px_2.csv      100%[===================>]  99.03K  --.-KB/s    in 0.02s   

2020-07-04 11:03:58 (4.83 MB/s) - ‘stock_px_2.csv’ saved [101402/101402]<jupyter_text>**rolling()** behaviour is similar to **resample** and **groupby**[Visualization of expanding window and rolling window](https://miro.medium.com/max/1626/1*fk5NTdtnZ0M7uP7eGWZJ0w.jpeg)<jupyter_code>close_px_all = pd.read_csv('examples/stock_px_2.csv', parse_dates=True, index_col=0)
close_px_all.head()
close_px = close_px_all[['AAPL', 'MSFT', 'XOM']]
close_px
close_px = close_px.resample('B').ffill()
close_px
close_px.AAPL.plot();
# rolling functions require all of the values in the window to be non-NA
close_px.rolling(250).mean().plot();
# we can change non-NA requirements by period option
appl_std250 = close_px.AAPL.rolling(250, min_periods=10).std()
appl_std250[5:12]
appl_std250.plot();<jupyter_output><empty_output><jupyter_text>**expanding()** function[Visualization of expanding window and rolling window](https://miro.medium.com/max/1626/1*fk5NTdtnZ0M7uP7eGWZJ0w.jpeg)<jupyter_code>expanding_mean = appl_std250.expanding().mean()
expanding_mean.plot(title='Apple 250-day daily return standard deviation');
close_px.rolling(60).mean().plot(logy=True, title='stock prices 60-day MA (log Y-axis)');<jupyter_output><empty_output><jupyter_text>Exponentially-weighted functions<jupyter_code>appl_px = close_px.AAPL['2006':'2007'] # filter apple stock price for 2006-07
appl_px.plot()
ma60 = appl_px.rolling(50, min_periods=20).mean() # rolling window (moving average)
ewma60 = appl_px.ewm(span=30).mean() # exponential weighted function (exponential-weighted average)
ma60.plot(style='k--', label='Simple MA');
ewma60.plot(style='k-', label='WE MA');
plt.legend();<jupyter_output><empty_output><jupyter_text>Binary Moving Window Functions<jupyter_code># apple stock's correlation with S&P 500
spx_px = close_px_all['SPX']
spx_rets = spx_px / spx_px.shift(1) - 1
returns = close_px.pct_change()
corr = returns.AAPL.rolling(125, min_periods=100).corr(spx_rets)
corr.plot()<jupyter_output><empty_output><jupyter_text>Compute correlation of a TimeSeries with each column of a DataFrame<jupyter_code>corr = returns.rolling(125, min_periods=100).corr(spx_rets)
corr.plot();<jupyter_output><empty_output><jupyter_text>User-Defined Moving Window Functions Apply more than 1 array function (custom) over a moving window<jupyter_code>#Percentile rank of a particular value while computing rolling().quantile()
from scipy.stats import percentileofscore
score_at_2_percent = lambda x: percentileofscore(x, 0.02)
result = returns.AAPL.rolling(250).apply(score_at_2_percent)
result.plot();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14596235579842137, 'fraction_numerical': 0.04225865209471767, 'mean_word_length': 6.799242424242424, 'pattern_counts': {'":': 0, '<': 96, '<?xml version=': 0, '>': 97, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '313722', 'license': 'MIT', 'n_tokens_mistral': 3183, 'n_tokens_neox': 2904, 'n_words': 567, 'path': 'ch11/ch11.ipynb', 'repo_name': 'PalashSharma15/python-for-data-analysis-2nd-ed-notes', 'size': '8238'}