<jupyter_start><jupyter_text>14.1 Statistical Functions  14.1.3 Correlation<jupyter_code>frame = pd.DataFrame(np.random.randn(1000, 5), columns=['a', 'b', 'c', 'd','e'])
frame.iloc[::2] = np.nan
frame['b'][:10]
frame['a'].corr(frame['b'][:10])<jupyter_output><empty_output><jupyter_text>从这里可以看出corr函数是支持两个长度不相同的serial进行运算的<jupyter_code>frame['a'].corr(frame['b'], method='spearman')
frame.corr()<jupyter_output><empty_output><jupyter_text>使用corr方法需要注意的一点是参与运算的两个对象的index需要由重叠的地方，否则无法计算出结果，下面是写的一个小程序<jupyter_code>index = ['a', 'b', 'c', 'd', 'e']
columns = ['one', 'two', 'three', 'four']
df1 = pd.DataFrame(np.random.randn(5, 4), index=index, columns=columns)
df2 = pd.DataFrame(np.random.randn(4, 4), index=index[:4], columns=columns)
df1.one.corr(df2.one)
df2 = pd.DataFrame(np.random.randn(4, 4), columns=columns)#此时df2的index和df1的index不同
df1.one.corr(df2.one)
df1.one.corrwith(df2.one)<jupyter_output><empty_output><jupyter_text>14.2 Window Functions 对于rolling操作，其实是有一个对应的`Rolling`来进行描述的，这就好比groupby操作类似，会生成`DataFrameGroupBy`类对象； Starting in version 0.18.1, the rolling() and expanding() functions can be used directly fromDataFrameGroupBy objects, see the groupby docs.从这段话可以看出，`Rolling`可以比`DataFrameGroupBy`操作更加细的粒度，即它既可以作用在一个完整的dataframe，也可以作用在一个比dataframe更加细的粒度上；<jupyter_code>%matplotlib inline
from __future__ import division
from numpy.random import randn
import numpy as np
np.set_printoptions(precision=4, suppress=True)
import pandas as pd
s = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000',periods=1000))
s
s = s.cumsum()
s
r = s.rolling(window=60)
r<jupyter_output><empty_output><jupyter_text>我目前比较关注pandas中跟时序相关的内容，在文档中看到了这样的一段描述：>Warning: The freq and how arguments were in the API prior to 0.18.0 changes. These are deprecated in thenew API. You can simply resample the input prior to creating a window function.For example, instead of s.rolling(window=5,freq='D').max() to get the max value on a rolling 5 Daywindow, one could use s.resample('D').max().rolling(window=5).max(), which first resamplesthe data to daily data, then provides a rolling 5 day window.这段话的意思是：从pandas0.18.0版本开始，rolling方法中移除了`freq`和`how`参数，在新版本的API中，它们是不建议使用的；在新版本中如果要实现和原来版本相同的功能，可以通过如下的方式得到：旧版本写法:`s.rolling(window=5,freq='D').max() `新版本写法：`s.resample('D').max().rolling(window=5).max()`<jupyter_code>r.mean()
type(s.index)
s.plot(style='k--')
r.mean().plot(style='k')<jupyter_output><empty_output><jupyter_text>在dataframe中使用windowfunction<jupyter_code>df = pd.DataFrame(np.random.randn(1000, 4),index=pd.date_range('1/1/2000', periods=1000),columns=['A', 'B', 'C', 'D'])
df = df.cumsum()
df.rolling(window=60).sum().plot(subplots=True)<jupyter_output><empty_output><jupyter_text>14.2.1 Method Summary 这一小节介绍了`Rolling`类提供的统计函数，`Rolling`类提供的统计函数都是一些常用的统计函数，那我们如果想要得到一些`Rolling`类提供的统计函数之外的功能要怎样做了？这一小节也对此进行了介绍。要对`Rolling`对象实现自定义的功能，可以通过`apply`来实现<jupyter_code>mad = lambda x: np.fabs(x - x.mean()).mean()
s.rolling(window=60).apply(mad).plot(style='k')<jupyter_output><empty_output><jupyter_text>14.2.2 Rolling Windows 将win_type传递给.rolling会生成一个通用的滚动窗口计算，根据win_type进行加权。关于win_type的解释可以参考下面的这个文章：https://stackoverflow.com/questions/42999175/what-is-the-intuition-behind-win-type-argument-in-rolling-function-of-pandas这跟信号处理相关，我目前还不了解。并且pandas的win_type是参考的scipy的，https://docs.scipy.org/doc/scipy/reference/signal.htmlwindow-functions<jupyter_code>ser = pd.Series(np.random.randn(10), index=pd.date_range('1/1/2000', periods=10))
 ser.rolling(window=5, win_type='triang').mean()
 ser.rolling(window=5, win_type='boxcar').mean()
 ser.rolling(window=5, win_type='gaussian').mean(std=0.1)<jupyter_output><empty_output><jupyter_text>14.2.3 Time-aware Rolling 该功能是在pandas的 version 0.19.0版本中新增的，`rolling`方法新增了一个表示offset的参数 New in version 0.19.0 are the ability to pass an offset (or convertible) to a .rolling() method and have it producevariable sized windows based on the passed time window. For each time point, this includes all preceding valuesoccurring within the indicated time delta. This can be particularly useful for a non-regular time frequency index.<jupyter_code>dft = pd.DataFrame({'B': [0, 1, 2, np.nan, 4]},index=pd.date_range('20130101 09:00:00', periods=5, freq='s'))
dft<jupyter_output><empty_output><jupyter_text>This is a regular frequency index. Using an integer window parameter works to roll along the window frequency.<jupyter_code>dft.rolling(2).sum()
dft.rolling(2, min_periods=1).sum()<jupyter_output><empty_output><jupyter_text>Specifying an offset allows a more intuitive specification of the rolling frequency.<jupyter_code>dft.rolling('2s').sum()<jupyter_output><empty_output><jupyter_text>Using a non-regular, but still monotonic index, rolling with an integer window does not impart any special calculation.<jupyter_code>dft = pd.DataFrame({'B': [0, 1, 2, np.nan, 4]},index = pd.Index([pd.Timestamp('20130101 09:00:00'),
                                                                 pd.Timestamp('20130101 09:00:02'),
                                                                 pd.Timestamp('20130101 09:00:02'),
                                                                 pd.Timestamp('20130101 09:00:05'),
                                                                 pd.Timestamp('20130101 09:00:06')],name='foo'))
dft
 dft.rolling(2).sum()<jupyter_output><empty_output><jupyter_text>Using the time-specification generates variable windows for this sparse data.<jupyter_code>dft.rolling('2s').sum()<jupyter_output><empty_output><jupyter_text>Furthermore, we now allow an optional on parameter to specify a column (rather than the default of the index) in aDataFrame.<jupyter_code>dft = dft.reset_index()
dft
dft.rolling('2s', on='foo').sum()<jupyter_output><empty_output><jupyter_text>14.2.4 Rolling Window Endpoints New in version 0.20.0.The inclusion of the interval endpoints in rolling window calculations can be specified with the closed parameter: For example, having the right endpoint open is useful in many problems that require that there is no contaminationfrom present information back to past information. This allows the rolling window to compute statistics “up to thatpoint in time”, but not including that point in time.这些内容都是和时序相关的；<jupyter_code>df = pd.DataFrame({'x': 1},index = [pd.Timestamp('20130101 09:00:01'),
                                    pd.Timestamp('20130101 09:00:02'),
                                    pd.Timestamp('20130101 09:00:03'),
                                    pd.Timestamp('20130101 09:00:04'),
                                    pd.Timestamp('20130101 09:00:06')])
df            
df["right"] = df.rolling('2s', closed='right').x.sum() # default

df["both"] = df.rolling('2s', closed='both').x.sum()

df["left"] = df.rolling('2s', closed='left').x.sum()
df<jupyter_output><empty_output><jupyter_text>Currently, this feature is only implemented for time-based windows. For fixed windows, the closed parameter cannotbe set and the rolling window will always have both endpoints closed.  14.2.5 Time-aware Rolling vs. Resampling  14.2.7 Binary Window Functions<jupyter_code>df = pd.DataFrame(np.random.randn(1000, 4),index=pd.date_range('1/1/2000', periods=1000),columns=['A', 'B', 'C', 'D'])
df = df.cumsum()
df2 = df[:20]
r=df2.rolling(window=5)
r.corr(df2['B'])
df.corr(df2['B'])<jupyter_output>D:\Program Files\Anaconda3\lib\site-packages\pandas\core\ops.py:1167: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  result = method(y)<jupyter_text>14.2.8 Computing rolling pairwise covariances and correlations<jupyter_code>covs = df[['B','C','D']].rolling(window=50).cov(df[['A','B','C']],pairwise=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14056224899598393, 'fraction_numerical': 0.046379064645679494, 'mean_word_length': 5.374896779521057, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 69, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '410017', 'license': 'Apache-2.0', 'n_tokens_mistral': 3122, 'n_tokens_neox': 2936, 'n_words': 523, 'path': 'CHAPTER 14 COMPUTATIONAL TOOLS.ipynb', 'repo_name': 'dengking/jupyter-notebook-of-pandas-official-doc', 'size': '7719'}