<jupyter_start><jupyter_text>Date and Time data Types and ToolsThe Python standard library includes data types for date and time data, as well as calendar-related functionality.The `datetime.time`, and `calendar` modules are the main places to start.<jupyter_code>from datetime import datetime
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt 
now = datetime.now()
now
print(now.year, now.month, now.day)<jupyter_output>2018 7 24<jupyter_text>`datetime` store both the date and time down to the microsecond.`datetime.timedelta` represents the temproal different two datetime objects.<jupyter_code>delta = datetime(2011, 1, 7) - datetime(2008, 6, 24, 8, 15)
delta
print("different day: %s, different second: %s" % (delta.days, delta.seconds))<jupyter_output>different day: 926, different second: 56700<jupyter_text>`timedelta`You can add(or subtract) a `timedelta` or multiple thereof to a `datetime` object to yield a new shifted object. Types in datetime moduleTypes         | Description                                                                                | --------------|:------------------------------------------------------------------------------------------:|**`date`**        | Store calendar date(year, month, day) using the Gregorian calendar                         |**`time`**        | Store time of days as hours, minutesm and microseconds                                     |**`datetime`**    | Stroes  both date and time                                                                 |**`timedelta`**   | Represents the difference between two datetime values(as days, secondsm and microsenconds) |<jupyter_code>from datetime import timedelta
start = datetime(2011, 1, 7)
start + timedelta(days = 3, hours = 3, minutes = 3, seconds = 12)
start - 2 * timedelta(days = 12)<jupyter_output><empty_output><jupyter_text>Convert between string and  datetime`datetime` objects and pandas `Timestamp` objects, which I'll introduce later, can be ***formatted as string using `str` or the `strftime` method***, passing a format specification.`strftime`(string format output time)***These same format codes can be used to convert strings to dates using `datetime.strptime`(new datetime parsed from a string).***<jupyter_code>stamp = datetime(2013, 1, 3)
stamp
stamp.strftime("%Y-%m-%d")
value = '2011-01-03'
datetime.strptime(value, "%Y-%m-%d")
datestr = ["7/6/2011", '8/6/2011']
[datetime.strptime(x, "%m/%d/%Y") for x in datestr]<jupyter_output><empty_output><jupyter_text>***`datetime.strptime` is the best way to parse a date with a known format.***However, it can be a bit annoying to have to write a format spec each time, especially date formats.***You can use the `parser.parse` method in the third party `dateutil` package.******`dateutil` is capable of parsing almost an human-intelligible date represention.***<jupyter_code>from dateutil.parser import parse
parse("2011-01-03")
parse('Jan 31, 1997 10:45 PM')<jupyter_output><empty_output><jupyter_text>In international locales, day appearing before month is very common, so you can pass `dayfirst = True` to indicate this.<jupyter_code>parse("6/12/2011", dayfirst = True)<jupyter_output><empty_output><jupyter_text>pandas is generally oriented toward working with arrays of dates, whether used as an axis index or a column in a DataFrame.***The `to_datetime` method parses many different kinds of date representayions.***<jupyter_code>import pandas as pd
datestr = ["7/6/2011", '8/6/2011']
pd.to_datetime(datestr)<jupyter_output><empty_output><jupyter_text>***`dateutil.parser.parse` method get `datetime` object, then use `strftime` method output specified format.****<jupyter_code>[parse(date).strftime("%Y-%m-%d") for date in datestr]<jupyter_output><empty_output><jupyter_text>***Using `parse` method can parse string of the diffent datetime format.***<jupyter_code>temp = ["12/6/2016", "2016-08-06"]
[parse(date).strftime("%Y-%m-%d") for date in temp]<jupyter_output><empty_output><jupyter_text>***`NaT`*** (Not a Time) is pandas's NA value for timestamp data.<jupyter_code>idx = pd.to_datetime(datestr + [None])
idx
print(idx[2]) 
print(pd.isnull(idx))<jupyter_output>NaT
[False False  True]<jupyter_text>Time Series BasicsThe most basic kind of time series object in pandas is a Series indexed by timestamps, which is often represented external to pandas as Python strings or `datetime` object.<jupyter_code>dates = [datetime(2011, 1, 2), datetime(2011, 1, 5), datetime(2011, 1, 7),
         datetime(2011, 1, 8), datetime(2011, 1, 10), datetime(2011, 1, 12)]
ts = pd.Series(data = np.random.randn(6), index = dates)
ts[-3:]
ts.index<jupyter_output><empty_output><jupyter_text>***Like other Series, arithmetic operations between differently-indexed time series automatically align on the dates.***A timestamp can be substituted anywhere you would use a `datetime` object.***Additionally, it can store frequency information(if any) and understands how time zone conversions and other kinds of maniplations.***<jupyter_code>print(ts)
ts + ts[::2]
stamp = ts.index[0]
stamp<jupyter_output><empty_output><jupyter_text>Indexing, Selection, Subseting***As a convenience, you can also pass a string that is interpretable as a date.***For longer time series, a year or only a year and month can be passed to easily select of data.<jupyter_code>stamp = ts.index[2]
ts[stamp]
print(ts['1/10/2011'])
print(ts['20110110'])
longer_ts = pd.Series(data = np.random.randn(1000), 
                      index = pd.date_range(start = '1/1/2000', periods = 1000))
print(len(longer_ts))
longer_ts[-5:]
# You can just pass year or month to slice with data.
print('Select 2001 :', longer_ts['2001'][-3:])
print("-"*50)
print('Select 2001-5:', longer_ts['2001-05'][-3:])<jupyter_output>Select 2001 : 2001-12-29    0.241518
2001-12-30   -0.598224
2001-12-31   -1.191834
Freq: D, dtype: float64
--------------------------------------------------
Select 2001-5: 2001-05-29    1.306188
2001-05-30   -0.326928
2001-05-31    0.496326
Freq: D, dtype: float64<jupyter_text>***Slice with dataset works just like a regular Series.***<jupyter_code>longer_ts[(longer_ts.index >= datetime(2000, 2, 1)) & (longer_ts.index <= datetime(2000, 2, 27))][-5:]
longer_ts[longer_ts.index < '2000-01-15'][-3:]
ts['1/6/2011':'1/11/2011']<jupyter_output><empty_output><jupyter_text>As before you can pass either a string data, datetime, ot Timestamp.Remember that slicing this manner prodeces views on the source time series just like slicing NumPy array.There is an equivalient instance method `truncate` which slices a TimeSeries between two dates.<jupyter_code>ts.truncate(after = '2011-01-09')
dates = pd.date_range(start = "1/1/2000", periods = 1000, freq = "W-WED")
long_df = pd.DataFrame(data = np.random.randn(1000, 4), 
                       columns=['Colorado', 'Texas', 'New York', 'Ohio'],
                       index = dates)
long_df['5-2001']<jupyter_output><empty_output><jupyter_text>Time Series with Duplicat IndicesIn some applications, there may be multiple data observations falling on a particular timestamp.We can tell that the index isn't unique by checking its `is_unique` property.<jupyter_code>dates = pd.DatetimeIndex(data = ['1/1/2000', '1/2/2000', '1/2/2000', 
                                 '1/2/2000', '1/3/2000'])
dup_ts = pd.Series(data = np.arange(5), index = dates)
dup_ts
dup_ts.index.is_unique<jupyter_output><empty_output><jupyter_text>***Indexing into this time series will now either produce scalar values or slices depending on whether a timestamp is duplicated.******Suppose you wanted to aggregate the data having non-unique timestamp. One way to do this is to use `groupby` and pass `level = 0`(the only level of indexing)***<jupyter_code>print("not duplicated:", dup_ts['1/3/2000'])
print('-'*50)
print("duplicated:", dup_ts['1/2/2000'])
grouped = dup_ts.groupby(level = 0)
grouped.count()<jupyter_output><empty_output><jupyter_text>Data Ranges, Frequencies, and ShiftingGeneric time series in pandas are assumed to be irregular, they have no fixed frequency.It often desirable to work relative to fixed frequency, duch as daily, monthly or every 15 minutes, even if that means introducing missing values into time series.Fortunately pandas has a full suite of standard time series frequencies and tools for resampling, inferring frequencies, and generating fix frequency data range.***Converting it to be fixed daily frequency can be a accomplished by calling `resample`.***<jupyter_code>dates = [datetime(2011, 1, 2), datetime(2011, 1, 5), datetime(2011, 1, 7),
         datetime(2011, 1, 8), datetime(2011, 1, 10), datetime(2011, 1, 12)]

ts = pd.Series(data = np.random.randn(6), index = dates)
ts
ts.resample('1D').asfreq()[:5]<jupyter_output><empty_output><jupyter_text>Generating Date Ranges`pd.date_range` generates daily timestamps.If you pass only a start or end date, you must pass a number of periods to generate.The start and end dates defind strict boundaries for the generated date index.<jupyter_code>pd.date_range(start = "4/1/2012", periods = 20)
pd.date_range(end = '6/1/2012', periods = 20)
pd.date_range(start = '1/1/2000', end = '12/1/2000', freq = 'BM')<jupyter_output><empty_output><jupyter_text>***Sometimes you will have start or end dates with time information but want to generate a set of timestamps `normalized` to midnight as a convention.***<jupyter_code>pd.date_range('5/2/2012 12:56:31', periods = 5, normalize = True)<jupyter_output><empty_output><jupyter_text>Frequencies and Data OffsetsFrequencies in pandas are composed of a base frequency and multiplier.Base frequencies are typically refferred to by a sting alias, like `M` for monthly or `H` for hourly.For each base frequency, there is an object defind generally referred to  as a `data offset`.<jupyter_code>from pandas.tseries.offsets import Hour, Minute
hour = Hour(n = 1)
hour<jupyter_output><empty_output><jupyter_text>In most applications, you would never need to explicitly create one of these objects, instead using string alias like "H" or "4H".Similary, you can pass frequency like `2h30min` whcih will effectively be parsrd to the same expression.<jupyter_code>pd.date_range(start = '1/1/2000', end = '1/3/2000', freq = '4H')
Hour(2) + Minute(30)
pd.date_range("1/1/2000", periods = 10, freq = 'MS') + Hour(15)<jupyter_output><empty_output><jupyter_text>***Once useful frequency class is `week of month`, starting with `WOM`.***This enbles you to get dates like the third Friday of each month.<jupyter_code># WOM = week of month
rng = pd.date_range("1/1/2012", "9/1/2012", freq = 'WOM-3FRI')
rng<jupyter_output><empty_output><jupyter_text>Shifting (Leading and Lagging) Data***`Shifting` refers to moving data backward and forward through time.***Both `Series` and `DataFrame` have a `shift` method for doing naive shifts forward or backward, leaving the index unmodified.<jupyter_code>ts = pd.Series(data = np.random.randn(4), 
               index = pd.date_range("1/1/2000", periods = 4, freq = 'M'))
ts<jupyter_output><empty_output><jupyter_text>***A common use of `shift` is computing percent changes in a time series or multiple time series as `DataFrame` columns.***<jupyter_code>ts.shift(periods = 1)
ts/ts.shift(1) -1<jupyter_output><empty_output><jupyter_text>***Because naive shifts leave the index unmodified, some data is discarded.***Thus if the frequency is known, it can be passed to `shift` to advance the timestamps instead of simply the data.Other frequencies can be passed, too, giving you a lot of  flexibility in how to lead and lag the data.<jupyter_code>ts.shift(periods = 2, freq = 'M')
ts.shift(3, freq = '3D')<jupyter_output><empty_output><jupyter_text>Shifting dates with offsetsThe pandas date offsets can also be used with `datetime` or `timestamp` objects.<jupyter_code>from pandas.tseries.offsets import Day, MonthEnd
now = datetime.now()
now + 3*Day(n = 1)<jupyter_output><empty_output><jupyter_text>If you add an anchored offset like `MonthEnd`, the first increment will roll forward a date to the next date accroding to the frequency rule.***Anchored offset can explicitly `roll` dates forward or backward using their `rollforward` and `rollback` methods.***<jupyter_code>now + MonthEnd(1)
offset = MonthEnd(n = 3)
print("rollforward:", offset.rollforward(now))
print("-"*50)
print('rollback :', offset.rollback(now))<jupyter_output>rollforward: 2018-07-31 22:34:57.496476
--------------------------------------------------
rollback : 2018-06-30 22:34:57.496476<jupyter_text>***A clever use of date offsets is to use these method with `groupby`.******An easier and faster way to do this is using `resample`.***<jupyter_code>ts = pd.Series(data = np.random.randn(20), 
               index = pd.date_range('1/15/2000', periods = 20, freq = "4d"))
ts[-10:]
print(ts[ts.index < datetime(2000, 2, 1)].mean())
print(ts[ts.index < "2/1/2000"].mean())
print(ts[ts.index < "2000/2/1"].mean())
ts.groupby(offest.rollforward).mean()
ts.resample("M").mean()<jupyter_output><empty_output><jupyter_text>Time Zone Handling<jupyter_code>import pytz
pytz.common_timezones[-5:]
tz = pytz.timezone(zone = 'Asia/Taipei')
tz<jupyter_output><empty_output><jupyter_text>Localization and Conversion`date_range` can be generated with a time zone set.`tz_convert` are also instance method on `DateIndex`.<jupyter_code>rng = pd.date_range('3/9/2012 9:30', periods = 6, freq = 'D', tz = tz)
ts = pd.Series(data = np.random.randn(len(rng)), index = rng)
print(ts)
print("-"*50)
print(ts.index.tz)<jupyter_output>2012-03-09 09:30:00+08:00    1.263575
2012-03-10 09:30:00+08:00    1.011629
2012-03-11 09:30:00+08:00    1.993148
2012-03-12 09:30:00+08:00   -0.717136
2012-03-13 09:30:00+08:00    0.935880
2012-03-14 09:30:00+08:00    0.249375
Freq: D, dtype: float64
--------------------------------------------------
Asia/Taipei<jupyter_text>***Conversion from naive to localized is handled by the `tz_convert` method.***<jupyter_code>ts_utc = ts.tz_convert(tz = 'UTC')
ts_utc
ts_utc.tz_convert(tz = 'US/Eastern')<jupyter_output><empty_output><jupyter_text>Operations with Time Zone - aware Timestamp Objects***Similar to time series and date ranges, individual Timestamp objects similarly can be localized from naive to time zone-aware and converted from one time zone to another.***If you don't defind time-zone when creating the `timestamp` or `date_range`, you can use `tz_localized` to setting time-zone.`tz_convert` use to change time zone from one to another.<jupyter_code>stamp = pd.Timestamp('2011-03-12 04:00')
stamp_utc = stamp.tz_localize('utc')
stamp_utc
stamp_utc.tz_convert('US/Eastern')<jupyter_output><empty_output><jupyter_text>You can also pass time zone when creating the timestamp.<jupyter_code>stamp_moscow = pd.Timestamp('2011-03-12 04:00', tz = 'Europe/Moscow')
stamp_moscow<jupyter_output><empty_output><jupyter_text>***Time zone aware `Timestamp` objects internally store a UTC timestamp value as nanoseconds since the UNIX epoch(January, 1, 1970).******This UTC value is invariant between time zone conversions.***<jupyter_code>stamp_utc.value
stamp_utc.tz_convert("US/Eastern").value<jupyter_output><empty_output><jupyter_text>When performing time arithmetic using pandas's `DateOffset` objects, daylight savings time transitions are respected where possible.<jupyter_code>from pandas.tseries.offsets import Hour
stamp = pd.Timestamp('2012-03-12 01:30', tz = 'US/Eastern')
stamp
stamp + Hour(2)<jupyter_output><empty_output><jupyter_text>Operations between Different Time Zones***If two time series with different time zones are combined, the result will be UTC.***Since the timestamps are stored under the hood in UTC, this is a straightforward operation and requires no conversion to happen.<jupyter_code>rng = pd.date_range('3/7/2012 9:30', periods = 10, freq = "B")
rng
ts = pd.Series(np.random.randn(len(rng)), index = rng)
ts[-3:]
ts1 = ts[:7].tz_localize('Europe/London')
print("ts1:", ts1)
print("ts1 dtype : ", ts1.index.dtype)
print("-"*50)
ts2 = ts1[2:].tz_convert('Europe/Moscow')
print("ts2:", ts2)
print("ts2 dtype : ", ts2.index.dtype)
print("-"*50)
result = ts1 + ts2
print('result:', result)
print('result dtype:', result.index.dtype)<jupyter_output>ts1: 2012-03-07 09:30:00+00:00   -0.161699
2012-03-08 09:30:00+00:00    1.308506
2012-03-09 09:30:00+00:00    0.081826
2012-03-12 09:30:00+00:00   -0.308216
2012-03-13 09:30:00+00:00    0.038805
2012-03-14 09:30:00+00:00    2.220636
2012-03-15 09:30:00+00:00   -1.299924
Freq: B, dtype: float64
ts1 dtype :  datetime64[ns, Europe/London]
--------------------------------------------------
ts2: 2012-03-09 13:30:00+04:00    0.081826
2012-03-12 13:30:00+04:00   -0.308216
2012-03-13 13:30:00+04:00    0.038805
2012-03-14 13:30:00+04:00    2.220636
2012-03-15 13:30:00+04:00   -1.299924
Freq: B, dtype: float64
ts2 dtype :  datetime64[ns, Europe/Moscow]
--------------------------------------------------
result: 2012-03-07 09:30:00+00:00         NaN
2012-03-08 09:30:00+00:00         NaN
2012-03-09 09:30:00+00:00    0.163651
2012-03-12 09:30:00+00:00   -0.616433
2012-03-13 09:30:00+00:00    0.077611
2012-03-14 09:30:00+00:00    4.441272
2012-03-15 09:30:00+00:00   -2.599847
Freq: B, dtype: float64
[...]<jupyter_text>Periods and Period Arithmetic`Periods` represent time span, like days, months, quarters, or years.The `Period` class represents this day type, requiring a string or integer and a frequency from the above table.In this case, the `Period` object represents the full timespan from January 1, 2007 at December 31, 2007, inclusive.Conveniently, adding and subtracting integers from periods has the effect of shifting by their frequency.<jupyter_code>p = pd.Period(value = 2007, freq = 'A-DEC')
p
p + 5<jupyter_output><empty_output><jupyter_text>If two periods have the **same frequency**, their difference is the number of units between them.<jupyter_code>pd.Period(value = '2014', freq = 'A-DEC') - p<jupyter_output><empty_output><jupyter_text>Regular ranges of periods can be constructed using the `period_range` function.***The `PeriodIndex` class store a sequence of period and can serve as an axis index in any `pandas` data structure.***<jupyter_code>rng = pd.period_range(start = '1/1/2000', end = '6/30/2000', freq = 'M')
rng
pd.Series(data = np.random.randn(6), index = rng)<jupyter_output><empty_output><jupyter_text>***If you have any array of strings, you can also appeal to the `PeriodIndex` class itself.***<jupyter_code>values = ['2001Q3', '2002Q2', '2003Q1']
index = pd.PeriodIndex(values, freq = 'Q-DEC')
index<jupyter_output><empty_output><jupyter_text>Period Frequency Conversion***`Period` and `PeriodIndex` objects can be converted to another frequency using their `asfreq` method.***You can think of `Period("2007", "A-DEC")` as being a cursor pointing to a span of time, subdivided by monthly periods.For a `fiscal year` ending on a month other than December, the monthly subperiods belonging are different.<jupyter_code>p = pd.Period('2007', freq = 'A-JUN')
p
p.asfreq(freq = "M", how = 'start')
p.asfreq(freq = 'M', how = 'end')
p = pd.Period(value = '2007-08', freq = 'M')
p.asfreq("A-JAN")
rng = pd.period_range('2006', '2009', freq='A-DEC')
ts = pd.Series(np.random.randn(len(rng)), index = rng)
ts
ts.asfreq(freq = 'M', how = 'start')
ts.asfreq(freq = 'B', how = 'end')<jupyter_output><empty_output><jupyter_text>Quarterly Period FrequenciesQuarterly data is standard in accounting, finance, and other fields.Much quarterly data is reported relative to a `fiscal year end`, typeically the last calender or business day of one of 12 months of the year.As such, the period `2012Q4` has a different meaning depending on fiscal year end.<jupyter_code>p = pd.Period('2012Q4', freq = "Q-JAN")
p
p.asfreq('D', 'start')
p.asfreq('D', 'end')
p4pm = (p.asfreq("B", 'e') - 1).asfreq("T", 's') + 16*60
p4pm 
p4pm.to_timestamp()<jupyter_output><empty_output><jupyter_text>Generating quarterly ranges works as you would expect using `period-range`.<jupyter_code>rng = pd.period_range(start = '2011Q3', end = '2012Q4', freq = "Q-JAN")
ts = pd.Series(np.arange(len(rng)), index=rng)
ts
new_rng = (rng.asfreq("B", 'e') - 1).asfreq("T", "s") + 16*60
ts.index = new_rng.to_timestamp()
ts<jupyter_output><empty_output><jupyter_text>Converting Timestamps to Periods (and Back)`pd.Series` and `pd.DataFrame` objects indexed by timestamps can be converted to periods using the `to_period` method.<jupyter_code>rng = pd.date_range(start = '1/1/2000', periods = 3, freq = 'M')
ts = pd.Series(data = np.random.randn(3), index=rng)
ts
pts = ts.to_period()
pts<jupyter_output><empty_output><jupyter_text>Since periods always refer to non-overlapping timespans, a timestamp can only belong to a single period for a given frequency.While the frequency of the new `PeriodIndex` is inferred from the timestamp by default, you can specify any frequency you want.<jupyter_code>rng = pd.date_range('1/29/2000', periods = 3, freq = 'M')
ts2 = pd.Series(np.random.randn(3), index=rng)
ts2
pts = ts2.to_period('M')
pts<jupyter_output><empty_output><jupyter_text>***To convert back to timestamp, use `to_timestamp`***<jupyter_code>print(pts.to_timestamp(how = 'start'))
print("-"*50)
print(pts.to_timestamp(how = 'end'))<jupyter_output>2000-01-01   -0.029120
2000-02-01    0.763057
2000-03-01   -4.009619
Freq: MS, dtype: float64
--------------------------------------------------
2000-01-31   -0.029120
2000-02-29    0.763057
2000-03-31   -4.009619
Freq: M, dtype: float64<jupyter_text>Creating a PeriodIndex from arrayFixed frequency data sets are sometimes stored with timespan information spread across multiple columns.***By passing these arrays to `PeriodIndex` with a frequency, they can be combined to form an index for the `DataFrame`.***<jupyter_code>data = pd.read_csv(".\\pydata-book\\examples\\macrodata.csv")
data[-3:]
index = pd.PeriodIndex(year = data['year'], 
                       quarter = data['quarter'], freq = 'Q-DEC')
index
data.index = index
data[-5:]<jupyter_output><empty_output><jupyter_text>Resampling and Frequency Conversion`Resampling` refers to the process of converting a time series from one frequency to another.***Aggregating higher frequency data to lower frequency is called `downsampling`, while converting lower frequency to higher frequency is called `upsampling`.***Not all resampling falls into either of these categories.For example, converting `W-WED`(weekly on Wednesday) to `W-FRI` is neither unsampling nor downsampling.<jupyter_code>rng = pd.date_range(start = "1/1/2000", periods = 100, freq = 'D')
rng<jupyter_output><empty_output><jupyter_text>***`resample` is flexible and high-performance method that can be used to process very large time series.***<jupyter_code>ts = pd.Series(data = np.random.randn(len(rng)), index = rng)
ts[-5:]
ts.resample(rule = 'M').mean()
ts.resample(rule = "M", kind = 'period').mean()<jupyter_output><empty_output><jupyter_text>DownsamplingAggregating data to regular, lower frequency is a pretty normal time series task.The data you're aggreagting doesn't need to be fixed frequently.The desired frequency defines `bin edges` that are used to slice the time series into pieces to aggregate.To convert to monthly, `M` or `BM`, the data need to be chopped up into one month interval, and the union of intervals must make up the whole time frame.There are a couple things to think about when using `resample` to downsample data: * Which side of each interval is closed * How to lable each aggregated bin, wither with the start of the interval or the end<jupyter_code>rng = pd.date_range("1/1/2000", periods = 12, freq = 'T')
rng
ts = pd.Series(data = np.arange(12), index = rng)
ts<jupyter_output><empty_output><jupyter_text>Suppose you wanted to aggregate this data into five-minute chucks or bars by taking the sum of each group.The frequency you pass define bin edges in five-minute increments.**The `right bin` edge is inclusive, so the 00:05 value is included in the 00:00 to 00:05 interval.***Passing `closed = 'ledt'` changes the interval to be closed on the left.<jupyter_code>print("-"*10 + 'Include left bin' + "-"*10)
print(ts.resample(rule = '5min', closed = 'left', label = 'right').sum())

print("-"*10 + 'Include right bin' + "-"*10)
print(ts.resample(rule = '5min', closed = 'right', label = 'right').sum())<jupyter_output>----------Include left bin----------
2000-01-01 00:05:00    10
2000-01-01 00:10:00    35
2000-01-01 00:15:00    21
Freq: 5T, dtype: int32
----------Include right bin----------
2000-01-01 00:00:00     0
2000-01-01 00:05:00    15
2000-01-01 00:10:00    40
2000-01-01 00:15:00    11
Freq: 5T, dtype: int32<jupyter_text>***Lastly, you might want to shift the result index by some amount, say subtracting one second from the right rdge to make it more clear which interval the timestamp refers to.***To do this, pass a string or date offset to `loffset`.<jupyter_code>ts.resample(rule = '5min', loffset = '-1s', label = 'right').sum()
# the same as `loffset`
temp = ts.resample(rule = '5min', label = 'right').sum()
temp.shift(periods = -1, freq = 's')<jupyter_output><empty_output><jupyter_text>Open-Hight-Low-Close (OHLC) resamplingIn finance, an ubiquitous way to aggregate a time series is to compute four values for each bucket.The first(open), last(close), maximum(hight) and minimal(low) values.By passing `ohlc` method you will obtain a DataFrame having colunms containing these four aggregates, which are efficiently computed in a single sweep of the data.***The default setting of the argument `closed` and `label` are `left`.***<jupyter_code>ts.resample(rule = '5min', closed = 'right', label = 'right').ohlc()
ts.resample(rule = '5min', closed = 'left', label = 'left').ohlc()<jupyter_output><empty_output><jupyter_text>Resampling with GroupbyAn alternate way to downsample is to use pandas's `groupby` functionality.***You can groupy by month or weekday by passing a function that accesses those fields on the time series's index.***<jupyter_code>rng = pd.date_range('1/1/2000', periods = 100, freq = 'D')
ts = pd.Series(data = np.random.randn(len(rng)), index = rng)
ts[-5:]
print('-'*25 + 'Groupby by month' + '-'*25)
print(ts.groupby(lambda x: x.month).mean())
print('-'*25 + 'Groupby by weekday' + '-'*25)
print(ts.groupby(lambda x: x.weekday).mean())
ts.resample(rule = 'M').mean()<jupyter_output><empty_output><jupyter_text>Unsampling and InterpolationWhen converting from a low frequency to a highter frequency, no aggregation is needed.<jupyter_code>frame = pd.DataFrame(data = np.random.randn(2, 4), 
                     index = pd.date_range('2000', periods = 2, freq = 'W-WED'),
                     columns = ['Colorado', 'Texas', 'New York', 'Ohio'])
frame[-2:]<jupyter_output><empty_output><jupyter_text>Suppose you wanted to fill forward each weekly value on the non-Wednesday.***The same filling or interpolation methods available in `fillna` and `reindex` methods are available for resampling.***<jupyter_code>df_daily = frame.resample('D').asfreq()
print(df_daily)

df_daily1 = frame.resample('D').ffill()
print(df_daily1)
df_daily.fillna(method = 'ffill') # forward fill, `bfill` => backward fill
frame.resample('D').bfill()<jupyter_output><empty_output><jupyter_text>You can similarly choose to only fill a certain number of periods forward to limit how far to continue using an observed value.<jupyter_code>temp = frame.resample('D')
temp.ffill(limit = 2)
print(frame.resample('W-THU').asfreq())
print("-"*25 + 'forward fill' + "-"*25)
print(frame.resample('W-THU').ffill())
print("-"*25 + 'backward fill' + "-"*25)
print(frame.resample('W-THU').ffill())<jupyter_output>Colorado  Texas  New York  Ohio
2000-01-06       NaN    NaN       NaN   NaN
2000-01-13       NaN    NaN       NaN   NaN
-------------------------forward fill-------------------------
            Colorado     Texas  New York      Ohio
2000-01-06 -1.307641 -0.197518   0.10100 -1.125414
2000-01-13 -0.583116 -1.157085  -0.46385  0.707580
-------------------------backward fill-------------------------
            Colorado     Texas  New York      Ohio
2000-01-06 -1.307641 -0.197518   0.10100 -1.125414
2000-01-13 -0.583116 -1.157085  -0.46385  0.707580<jupyter_text>Resampling with Periods<jupyter_code>frame = pd.DataFrame(np.random.randn(24, 4), 
                     index=pd.period_range('1-2000', '12-2001', freq='M'), 
                     columns=['Colorado', 'Texas', 'New York', 'Ohio'])
frame[-5:]
annual_frame = frame.resample(rule = "A-DEC").mean()
annual_frame<jupyter_output><empty_output><jupyter_text>Upsampling is more nuanced as you must make a decision about which end of the timespan in the new frequency to place the value before resampling, just like the `asfreq` method.<jupyter_code># Q-DEC : Quarterly, year ending in Decmber
print("-"*20 + 'convention = "start"' + "-"*20)
print(annual_frame.resample(rule = 'Q-DEC', convention = 'start').ffill())
print("-"*20 + 'convention = "start"' + "-"*20)
print(annual_frame.resample(rule = 'Q-DEC', convention = 'end').ffill())<jupyter_output>--------------------convention = "start"--------------------
        Colorado     Texas  New York      Ohio
2000Q1 -0.097080 -0.128213 -0.630757  0.260586
2000Q2 -0.097080 -0.128213 -0.630757  0.260586
2000Q3 -0.097080 -0.128213 -0.630757  0.260586
2000Q4 -0.097080 -0.128213 -0.630757  0.260586
2001Q1  0.355655  0.597431  0.026471  0.095409
2001Q2  0.355655  0.597431  0.026471  0.095409
2001Q3  0.355655  0.597431  0.026471  0.095409
2001Q4  0.355655  0.597431  0.026471  0.095409
--------------------convention = "start"--------------------
        Colorado     Texas  New York      Ohio
2000Q4 -0.097080 -0.128213 -0.630757  0.260586
2001Q1 -0.097080 -0.128213 -0.630757  0.260586
2001Q2 -0.097080 -0.128213 -0.630757  0.260586
2001Q3 -0.097080 -0.128213 -0.630757  0.260586
2001Q4  0.355655  0.597431  0.026471  0.095409<jupyter_text>Since periods refer to timespans, the rules about upsampling and downsampling are more rigid: * ***In downsampling, the target frequency must be a subperiod of the source frequency.*** * ***In upsampling, the target frequency must be a superpriod of the source frequency.***<jupyter_code># The original index start from 1-2000, so the index after transforming will start from 2000Q4. 
annual_frame.resample('Q-MAR').ffill()<jupyter_output><empty_output><jupyter_text>Time Series Plotting<jupyter_code>close_px_all = pd.read_csv(".\\pydata-book\\examples\\stock_px.csv", parse_dates = True, index_col = 0)
close_px_all.head(5)
close_px = close_px_all[['AAPL', "MSFT", "XOM"]]
close_px = close_px.resample('B').ffill()
close_px.head(5)<jupyter_output><empty_output><jupyter_text>***When called on a DataFrame all of the time series are drawn on a single subplot with a legend indicating which is which.***<jupyter_code>close_px['AAPL'].plot(grid = True)
close_px.loc['2009'].plot(grid = True)
close_px['AAPL'].loc["01-2011":"03-2011"].plot()
appl_q = close_px['AAPL'].resample("Q-DEC").ffill()
appl_q['2009':].plot(grid = True)<jupyter_output><empty_output><jupyter_text>Moving Window FunctionsA common class of array transformations intended for time series operations are statistic and other functions evaluated over a sliding window or with exponentially decaying weights.***Like other statistical functionsm these also automatically exclude missing data.***`rolling_mean` is one of the simplest such functions.It takes a TimeSeries or DataFrame along with a window(expressed as a number of periods).<jupyter_code>close_px[['AAPL']].plot()<jupyter_output><empty_output><jupyter_text>By default functions like `rolling` require the indicated number of non-NA observations.***This behavior can be changed to account for missing data and, in particular, the fact that you will have fewer than `window` periods of data at the begining of the time series.***<jupyter_code>close_px['AAPL'].rolling(window = 250, center = False).mean().plot()
close_px['AAPL'].plot(grid = True, title = 'Apple Price with 250-day MA')
appl_std250 = close_px['AAPL'].rolling(window = 250, min_periods = 10).std()
print(close_px[close_px.index <= '1990-02-14']['AAPL'].std())
appl_std250[:11]
appl_std250.plot(grid = True, 
                 title = 'Apple 250-day daily return standard deviation')<jupyter_output><empty_output><jupyter_text>To compute an `expanding window mean`, you can see that an expanding window is just a special case where the window is the length of the time series, but only one or more period is required to compute a value.<jupyter_code># Define expanding mean in terms of rolling mean
expanding_mean = lambda x: rolling_mean(x, len(x), min_periods = 1)
close_px.rolling(window = 60).mean().plot(logy = True)<jupyter_output><empty_output><jupyter_text>Exponentially-weighted functionsAn alternative to using a static window size with equally-weighted observations is to specify a constant `decay factor` to give more weight to more recent observations.In mathematical terms, if $ma_t$ is the moving average result at time t and x is the time series in question, each value in the reslut is computed as $ma_t = a^* ma_{t-1} + (a-1)*x_t$, where a is the decay factor.***Since an exponentially-weighted statistic place more weight on more recent observations, it `adapts` faster to changes compared with the equal-weighted version.***<jupyter_code>fig, axes = plt.subplots(nrows=2, ncols=1, sharex=True, 
                         sharey=True, figsize=(12, 7))
aapl_px = close_px.AAPL['2005':'2009']
ma60 = aapl_px.rolling(window = 60, min_periods = 50).mean()
ewma60 = aapl_px.ewm(span = 60).mean()

aapl_px.plot(style = 'k--', ax = axes[0], grid = True)
ma60.plot(style = 'k--', ax = axes[0], grid = True)

aapl_px.plot(style='k-', ax=axes[1])
ewma60.plot(style='k--', ax=axes[1])

axes[0].set_title('Simple MA')
axes[1].set_title('Exponentially-weight MA')<jupyter_output><empty_output><jupyter_text>Binary Moving Window FunctionsSome statistical operator, like correlation and covariance, need to operate on two time series.<jupyter_code>spx_px = pd.read_csv(".\\pydata-book\\examples\\spx.csv", parse_dates = True, index_col = 0)
spx_px[:3]
spx_rets = spx_px/spx_px.shift(1)-1 # compute percent changes
returns = close_px.pct_change()
spx_px.pct_change()[:3] # the same as the above 
corr = returns['AAPL'].rolling(window = 125, min_periods = 100).corr(spx_rets)
corr.plot(grid = True, title = 'Six-month AAPL return correlation to S&P 500')
# need to transform to Series
spx_ser = pd.Series(spx_rets.values.flatten(), index = spx_rets.index)
corr = returns.rolling(window = 125, min_periods = 100).corr(spx_ser)
corr.plot(grid = True, title = 'Six-month return correlation to S&P 500')<jupyter_output><empty_output><jupyter_text>User-Defined Moving Window Functions***The `rolling_apply` function provides a means to apply an array function of your own devising over a moving window.***<jupyter_code>from scipy.stats import percentileofscore
scrore_at_2percent = lambda x: percentileofscore(x, score = 0.02, kind = 'rank')
result = returns['AAPL'].rolling(window = 250).apply(scrore_at_2percent)
result.plot(grid = True, 
            title = 'Percentile rank of 2% AAPL return over 1 year window')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14432960417968085, 'fraction_numerical': 0.07209381566244534, 'mean_word_length': 4.577038796516232, 'pattern_counts': {'":': 1, '<': 272, '<?xml version=': 0, '>': 268, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '69', 'dir': 'jupyter-structured-clean-dedup', 'id': '505739', 'license': 'MIT', 'n_tokens_mistral': 13324, 'n_tokens_neox': 11566, 'n_words': 3507, 'path': 'Ch10 Time Series.ipynb', 'repo_name': 'hengbinxu/Python4DataAnalysis-Note', 'size': '35218'}