<jupyter_start><jupyter_text>Time Series<jupyter_code>%matplotlib inline
# @quangphu1912: this is to show the plot in the notebook 
# https://stackoverflow.com/questions/43545050/using-matplotlib-notebook-after-matplotlib-inline-in-jupyter-notebook-doesnt
import numpy as np
import pandas as pd
np.random.seed(12345)
import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
PREVIOUS_MAX_ROWS = pd.options.display.max_rows
pd.options.display.max_rows = 20
np.set_printoptions(precision=4, suppress=True)<jupyter_output><empty_output><jupyter_text>`Time series data is an important form of structured data in many different fields`, such as finance, economics, ecology, neuroscience, and physics. Anything that is observed or measured at many points in time forms a time series. Many time series are `fixed frequency`, which is to say that data points *occur at regular intervals according to some rule*, such as every 15 seconds, every 5 minutes, or once per month. Time series can also be `irregular` *without a fixed unit of time or offset between units*. How you mark and refer to time series data depends on the application, and you may have one of the following: * `Timestamps`, specific instants in time* Fixed `periods`, such as the month January 2007 or the full year 2010* `Intervals` of time, indicated by a start and end timestamp. Periods can be thought of as special cases of intervals* Experiment or elapsed time; each timestamp is a measure of time relative to a particular start time (e.g., the diameter of a cookie baking each second since being placed in the oven)  11.1 Date and Time Data Types and Tools The Python standard library includes data types for date and time data, as well as calendar-related functionality. The `datetime`, `time`, and `calendar` modules are the main places to start. The `datetime.datetime` type, or simply `datetime`, is widely used:<jupyter_code>from datetime import datetime
now = datetime.now()
now
now.year, now.month, now.day<jupyter_output><empty_output><jupyter_text>`datetime` stores both the date and time down to the microsecond. `timedelta` represents the temporal difference between two datetime objects:<jupyter_code>delta = datetime(2011, 1, 7) - datetime(2008, 6, 24, 8, 15)
delta
delta.days
delta.seconds<jupyter_output><empty_output><jupyter_text>You *can add (or subtract) a* `timedelta` or multiple thereof to a `datetime` object to yield a new shifted object:<jupyter_code>from datetime import timedelta
start = datetime(2011, 1, 7)
start + timedelta(1, 12)
timedelta?
start - 2 * timedelta(12)<jupyter_output><empty_output><jupyter_text>11.1.1. Converting Between String and Datetime You can *format `datetime` objects* and pandas Timestamp objects, which I’ll introduce later, as *strings* using `str` or the `strftime` method, passing a format specification<jupyter_code>stamp = datetime(2011, 1, 3)
str(stamp)
stamp.strftime('%Y-%m-%d')<jupyter_output><empty_output><jupyter_text>You can use these same format codes to convert strings to dates using `datetime.strptime`:<jupyter_code>value = '2011-01-03'
datetime.strptime(value, '%Y-%m-%d')
datestrs = ['7/6/2011', '8/6/2011']
[datetime.strptime(x, '%m/%d/%Y') for x in datestrs]<jupyter_output><empty_output><jupyter_text>`datetime.strptime` is a good way to parse a date with a known format. *However, it can be a bit annoying to have to write a format spec each time, especially for common date formats*. In this case, you can use the `parser.parse` method in the third-party `dateutil` package (this is installed automatically when you install pandas):<jupyter_code>from dateutil.parser import parse
parse('2011-01-03')<jupyter_output><empty_output><jupyter_text>`dateutil` is capable of parsing most human-intelligible date representations:<jupyter_code>parse('Jan 31, 1997 10:45 PM')<jupyter_output><empty_output><jupyter_text>In international locales, day appearing before month is very common, so you can pass `dayfirst=True` to indicate this:<jupyter_code>parse('6/12/2011', dayfirst=True)<jupyter_output><empty_output><jupyter_text>pandas is *generally oriented toward working with arrays of dates*, whether used as an axis index or a column in a DataFrame. The `to_datetime` method parses many different kinds of date representations. Standard date formats like ISO 8601 can be parsed very quickly:<jupyter_code>datestrs = ['2011-07-06 12:00:00', '2011-08-06 00:00:00']
pd.to_datetime(datestrs)<jupyter_output><empty_output><jupyter_text>It also handles values that should be considered missing (None, empty string, etc.):<jupyter_code>idx = pd.to_datetime(datestrs + [None])
idx
idx[2]
#NaT (Not a Time) is pandas’s null value for timestamp data.
pd.isnull(idx)<jupyter_output><empty_output><jupyter_text>**CAUTION**`dateutil.parser` is a useful but imperfect tool. Notably, *it will recognize some strings as dates that you might prefer that it didn’t* — for example, '42' will be parsed as the year 2042 with today’s calendar date.  11.2 Time Series Basics  11.2.0 A basic kind of time series object in pandas is a Series indexed by timestamps, which is often represented external to pandas as `Python strings` or `datetime` objects:<jupyter_code>from datetime import datetime
dates = [datetime(2011, 1, 2), datetime(2011, 1, 5),
         datetime(2011, 1, 7), datetime(2011, 1, 8),
         datetime(2011, 1, 10), datetime(2011, 1, 12)]
ts = pd.Series(np.random.randn(6), index=dates)
ts
#@P: index = column date, value =random value, ts = timeseries<jupyter_output><empty_output><jupyter_text>Under the hood, these `datetime` objects have been put in a `DatetimeIndex`:<jupyter_code>ts.index<jupyter_output><empty_output><jupyter_text>Like other Series, arithmetic operations between differently indexed time series automatically align on the dates:<jupyter_code>ts[::2] #@P: slice by step ever 2 lines
ts + ts[::2]
#@P note: in the odd column, where they do not have the value to perform add calculation -> NaN return<jupyter_output><empty_output><jupyter_text>Recall that ts[::2] selects every second element in ts. pandas stores timestamps using NumPy’s `datetime64` data type at the nanosecond resolution<jupyter_code>ts.index.dtype<jupyter_output><empty_output><jupyter_text>Scalar values from a `DatetimeIndex` are pandas `Timestamp` objects:<jupyter_code>stamp = ts.index[0]
stamp<jupyter_output><empty_output><jupyter_text>A `Timestamp` can be substituted anywhere you would use a `datetime` object. Additionally, it can store frequency information (if any) and understands how to do time zone conversions and other kinds of manipulations. More on both of these things later.  11.2.1. Indexing, Selection, Subsetting Time series behaves like any other `pandas.Series` when you are indexing and selecting data based on label:<jupyter_code>ts
stamp = ts.index[2]
ts[stamp]<jupyter_output><empty_output><jupyter_text>As a convenience, you can also *pass a string that is interpretable as a date:*<jupyter_code>ts['1/10/2011']
ts['20110110']<jupyter_output><empty_output><jupyter_text>For longer time series, a year or only a year and month can be passed to easily select slices of data:<jupyter_code>longer_ts = pd.Series(np.random.randn(1000),
                      index=pd.date_range('1/1/2000', periods=1000))
longer_ts
#@P: to create a time series with index start from 1/1/2020
longer_ts.tail()
#@P: this is lase entries of this timeseries
longer_ts['2001']
#Here, the string '2001' is interpreted as a year and selects that time period.
#This also works if you specify the month:
longer_ts['2001-05']<jupyter_output><empty_output><jupyter_text>Slicing with `datetime` objects works as well:<jupyter_code>ts[datetime(2011, 1, 7):]<jupyter_output><empty_output><jupyter_text>Because most time series data is ordered chronologically, you can slice with timestamps not contained in a time series to perform a range query:<jupyter_code>ts
ts['1/6/2011':'1/11/2011']<jupyter_output><empty_output><jupyter_text>As before, you can pass either a string date, `datetime`, or timestamp. Remember that slicing in this manner produces views on the source time series like slicing NumPy arrays. This means that no data is copied and modifications on the slice will be reflected in the original data. There is an equivalent instance method, `truncate`, that slices a Series between two dates:<jupyter_code>ts.truncate(after='1/9/2011')<jupyter_output><empty_output><jupyter_text>All of *this holds true for DataFrame as well*, indexing on its rows:<jupyter_code>dates = pd.date_range('1/1/2000', periods=100, freq='W-WED')
long_df = pd.DataFrame(np.random.randn(100, 4),
                       index=dates,
                       columns=['Colorado', 'Texas',
                                'New York', 'Ohio'])
long_df.loc['5-2001']<jupyter_output><empty_output><jupyter_text>11.2.2. Time Series with Duplicate Indices In some applications, there may be multiple data observations falling on a particular timestamp. Here is an example:<jupyter_code>dates = pd.DatetimeIndex(['1/1/2000', '1/2/2000', '1/2/2000',
                          '1/2/2000', '1/3/2000'])
dup_ts = pd.Series(np.arange(5), index=dates)
dup_ts<jupyter_output><empty_output><jupyter_text>We can tell that *the index is not unique by checking* its `is_unique` property:<jupyter_code>dup_ts.index.is_unique<jupyter_output><empty_output><jupyter_text>Indexing into this time series will now either produce scalar values or slices depending on whether a timestamp is duplicated:<jupyter_code>dup_ts['1/3/2000']  # not duplicated
dup_ts['1/2/2000']  # duplicated<jupyter_output><empty_output><jupyter_text>Suppose you wanted to *aggregate the data having non-unique timestamps*.@P: duplicated timestampOne way to do this is to use `groupby` and pass `level=0`:<jupyter_code>grouped = dup_ts.groupby(level=0)
grouped.mean()
grouped.count()<jupyter_output><empty_output><jupyter_text>11.3. Date Ranges, Frequencies, and Shifting Generic time series in pandas are assumed to be irregular; that is, they have no fixed frequency. For many applications this is sufficient. However, *it’s often desirable to work relative to a fixed frequency, such as daily, monthly, or every 15 minutes, even if that means introducing missing values into a time series*. Fortunately pandas has a full suite of standard time series frequencies and tools for resampling, inferring frequencies, and generating fixed-frequency date ranges. For example, you can convert the sample time series to be fixed daily frequency by calling resample:<jupyter_code>ts
resampler = ts.resample('D')
#The string 'D' is interpreted as daily frequency.<jupyter_output><empty_output><jupyter_text>11.3.1. Generating Date Ranges While I used it previously without explanation, `pandas.date_range` is responsible for generating a `DatetimeIndex` with an indicated length according to a particular frequency:<jupyter_code>index = pd.date_range('2012-04-01', '2012-06-01')
index<jupyter_output><empty_output><jupyter_text>By default, `date_range` generates daily timestamps. If you pass only a start or end date, *you must pass a number of periods* to generate:<jupyter_code>pd.date_range(start='2012-04-01', periods=20)
pd.date_range(end='2012-06-01', periods=20)<jupyter_output><empty_output><jupyter_text>`The start and end dates define strict boundaries for the generated date index.` For example, if you wanted a date index containing the last business day of each month, you would pass the 'BM' frequency (business end of month; see more complete listing of frequencies in Table 11-4) and *only dates falling on or inside the date interval will be included*:<jupyter_code>pd.date_range('2000-01-01', '2000-12-01', freq='BM')<jupyter_output><empty_output><jupyter_text>`date_range` by default preserves the time (if any) of the start or end timestamp:<jupyter_code>pd.date_range('2012-05-02 12:56:31', periods=5)<jupyter_output><empty_output><jupyter_text>Sometimes you will have start or end dates with time information but want to generate a set of timestamps *normalized* to midnight as a convention. To do this, there is a normalize option:<jupyter_code>pd.date_range('2012-05-02 12:56:31', periods=5, normalize=True)<jupyter_output><empty_output><jupyter_text>11.3.2. Frequencies and Date Offsets Frequencies in pandas are composed of a *base frequency* and a multiplier. Base frequencies are typically referred to by a string alias, like `'M'` for monthly or `'H'` for hourly. For each base frequency, there is an object defined generally referred to as a `date offset`. For example, hourly frequency can be represented with the `Hour` class:<jupyter_code>from pandas.tseries.offsets import Hour, Minute
hour = Hour()
hour<jupyter_output><empty_output><jupyter_text>You can define a *multiple of an offset* by passing an integer:<jupyter_code>four_hours = Hour(4)
four_hours<jupyter_output><empty_output><jupyter_text>In most applications, you would never need to explicitly create one of these objects, instead using a string alias like 'H' or '4H'. Putting an integer before the base frequency creates a multiple:<jupyter_code>pd.date_range('2000-01-01', '2000-01-03 23:59', freq='4h')<jupyter_output><empty_output><jupyter_text>Many offsets can be combined together by addition:<jupyter_code>Hour(2) + Minute(30)<jupyter_output><empty_output><jupyter_text>Similarly, you can pass frequency strings, like '1h30min', that will effectively be parsed to the same expression:<jupyter_code>pd.date_range('2000-01-01', periods=10, freq='1h30min')<jupyter_output><empty_output><jupyter_text>Some frequencies describe points in time that are not evenly spaced. For example, 'M' (calendar month end) and 'BM' (last business/weekday of month) depend on the number of days in a month and, in the latter case, whether the month ends on a weekend or not. We refer to these as `anchored offsets`.  Week of month dates One useful frequency class is “week of month,” starting with WOM. This enables you to get dates like the third Friday of each month:<jupyter_code>rng = pd.date_range('2012-01-01', '2012-09-01', freq='WOM-3FRI')
list(rng)<jupyter_output><empty_output><jupyter_text>11.3.3. Shifting (Leading and Lagging) Data “Shifting” refers to moving data backward and forward through time. Both Series and DataFrame have a `shift` method for doing naive shifts forward or backward, leaving the index unmodified:<jupyter_code>ts = pd.Series(np.random.randn(4),
               index=pd.date_range('1/1/2000', periods=4, freq='M'))
ts
ts.shift(2)
ts.shift(-2)<jupyter_output><empty_output><jupyter_text>When we shift like this, missing data is introduced either at the start or the end of the time series. *A common use* of `shift` is computing percent changes in a time series or multiple time series as DataFrame columns. This is expressed as: ts / ts.shift(1) - 1 Because naive shifts leave the index unmodified, some data is discarded. Thus if the frequency is known, it can be passed to `shift` to advance the timestamps instead of simply the data:<jupyter_code>ts.shift(2, freq='M')<jupyter_output><empty_output><jupyter_text>Other frequencies can be passed, too, giving you some flexibility in how to lead and lag the data:<jupyter_code>ts.shift(3, freq='D')
ts.shift(1, freq='90T')
#The T here stands for minutes.<jupyter_output><empty_output><jupyter_text>Shifting dates with offsets The pandas date offsets can also be used with `datetime` or `Timestamp` objects:<jupyter_code>from pandas.tseries.offsets import Day, MonthEnd
now = datetime(2011, 11, 17)
now + 3 * Day()<jupyter_output><empty_output><jupyter_text>If you add an anchored offset like `MonthEnd`, the first increment will “roll forward” a date to the next date according to the frequency rule:<jupyter_code>now + MonthEnd()
now + MonthEnd(2)<jupyter_output><empty_output><jupyter_text>Anchored offsets can explicitly “roll” dates forward or backward by simply using their `rollforward` and `rollback` methods, respectively:<jupyter_code>offset = MonthEnd()
offset
offset.rollforward(now)
offset.rollback(now)<jupyter_output><empty_output><jupyter_text>A creative use of date offsets is to use these methods with `groupby`:<jupyter_code>ts = pd.Series(np.random.randn(20),
               index=pd.date_range('1/15/2000', periods=20, freq='4d'))
ts
ts.groupby(offset.rollforward).mean()<jupyter_output><empty_output><jupyter_text>Of course, an easier and faster way to do this is using resample (we’ll discuss this in much more depth in Section 11.6, “Resampling and Frequency Conversion,”):<jupyter_code>ts.resample('M').mean()<jupyter_output><empty_output><jupyter_text>11.4. Time Zone Handling Many time series users choose to work with time series in `coordinated universal time` or `UTC`, which is the successor to Greenwich Mean Time and is the current international standard. Time zones are expressed as offsets from UTC; for example, New York is four hours behind UTC during daylight saving time and five hours behind the rest of the year. In Python, time zone information comes from the third-party `pytz library` (installable with pip or conda), which exposes the Olson database, a compilation of world time zone information. This is especially important for historical data because the daylight saving time (DST) transition dates (and even UTC offsets) have been changed numerous times depending on the whims of local governments. In the United States, the DST transition times have been changed many times since 1900! For detailed information about the pytz library, you’ll need to look at that library’s documentation. As far as this book is concerned, pandas wraps `pytz`’s functionality so you can ignore its API outside of the time zone names Time zone names can be found interactively and in the docs:<jupyter_code>import pytz #@P:tips to remember: python timezone
pytz.common_timezones[-5:]<jupyter_output><empty_output><jupyter_text>*To get a time zone object from pytz*, use `pytz.timezone`:<jupyter_code>tz = pytz.timezone('America/New_York')
tz<jupyter_output><empty_output><jupyter_text>*Methods in pandas will accept* either `time zone names` or `these objects`.  11.4.1. Time Zone Localization and Conversion By default, time series in pandas are *time zone naive*. For example, consider the following time series:<jupyter_code>rng = pd.date_range('3/9/2012 9:30', periods=6, freq='D')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts<jupyter_output><empty_output><jupyter_text>The index’s tz field is *None*:<jupyter_code>print(ts.index.tz)<jupyter_output>None<jupyter_text>Date ranges can be generated with a time zone set:<jupyter_code>pd.date_range('3/9/2012 9:30', periods=10, freq='D', tz='UTC')<jupyter_output><empty_output><jupyter_text>*Conversion from naive to localized* is handled by the *tz_localize* method:<jupyter_code>ts

ts_utc = ts.tz_localize('UTC')
ts_utc
ts_utc.index<jupyter_output><empty_output><jupyter_text>Once a time series has been localized to a particular time zone, it *can be converted to another time zone* with `tz_convert`:<jupyter_code>ts_utc.tz_convert('America/New_York')<jupyter_output><empty_output><jupyter_text>In the case of the preceding time series, which straddles a DST transition in the *America/New_York* time zone, we could localize to EST and convert to, say, UTC or Berlin time:<jupyter_code>ts_eastern = ts.tz_localize('America/New_York')
ts_eastern.tz_convert('Europe/Berlin')<jupyter_output><empty_output><jupyter_text>`tz_localize` and `tz_convert` are also instance methods on DatetimeIndex:<jupyter_code>ts.index.tz_localize('Asia/Shanghai')<jupyter_output><empty_output><jupyter_text>**CAUTION**Localizing naive timestamps also checks for ambiguous or non-existent times around *daylight saving time transitions*.  11.4.2. Operations with Time Zone−Aware *Timestamp Objects* Similar to time series and date ranges, individual `Timestamp` objects similarly can be localized from naive to time zone–aware and converted from one time zone to another:<jupyter_code>stamp = pd.Timestamp('2011-03-12 04:00')
stamp_utc = stamp.tz_localize('utc')
stamp_utc.tz_convert('America/New_York')<jupyter_output><empty_output><jupyter_text>You can also *pass a time zone* when creating the Timestamp:<jupyter_code>stamp_moscow = pd.Timestamp('2011-03-12 04:00', tz='Europe/Moscow')
stamp_moscow<jupyter_output><empty_output><jupyter_text>Time zone–aware `Timestamp objects internally store a UTC timestamp value as nanoseconds` since the Unix epoch (January 1, 1970); this UTC value is *invariant between time zone conversions:*<jupyter_code>stamp_utc.value
stamp_utc.tz_convert('America/New_York').value
#P note: this value is a constant as explain above<jupyter_output><empty_output><jupyter_text>When performing time arithmetic using pandas’s `DateOffset` objects, pandas respects daylight saving time transitions where possible. Here we construct timestamps that occur right before DST transitions (forward and backward). First, 30 minutes before transitioning to DST:<jupyter_code>from pandas.tseries.offsets import Hour
stamp = pd.Timestamp('2012-03-12 01:30', tz='US/Eastern')
stamp
stamp + Hour()<jupyter_output><empty_output><jupyter_text>Then, 90 minutes before transitioning out of DST:<jupyter_code>stamp = pd.Timestamp('2012-11-04 00:30', tz='US/Eastern')
stamp
stamp + 2 * Hour()<jupyter_output><empty_output><jupyter_text>11.4.3. Operations Between Different Time Zones `If two time series with different time zones are combined, the result will be UTC.` Since the timestamps are stored under the hood in UTC, *this is a straightforward operation and requires no conversion to happen*:<jupyter_code>rng = pd.date_range('3/7/2012 9:30', periods=10, freq='B')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts
ts1 = ts[:7].tz_localize('Europe/London')
ts2 = ts1[2:].tz_convert('Europe/Moscow')
result = ts1 + ts2 #@P: when add 2 timestamps that has been coverted to different timezone -> result is store in UTC
result.index<jupyter_output><empty_output><jupyter_text>11.5. Periods and Period Arithmetic `Periods` represent timespans, like days, months, quarters, or years. The Period class represents this data type, requiring a string or integer and a frequency from Table 11-4:<jupyter_code>p = pd.Period(2007, freq='A-DEC')
p<jupyter_output><empty_output><jupyter_text>In this case, the Period object represents the full timespan from January 1, 2007, to December 31, 2007, inclusive. Conveniently, *adding and subtracting integers* from periods has the *effect of shifting by their frequency*:<jupyter_code>p + 5
p - 2<jupyter_output><empty_output><jupyter_text>If two periods have the same frequency, their difference is the number of units between them:<jupyter_code>pd.Period('2014', freq='A-DEC') - p<jupyter_output><empty_output><jupyter_text>Regular ranges of periods can be constructed with the `period_range` function:<jupyter_code>rng = pd.period_range('2000-01-01', '2000-06-30', freq='M')
rng<jupyter_output><empty_output><jupyter_text>The `PeriodIndex` class stores a sequence of periods and can serve as an axis index in any pandas data structure:<jupyter_code>pd.Series(np.random.randn(6), index=rng)<jupyter_output><empty_output><jupyter_text>If you have an array of strings, you can also use the `PeriodIndex` class:<jupyter_code>values = ['2001Q3', '2002Q2', '2003Q1']
index = pd.PeriodIndex(values, freq='Q-DEC')
index<jupyter_output><empty_output><jupyter_text>11.5.1. Period Frequency Conversion Periods and `PeriodIndex` objects can be converted to another frequency with their asfreq method.As an example, suppose we had an annual period and wanted to convert it into a monthly period either at the start or end of the year. This is fairly straightforward:<jupyter_code>p = pd.Period('2007', freq='A-DEC')
p
p.asfreq('M', how='start')
p.asfreq('M', how='end')<jupyter_output><empty_output><jupyter_text>You can think of Period('2007', 'A-DEC') as being a sort of cursor pointing to a `span of time`, subdivided by monthly periods. @P vietsub: this case point to a year and could be divided to smaller frequency, such as month * For a *fiscal year* ending on a month other than December, the corresponding monthly subperiods are different:<jupyter_code>p = pd.Period('2007', freq='A-JUN') #@P: to specify the frequency as annual and start in June
p
p.asfreq('M', 'start')
p.asfreq('M', 'end')<jupyter_output><empty_output><jupyter_text>When you are converting from high to low frequency, pandas determines the superperiod depending on where the subperiod “belongs.” For example, in A-JUN frequency, the month Aug-2007 is actually part of the 2008 period:<jupyter_code>p = pd.Period('Aug-2007', 'M')
p.asfreq('A-JUN')<jupyter_output><empty_output><jupyter_text>Whole `PeriodIndex` objects or time series can be similarly converted with the same semantics:<jupyter_code>rng = pd.period_range('2006', '2009', freq='A-DEC')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts
ts.asfreq('M', how='start')<jupyter_output><empty_output><jupyter_text>Here, the annual periods are replaced with monthly periods corresponding to the first month falling within each annual period. If *we instead wanted the last business day of each year*, we can use the 'B' frequency and indicate that we want the end of the period:<jupyter_code>ts.asfreq('B', how='end')<jupyter_output><empty_output><jupyter_text>11.5.2.Quarterly Period Frequencies Quarterly data is standard in accounting, finance, and other fields. Much quarterly data is reported relative to a `fiscal year end`, typically the last calendar or business day of one of the 12 months of the year. Thus, the period 2012Q4 has a different meaning depending on fiscal year end. pandas supports all 12 possible quarterly frequencies as Q-JAN through Q-DEC:<jupyter_code>p = pd.Period('2012Q4', freq='Q-JAN')
p
p.asfreq('D', 'start')
p.asfreq('D', 'end')<jupyter_output><empty_output><jupyter_text>Thus, it’s possible to do easy period arithmetic; for example, *to get the timestamp at 4 PM on the second-to-last business day of the quarter*, you could do:<jupyter_code>p4pm = (p.asfreq('B', 'e') - 1).asfreq('T', 's') + 16 * 60
p4pm
p4pm.to_timestamp()<jupyter_output><empty_output><jupyter_text>You can generate quarterly ranges using `period_range`. Arithmetic is identical, too:<jupyter_code>rng = pd.period_range('2011Q3', '2012Q4', freq='Q-JAN')
ts = pd.Series(np.arange(len(rng)), index=rng)
ts
new_rng = (rng.asfreq('B', 'e') - 1).asfreq('T', 's') + 16 * 60
ts.index = new_rng.to_timestamp()
ts<jupyter_output><empty_output><jupyter_text>11.5.3. Converting Timestamps to Periods (and Back) Series and DataFrame objects indexed by timestamps can be converted to periods with the `to_period` method:<jupyter_code>rng = pd.date_range('2000-01-01', periods=3, freq='M')
ts = pd.Series(np.random.randn(3), index=rng)
ts
pts = ts.to_period()
pts<jupyter_output><empty_output><jupyter_text>Since periods refer to non-overlapping timespans, a timestamp can only belong to a single period for a given frequency. While the frequency of the new `PeriodIndex` is inferred from the timestamps by default, you can specify any frequency you want. There is also no problem with having duplicate periods in the result:<jupyter_code>rng = pd.date_range('1/29/2000', periods=6, freq='D')
ts2 = pd.Series(np.random.randn(6), index=rng)
ts2
ts2.to_period('M')<jupyter_output><empty_output><jupyter_text>To convert back to timestamps, use `to_timestamp`:<jupyter_code>pts = ts2.to_period()
pts
pts.to_timestamp(how='end')<jupyter_output><empty_output><jupyter_text>11.5.4. Creating a PeriodIndex from Arrays Fixed frequency datasets are sometimes stored with timespan information spread across multiple columns. For example, in this macroeconomic dataset, the year and quarter are in different columns:<jupyter_code>data = pd.read_csv('examples/macrodata.csv')
data.head(5)
data.year
data.quarter<jupyter_output><empty_output><jupyter_text>By passing these arrays to `PeriodIndex` with a frequency, you can combine them to form an index for the DataFrame:<jupyter_code>index = pd.PeriodIndex(year=data.year, quarter=data.quarter,
                       freq='Q-DEC')#@P: this periodindex combine year and quarter from 2 different column to a single period
index
data.index = index
data.infl
data.infl<jupyter_output><empty_output><jupyter_text>11.6. Resampling and Frequency Conversion `Resampling` refers to the process of converting a time series from one frequency to another. Aggregating higher frequency data to lower frequency is called `downsampling`, while converting lower frequency to higher frequency is called `upsampling`. `Not all resampling falls into either of these categories;` for example, converting W-WED (weekly on Wednesday) to W-FRI is neither upsampling nor downsampling. pandas objects are equipped with a `resample` method, *which is the workhorse function for all frequency conversion*. resample has a `similar API to groupby`; you call resample to group the data, then call an aggregation function:<jupyter_code>rng = pd.date_range('2000-01-01', periods=100, freq='D')
ts = pd.Series(np.random.randn(len(rng)), index=rng)
ts
ts.resample('M').mean()
ts.resample('M', kind='period').mean()<jupyter_output><empty_output><jupyter_text>11.6.1. Downsampling Aggregating data to a regular, lower frequency is a pretty normal time series task. The data you’re aggregating doesn’t need to be fixed frequently; the desired frequency defines bin edges that are used to slice the time series into pieces to aggregate. For example, to convert to monthly, 'M' or 'BM', you need to chop up the data into one-month intervals. Each interval is said to be half-open; a data point can only belong to one interval, and the union of the intervals must make up the whole time frame.There are a couple things to think about when using resample to downsample data: * Which side of each interval is closed* How to label each aggregated bin, either with the start of the interval or the end To illustrate, let’s look at some one-minute data:<jupyter_code>rng = pd.date_range('2000-01-01', periods=12, freq='T')
ts = pd.Series(np.arange(12), index=rng)
ts<jupyter_output><empty_output><jupyter_text>Suppose you wanted to *aggregate this data into five-minute chunks* or bars by taking the sum of each group:<jupyter_code>ts.resample('5min', closed='right').sum()
ts.resample('5min', closed='right').sum()<jupyter_output><empty_output><jupyter_text>The frequency you pass defines bin edges in five-minute increments. *By default, the left bin edge is inclusive,* so the 00:00 value is included in the 00:00 to 00:05 interval.1 Passing closed='right' changes the interval to be closed on the right:<jupyter_code>ts.resample('5min', closed='right', label='right').sum()<jupyter_output><empty_output><jupyter_text>Lastly, you might want to shift the result index by some amount, say *subtracting one second from the right edge to make it more clear which interval the timestamp refers to.* To do this, pass a string or date offset to loffset:<jupyter_code>ts.resample('5min', closed='right',
            label='right', loffset='-1s').sum()<jupyter_output><ipython-input-172-b63ded3800c4>:1: FutureWarning: 'loffset' in .resample() and in Grouper() is deprecated.

>>> df.resample(freq="3s", loffset="8H")

becomes:

>>> from pandas.tseries.frequencies import to_offset
>>> df = df.resample(freq="3s").mean()
>>> df.index = df.index.to_timestamp() + to_offset("8H")

  ts.resample('5min', closed='right',<jupyter_text>11.6.1.1 Open-High-Low-Close (OHLC) resampling In finance, a popular way to aggregate a time series is to `compute four values for each bucket: the first (open), last (close), maximum (high), and minimal (low) values`. By using the ohlc aggregate function you will obtain a DataFrame having columns containing these four aggregates, which are efficiently computed in a single sweep of the data:<jupyter_code>ts.resample('5min').ohlc() #@P: open high low close<jupyter_output><empty_output><jupyter_text>11.6.2. Upsampling and Interpolation When converting from a low frequency to a higher frequency, no aggregation is needed. Let’s consider a DataFrame with some weekly data:<jupyter_code>frame = pd.DataFrame(np.random.randn(2, 4),
                     index=pd.date_range('1/1/2000', periods=2,
                                         freq='W-WED'),
                     columns=['Colorado', 'Texas', 'New York', 'Ohio'])
frame<jupyter_output><empty_output><jupyter_text>When you are using an aggregation function with this data, there is only one value per group, and missing values result in the gaps. We use the `asfreq` method *to convert to the higher frequency without any aggregation:*<jupyter_code>df_daily = frame.resample('D').asfreq()
df_daily<jupyter_output><empty_output><jupyter_text>Suppose you wanted to *fill forward* each weekly value on the non- Wednesdays. The same filling or interpolation methods available in the filler and reindex methods are available for resampling:<jupyter_code>frame.resample('D').ffill()<jupyter_output><empty_output><jupyter_text>* You can similarly choose to `only fill a certain number of periods forward` to limit how far to continue using an observed value:<jupyter_code>frame.resample('D').ffill(limit=2)<jupyter_output><empty_output><jupyter_text>Notably, the new date index need not overlap with the old one at all: P: don't understand 20211102<jupyter_code>frame.resample('W-THU').ffill()<jupyter_output><empty_output><jupyter_text>11.6.3. Resampling with Periods Resampling data indexed by periods is similar to timestamps:<jupyter_code>frame = pd.DataFrame(np.random.randn(24, 4),
                     index=pd.period_range('1-2000', '12-2001',
                                           freq='M'),
                     columns=['Colorado', 'Texas', 'New York', 'Ohio'])
frame[:5]
annual_frame = frame.resample('A-DEC').mean()
annual_frame<jupyter_output><empty_output><jupyter_text>Upsampling is more nuanced, as you must make a decision about which end of the timespan in the new frequency to place the values before resampling, just like the asfreq method. The convention argument defaults to `'start'` but can also be `'end'`:<jupyter_code># Q-DEC: Quarterly, year ending in December
annual_frame.resample('Q-DEC').ffill()
annual_frame.resample('Q-DEC', convention='end').ffill()<jupyter_output><empty_output><jupyter_text>Since periods refer to timespans, the rules about upsampling and downsampling are more rigid:* In downsampling, the target frequency must be a subperiod of the source frequency.* In upsampling, the target frequency must be a superperiod of the source frequency.If these rules are not satisfied, an exception will be raised. This mainly affects the quarterly, annual, and weekly frequencies; for example, the timespans defined by Q-MAR only line up with A-MAR, A-JUN, A-SEP, and A-DEC:<jupyter_code>annual_frame.resample('Q-MAR').ffill()<jupyter_output><empty_output><jupyter_text>11.7. Moving Window Functions `An important class of array transformations used for time series operations are statistics and other functions evaluated over a sliding window or with exponentially decaying weights.` This can be useful for smoothing noisy or gappy data. I call these moving window functions, even though it includes functions without a fixed-length window like exponentially weighted moving average. Like other statistical functions, these also automatically exclude missing data. Before digging in, we can load up some time series data and resample it to business day frequency:<jupyter_code>close_px_all = pd.read_csv('examples/stock_px_2.csv',
                           parse_dates=True, index_col=0)
close_px = close_px_all[['AAPL', 'MSFT', 'XOM']]
close_px = close_px.resample('B').ffill()<jupyter_output><empty_output><jupyter_text>I now introduce the rolling operator, which behaves similarly to `resample` and `groupby`. It can be called on a Series or DataFrame along with a window (expressed as a number of periods; see Figure 11-4 for the plot created):<jupyter_code>close_px.AAPL.plot()
close_px.AAPL.rolling(250).mean().plot()
plt.figure()<jupyter_output><empty_output><jupyter_text>The expression rolling(250) is similar in behavior to groupby, but *instead of grouping it creates an object that enables grouping over a 250-day sliding window*. So here we have the 250-day moving window average of Apple’s stock price.By default rolling functions require all of the values in the window to be non-NA. This behavior can be changed to account for missing data and, in particular, the fact that you will have fewer than window periods of data at the beginning of the time series (see Figure 11-5):<jupyter_code>appl_std250 = close_px.AAPL.rolling(250, min_periods=10).std()
appl_std250[5:12]
appl_std250.plot()<jupyter_output><empty_output><jupyter_text>In order to compute an `expanding window mean`, use the expanding operator instead of rolling. *The expanding mean starts the time window from the beginning of the time series and increases the size of the window until it encompasses the whole series*. An expanding window mean on the apple_std250 time series looks like this:<jupyter_code>expanding_mean = appl_std250.expanding().mean()<jupyter_output><empty_output><jupyter_text>Calling a moving window function on a DataFrame applies the transformation to each column (see Figure 11-6):<jupyter_code>plt.figure()
close_px.rolling(60).mean().plot(logy=True)<jupyter_output><empty_output><jupyter_text>The `rolling` function also accepts a string indicating a fixed-size time offset rather than a set number of periods. Using this notation can be useful for irregular time series. These are the same strings that you can pass to `resample`. For example, we could compute a 20-day rolling mean like so:<jupyter_code>close_px.rolling('20D').mean()<jupyter_output><empty_output><jupyter_text>11.7.1. Exponentially Weighted Functions `An alternative to using a static window size with equally weighted observations is to specify a constant decay factor to give more weight to more recent observations.` There are a couple of ways to specify the decay factor. A popular one is using a span, which makes the result comparable to a simple moving window function with window size equal to the span.Since an *exponentially weighted statistic places more weight on more recent observations*, `it “adapts” faster to changes compared with the equal weighted version.`pandas has the `ewm` operator to go along with rolling and expanding. Here’s an example comparing a 60-day moving average of Apple’s stock price with an EW moving average with span=60 (see Figure 11-7):<jupyter_code>aapl_px = close_px.AAPL['2006':'2007']
ma60 = aapl_px.rolling(30, min_periods=20).mean()
ewma60 = aapl_px.ewm(span=30).mean()
ma60.plot(style='k--', label='Simple MA')
ewma60.plot(style='k-', label='EW MA')
plt.legend()<jupyter_output><empty_output><jupyter_text>11.7.2. Binary Moving Window Functions `Some statistical operators, like correlation and covariance, need to operate on two time series`.As an example, financial analysts are often interested in a stock’s correlation to a benchmark index like the S&P 500. To have a look at this, we first compute the percent change for all of our time series of interest:<jupyter_code>plt.figure()
spx_px = close_px_all['SPX']
spx_rets = spx_px.pct_change()
returns = close_px.pct_change()<jupyter_output><empty_output><jupyter_text>The `corr` aggregation function after we call `rolling` can then compute the rolling correlation with spx_rets (see Figure 11-8 for the resulting plot):<jupyter_code>corr = returns.AAPL.rolling(125, min_periods=100).corr(spx_rets)
corr.plot()<jupyter_output><empty_output><jupyter_text>`Suppose you wanted to compute the correlation of the S&P 500 index with many stocks at once`. Writing a loop and creating a new DataFrame would be easy but might get repetitive, so if you pass a Series and a DataFrame, a function like rolling_corr will compute the correlation of the Series (spx_rets, in this case) with each column in the DataFrame (see Figure 11-9 for the plot of the result):<jupyter_code>plt.figure()
corr = returns.rolling(125, min_periods=100).corr(spx_rets)
corr.plot()<jupyter_output><empty_output><jupyter_text>11.7.3. User-Defined Moving Window Functions The `apply` method on `rolling` and related methods provides a means to apply an array function of your own devising over a moving window. The only requirement is that the function produce a single value (a reduction) from each piece of the array. For example, while we can compute sample quantiles using *rolling(...).quantile(q)*, we might be interested in the percentile rank of a particular value over the sample. The scipy.stats.percentileofscore function does just this (see Figure 11-10 for the resulting plot):<jupyter_code>plt.figure()
from scipy.stats import percentileofscore
score_at_2percent = lambda x: percentileofscore(x, 0.02)
result = returns.AAPL.rolling(250).apply(score_at_2percent)
result.plot()
pd.options.display.max_rows = PREVIOUS_MAX_ROWS<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09468794084739991, 'fraction_numerical': 0.026560295763000438, 'mean_word_length': 5.993536315699949, 'pattern_counts': {'":': 0, '<': 424, '<?xml version=': 0, '>': 438, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '106', 'dir': 'jupyter-structured-clean-dedup', 'id': '438056', 'license': 'MIT', 'n_tokens_mistral': 13324, 'n_tokens_neox': 12348, 'n_words': 4603, 'path': '11_Time Series.ipynb', 'repo_name': 'quangphu1912/Py-data-analysis-McKinney', 'size': '41114'}