<jupyter_start><jupyter_text>Expressing Recurring Events                    <NAME>        Bloomberg                dateutil on GithubGithub repo for this talk: pganssle-talks/boston-python-rrules<jupyter_code>from dateutil.relativedelta import relativedelta
from dateutil.rrule import rrule
from dateutil import tz
from dateutil.parser import parse, isoparse

from datetime import datetime, timedelta
from dateutil.relativedelta import MO, TU, WE, TH, FR, SA, SU
from helper_functions import print_dts<jupyter_output><empty_output><jupyter_text>Recurring events - Check for updates at 23:00 every night- Pay bills on the first of the month- Meeting at noon on Mondays- Schedule releases on Tuesdays and Thursdays- School Monday to Friday between September and July<jupyter_code>from dateutil.rrule import rrule, rruleset, rrulestr
from dateutil.rrule import YEARLY, MONTHLY, WEEKLY, DAILY, HOURLY, MINUTELY, SECONDLY
def print_rr_elems(r, n=2):
    out = []
    for ii, dt in enumerate(r):
        if ii >= n:
            out.append('...')
            break
        out.append(str(dt))
    print(', '.join(out))<jupyter_output><empty_output><jupyter_text>`dateutil.rrule``rrule` is an implementation of *recurrence rules*. [RFC 5545](https://tools.ietf.org/html/rfc5545): "Internet Calendaring and Scheduling Core Object Specification"- Replaced [RFC 2445](https://tools.ietf.org/html/rfc2445) in Sept. 2009 ![Top level ToC for RFC 5545, illustrating that it's 170 pages!](images/RFC5545-toc.png)  <NAME> Birthday<jupyter_code><div style="display: flex; justify-content: space-around; width: 95%">
<div>
<img src="images/pat_morita_pd.jpg"
     alt="<NAME> and <NAME>, Public Domain photo"
     style="width: 450px; margin-top: -15px; order: 2"/>
</div>
<div style="flex-grow: 1; order: -1;"><jupyter_output><empty_output><jupyter_text>rr = rrule(freq=YEARLY, bymonth=6, bymonthday=28,   June 6 every year           dtstart=datetime(1932, 6, 28),           Starting in 1938           until=datetime(2005, 11, 24))            Ending in 2005print_rr_elems(rr)<jupyter_code><div class="fragment fade-in"><jupyter_output><empty_output><jupyter_text>... falling on a Monday<jupyter_code>rr = rr.replace(byweekday=MO)
print_rr_elems(rr)<jupyter_output>1937-06-28 00:00:00, 1943-06-28 00:00:00, ...<jupyter_text><jupyter_code>### ... during the Vietnam War<jupyter_output><empty_output><jupyter_text>rr.between(datetime(1955, 11, 1),           datetime(1975, 4, 30))<jupyter_code></div><jupyter_output><empty_output><jupyter_text>RRULE componentsFundamental elements of an `rrule` are:- **`dtstart`**: The start point of the recurrence (this is similar to a phase)- **`freq`**: The *units* of the fundamental frequency of the recurrence. It takes the values `YEARLY`, `MONTHLY`, `WEEKLY`, `DAILY`, `HOURLY`, `MINUTELY`, `SECONDLY`- **`interval`**: The fundamental frequency of the recurrence, in units of `freq`. If unspecified, this is 1.<jupyter_code>from itertools import zip_longest

def print_rrs(rrs, names=None, fmt_str='%Y-%m-%d %H:%M'):
    
    n = len(rrs)
    section_width = 80 // n
    base_fmt_str = '|'.join(['{{:^{}}}'.format(section_width)] * n)
    
    if names is not None:
        name_str = base_fmt_str.format(*names)
        print(name_str)
        print('-' * len(name_str))
    
    for rr_dates in zip_longest(*rrs, fillvalue=None):
        strs = [dt.strftime(fmt_str) if dt else '-' for dt in rr_dates]
        print(base_fmt_str.format(*strs))    
    
    
hourly = rrule(freq=HOURLY, interval=1, dtstart=datetime(2016, 7, 18, 9),  count=3)
interval_2 = hourly.replace(interval=2)
dtstart_rr = hourly.replace(dtstart=datetime(2016, 7, 18, 10))

print_rrs([hourly, dtstart_rr, interval_2], ['Hourly', 'dtstart', 'interval=2'])<jupyter_output>Hourly          |         dtstart          |        interval=2        
--------------------------------------------------------------------------------
     2016-07-18 09:00     |     2016-07-18 10:00     |     2016-07-18 09:00     
     2016-07-18 10:00     |     2016-07-18 11:00     |     2016-07-18 11:00     
     2016-07-18 11:00     |     2016-07-18 12:00     |     2016-07-18 13:00<jupyter_text>`byxxx` rules`byxxx` rules serve to modify the frequency of the recurrence in some way. The supported rules are `bymonth`, `bymonthday`, `byyearday`, `byweekno`, `byweekday`, `byhour`, `byminute` and `bysecond`, `bysetpos` and `byeaster`. - `byxxx` rules greater than or equal to `freq` are constraints and (generally) reduce the frequency of the recurrence:<jupyter_code># Base is DAILY, but by restricted to Tuesdays in November
list(rrule(DAILY, bymonth=11, byweekday=(TU, ),
           dtstart=datetime(2015, 1, 1, 12), count=5))<jupyter_output><empty_output><jupyter_text>- `byxxx` rules less than `freq` will generally *increase* the frequency of the recurrence:<jupyter_code>list(rrule(MONTHLY, bymonthday=(1, 15, 30),
           dtstart=datetime(2015, 1, 16, 12, 15), count=4))<jupyter_output><empty_output><jupyter_text>Take note of the fact that the "30" is *ignored* for February  Limiting rulesIf otherwise unspecified, recurrences can be generated to infinity (or at least until Python can't represent the date anymore). The two ways to specify a termination point *as part of the rule* are with the mutually exclusive `count` and `until` arguments.* `count` terminates the rule after a specific number of instances have been generated<jupyter_code># The next 2 instances where the 4th of July falls on a Friday
list(rrule(YEARLY, bymonth=7, bymonthday=4, byweekday=FR,
           dtstart=datetime(2016, 7, 5), count=2))<jupyter_output><empty_output><jupyter_text>* `until` terminates the rule on a specific date:<jupyter_code># The Friday the 13ths before January 1st, 2018
list(rrule(MONTHLY, bymonthday=13, byweekday=FR,
           dtstart=datetime(2016, 7, 17, 12), until=datetime(2018, 1, 1)))<jupyter_output><empty_output><jupyter_text>Make note of the fact that the unlimited nature of recurrence rules makes it clear that they are lazily constructed.  Using rrulesIt is also possible to retrieve specific subsets of the recurrence, e.g. the first recurence `after` a given date:<jupyter_code>rr = rrule(DAILY, byhour=(9), byweekday=range(0, 5), dtstart=datetime(2016, 7, 1))

rr.after(datetime.now())      # The beginning of the next weekday<jupyter_output><empty_output><jupyter_text>You can retrieve the most recent recurrence *before* a given date:<jupyter_code>rr.before(datetime(2017, 3, 14))   # Apparently this is a Saturday<jupyter_output><empty_output><jupyter_text>You can also get all the recurrences *between* two dates:<jupyter_code># byeaster is a non-standard extension in dateutil that calculates a day
# offset from easter. This rule generates all the easters between 1995 and 2000.
rr = rrule(YEARLY, byeaster=0, dtstart=datetime(1990, 1, 1))

rr.between(datetime(1995, 1, 1), datetime(2000, 1, 1))<jupyter_output><empty_output><jupyter_text>Missing datetimes<jupyter_code>rr = rrule(freq=MONTHLY, dtstart=datetime(2019, 1, 1), bymonthday=31, count=5)
for dt in rr:
    print(dt)<jupyter_output>2019-01-31 00:00:00
2019-03-31 00:00:00
2019-05-31 00:00:00
2019-07-31 00:00:00
2019-08-31 00:00:00<jupyter_text>**RFC 5545**, §3.3.10:```Recurrence rules may generate recurrence instances with an invalid date (e.g., February 30)or nonexistent local time (e.g., 1:30 AM on a day where the local time is moved forward byan hour at 1:00 AM). Such recurrence instances MUST be ignored and MUST NOT be counted aspart of the recurrence set.```  Relativedelta<jupyter_code>from dateutil.relativedelta import relativedelta
def end_of_month(dtstart, interval=1):
    delta = relativedelta(day=31)
    dt = dtstart + delta    # Start with the end of the first month
    
    delta = relativedelta(months=1, day=31)  # "End of next month"
    while True:
        yield dt
        dt += delta
print_rr_elems(end_of_month(datetime(2018, 1, 1)), n=4)<jupyter_output>2018-01-31 00:00:00, 2018-02-28 00:00:00, 2018-03-31 00:00:00, 2018-04-30 00:00:00, ...<jupyter_text>SKIP option **RFC 7529**: "Non-Gregorian Recurrence Rules in the Internet Calendaring and Scheduling Core Object Specification"Section 4.1, adds:    - `SKIP=OMIT` - Move on to the next rule  (default)- `SKIP=BACKWARD` - Fall back to the last valid date (doesn't have to match the rule)- `SKIP=FORWARD` - Return the next valid date  Example:DTSTART:20141231T000000RRULE:FREQ=MONTHLY;COUNT=4;RSCALE=GREGORIAN;SKIP=BACKWARDGenerates: 2014-12-31, 2015-01-31, 2015-02-28, 2015-03-31  ![Pull request implementing Skip option](images/skip-option.png)  rrulesetSome recurrences cannot be expressed in a single rrule. `rruleset` allows you to combine `rrule`s and `datetime`s to generate an arbitrary recurrence schedule. `rruleset` interface:- `rruleset.rrule()`: Add a recurrence rule to the set- `rruleset.exrule()`: *Subtract* a recurrence rule from the set- `rruleset.rdate()`: Add a specific datetime to the set- `rruleset.exdate()`: *Subtract* a specific datetime from the set  Bus schedule<jupyter_code>dtstart = datetime(2016, 11, 1, 0, 0)    # The base date
WEEKDAYS = (MO, TU, WE, TH, FR);    WEEKENDS = (SA, SU)
bus_schedule = rruleset()
# During the week, it comes every hour on the 37 from 6:37AM to 10:37PM...
weekday_schedule = rrule(DAILY, byweekday=WEEKDAYS,
                         byhour=range(6, 22), byminute=37, dtstart=dtstart)
bus_schedule.rrule(weekday_schedule)       # Add an rrule to the rule set
# ..except after 6, when it comes every other hour - so exclude 7:37PM and 9:37PM!
weeknight_schedule = weekday_schedule.replace(byhour=(19, 21))
bus_schedule.exrule(weeknight_schedule)
# During the weekend, it comes every hour on the :07, from 8AM to 7PM
weekend_schedule = rrule(DAILY, byweekday=WEEKENDS,
                         byhour=range(8, 20), byminute=7, dtstart=dtstart)
bus_schedule.rrule(weekend_schedule)<jupyter_output><empty_output><jupyter_text>`rdate` and `exdate`<jupyter_code># But on November 8th, 2016, politicians have arranged for busses to undergo
# "service", so the normal bus schedule is canceled that day
exdates = bus_schedule.between(datetime(2016, 11, 8, 0), datetime(2016, 11, 9))
for exdate in exdates:
    bus_schedule.exdate(exdate)
# And in its place they've added one bus at 4:32 AM
bus_schedule.rdate(datetime(2016, 11, 8, 4, 37))

# And one at 7:49 PM
bus_schedule.rdate(datetime(2016, 11, 8, 19, 49))<jupyter_output><empty_output><jupyter_text>And display the schedule:<jupyter_code>from IPython.display import display, HTML
from itertools import groupby
from operator import itemgetter
import pandas as pd

style_sheet = """
<style type="type/css">

.dataframe th {
    font-size: 18px;
    background #cacaca;
}

.reveal .dataframe th {
    font-size: 18px;
    background #cacaca;
}

.dataframe th {
    font-size: 16px;
}

.reveal .dataframe th {
    font-size: 16px;
}

</style>
"""

def print_bus_schedule(bus_list):
    gby = groupby(bus_list, key=datetime.date)
    l = [(g, list(map(datetime.time, v))) for g, v in gby]

    labels, vals = zip(*l)
    labels = [(dt.strftime('%Y-%m-%d'), dt.strftime('%a')) for dt in labels]
    df = pd.DataFrame(list(vals), index=pd.MultiIndex.from_tuples(labels)).T
    html_out = (style_sheet)
    x = df.to_html(index=False)
    return html_out + x

bus_list = bus_schedule.between(datetime(2016, 11, 7), datetime(2016, 11, 14))
o = print_bus_schedule(bus_list)
HTML(o)
import qrcode
qr = qrcode.QRCode()
qr.add_data(
"""<NAME>
https://github.com/pganssle

pgp key:
6B49 ACBA DCF6 BD1C A206
67AB CD54 FCE3 D964 BEFB
""")
qr.make()
qr.make_image().save('images/key_qr.png')<jupyter_output><empty_output><jupyter_text>More information- Github: https://github.com/dateutil/dateutil- Check [open issues with the "help wanted" label](https://github.com/dateutil/dateutil/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)- Documentation: https://dateutil.readthedocs.io/en/stable/<jupyter_code><div style="display:flex;
            flex-direction: column;
            justify-content: space-between;
            margin-top: 2.5em;
            padding-left: 20%;
            padding-right: 20%;">
    <div class="pganssle-info-header"
         style="display:flex;
                justify-content: space-between;
                background: lightgray">
        <div>
            <b><NAME></b>
        </div>
        <div style="text-align:  right;">
            <a href="https://ganssle.io">https://ganssle.io</a><br>
            Blog: <a href="https://blog.ganssle.io">https://blog.ganssle.io</a>
        </div>
    </div>    
    <div class="pganssle-info-body"
         style="display:flex;
                justify-content: space-between">
        <div style="font-family: monospace;
                    display:flex;
                    align-items: center;">pgp key<br>
            <KEY>A DCF6 BD1C A206<br>
            67AB CD54 FCE3 D964 BEFB
        </div>
        <div>
            <img src="images/key_qr.png"
                 width="300px"
                 style="border: 0px; box-shadow: none">
        </div>
    </div>
</div><jupyter_output><empty_output><jupyter_text>``rrulestr``The iCalendar spec originally refers to a specific string format for specifying recurrence rules.`rrule`s can also be generated from these string using the `rrulestr` class:<jupyter_code># DST start and stop transition rules for Pacific Time
dst_start = rrulestr('DTSTART:19671029T020000;\n'
                     'FREQ=YEARLY;BYDAY=1SU;BYMONTH=4')
dst_end = rrulestr('DTSTART:19671029T020000;\n'
                   'FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10')
rrset = rruleset()
rrset.rrule(dst_start)
rrset.rrule(dst_end)

rrset.between(datetime(2016, 1, 1), datetime(2018, 1, 1))<jupyter_output><empty_output><jupyter_text>`str(rrule)`You can generate `RRULE` strings from `rrule` objects as well:<jupyter_code># This string should be compatible with other applications using the iCalendar spec
print(rrule(YEARLY, byyearday=180, byhour=(1, 4, 12), dtstart=datetime(2014, 9, 13)))
# Note that the BYEASTER directive is the only RFC-incompatible output
print(rrule(YEARLY, byeaster=0, dtstart=datetime(1990, 1, 1), count=14))<jupyter_output>DTSTART:19900101T000000
RRULE:FREQ=YEARLY;COUNT=14;BYEASTER=0
{'fraction_non_alphanumeric': 0.12796408025817316, 'fraction_numerical': 0.0573874000280623, 'mean_word_length': 4.071149057274991, 'pattern_counts': {'":': 0, '<': 128, '<?xml version=': 0, '>': 129, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '598556', 'license': 'Apache-2.0', 'n_tokens_mistral': 5550, 'n_tokens_neox': 4843, 'n_words': 1334, 'path': 'dateutil_presentation.ipynb', 'repo_name': 'pganssle-talks/boston-python-rrules', 'size': '14289'}