<jupyter_start><jupyter_text>Emailed Digest of Clubhouse.IO story statuses**This is provided as a base; it is suspected that you'll need to alter for your needs.**Provided without warranty of any kind and released under MIT License by Junction Applications.[Clubhouse.IO]((https://clubhouse.io/)) is in their words >"Where software teams do their best work. Clubhouse is the first project management platform for software development that brings everyone on every team together to build better products." I've come to depend on it for a number of projects, and general task managmement. I find it a joy to use and easy to integrate with; as such providing this Notebook as a short demo of how one might use it to send status updates for particular projects to interested parties. This notebook outlines how to connect to your [Clubhouse.IO](https://clubhouse.io/) organization using their [API](https://clubhouse.io/api/rest/v3/), run a query to get the stories needed, build an email and send it. You'll probably want to either extract out the bits needed and run as a proper scheduled script, or by using something like [PaperMill](https://github.com/nteract/papermill) to parameterize and schedule.This is a lengthy Notebook as it contains all the Clubhouse connection code which would normally be in it's own file and imported. There is also a function to send email via smtp here which may not suit your needs; you'll need to supply your own mail server if you are using it. RequirementsYou may require some libraries to be installed to make this work. Some known ones:- [requests](https://pypi.org/project/requests/)- [parse](https://pypi.org/project/parse/)  Configurable setttings<jupyter_code>report_to_run = 'recent'  # see the report_params dictionary id's in the next section for options
default_send_to_address = "<EMAIL>"
smtp_mail_server = "smtp.yourmailserver.law"
# Enter your own token below
# https://help.clubhouse.io/hc/en-us/articles/205701199-Clubhouse-API-Tokens
clubhouse_api_token = "<PASSWORD>-7890-123456789012"  # os.getenv('CLUBHOUSE_TOKEN')<jupyter_output><empty_output><jupyter_text>Report parametersThis is where you'd set up any reports you want to run. Create new dictionary items as necessary in the report_params dictionary. The idea here is that we have a number of people who may want a status update. The parameters are stored in a dictionary, with ability to add a new report as time goes on. If this got out of hand, I'd throw all of this in a database somewhere, but for illustration purposes, it is stored here.`reporting_states` is a tuple of story workflow states you want to report on, in the order you want them to appear in the email. This allows you to restrict something like "Backlog" from appearing in certain reports. You'll need to alter this in the default_params to suit your organization.<jupyter_code># let's manage some imports and set a date format Clubhouse likes
from datetime import datetime, timezone, timedelta
now = datetime.now(timezone.utc)
ch_date_format = '%Y-%m-%d'

# complete_days is the number of days to go back to include in the "Complete" status
# clubhouse_query is any valid query string outlined here:
# https://help.clubhouse.io/hc/en-us/articles/360018875792-Searching-in-Clubhouse-Text-Strings
report_params = {"hold": {"clubhouse_query":"label:onhold",
                          "complete_days": 10,
                          "email_subject": "Items on Hold"},
                 # recent in this case is anything updated in the last 14 days
                 "recent": {"clubhouse_query":f"updated:{now - timedelta(days=14):{ch_date_format}}..*",
                            "complete_days": 10,
                            "email_subject": "Work Status with Recent Updates"},
                 # add additional reports here. 
                 
                }

default_params = {"send_to_addrs": [default_send_to_address, ],
                  "reporting_states": ('Backlog', 'Specify/Breakdown', 'Implementing', 'Implemented', 'Validating', 'Completed'),
                  "complete_days": 10,
                  "send_from_addr": '"Change this Friendly Name and Email" <<EMAIL>>',
                  "email_subject": "Clubhouse Story Status Digest"      
                 }

# shallow merge the two dictionaries (Python 3.5+)
# https://www.python.org/dev/peps/pep-0448/
run_params = {**default_params, **report_params[report_to_run]}<jupyter_output><empty_output><jupyter_text>Create a little mailer function Suggested this goes in its own file, and imported, but we're trying to build this notebook with everything needed. Use whatever method you like for sending mail, this one uses an smtp server that requires no authentication.<jupyter_code># following has been conflated from several StackOverflow posts. Apologies to original authors 
# for lack of credit, but this has been evolving for a few years. Use whatever you need to send
# an email (sendgrid or other for example), this is if you have a smtp server available.
import smtplib

from email.mime.application import MIMEApplication
from email.mime.image import MIMEImage
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.utils import COMMASPACE, formatdate


def send_mail(send_from, 
              send_to, 
              subject, 
              plain_text, 
              html_text, 
              server=smtp_mail_server):
    assert isinstance(send_to, list)

    msg = MIMEMultipart('related')
    msg['From'] = send_from
    msg['To'] = COMMASPACE.join(send_to)
    msg['Date'] = formatdate(localtime=True)
    msg['Subject'] = subject
    msg.preamble = 'This is a multi-part message in MIME format.'
    
    # alternative will contain plain text and html
    msg_alt = MIMEMultipart('alternative')
    msg.attach(msg_alt)
    
    msg_alt.attach(MIMEText(plain_text, 'plain'))
    msg_alt.attach(MIMEText(html_text, 'html', 'utf-8'))

    smtp = smtplib.SMTP(server)
    smtp.sendmail(send_from, send_to, msg.as_string())
    smtp.close()<jupyter_output><empty_output><jupyter_text>Clubhouse API Hooks This would also most likely be imported normally as a class but reproduced here and stripped down to the essenstials for this demo. It provides a number of Python wrappers to the api calls.<jupyter_code># Some Clubhouse.IO tools:
import os
import sys
import time
import requests

# CH_TOKEN = os.getenv('CLUBHOUSE_TOKEN')   # the normal way I'd store the token
CH_TOKEN = clubhouse_api_token
CH_BASE = 'https://api.clubhouse.io/'
CH_API_VER = 'api/v3/'


def url(entity, ver=None):
    """
    Returns a properly formatted url given module's constants
    entity is one of the Clubhouse entities (epics, labels, stories etc.)
    :param entity: the clubhouse entity (can be combined like search/stories)
    :param ver: allows override of the api version
    """
    if ver:
        ch_ver = ver
    else:
        ch_ver = CH_API_VER
    return f'{CH_BASE}{ch_ver}{entity}?token={CH_TOKEN}'


def entity_list(entity):
    """
    :return: a JSON packet listing of entities from Clubhouse
    """
    response = requests.get(url(entity))
    return response.json()


def project_list():
    """
    :return: a JSON packet of projects
    """
    return entity_list(entity='projects')


def epic_list():
    """
    :return: a JSON packet of epics
    """
    return entity_list(entity='epics')


def workflow_list():
    """
    :return: a JSON packet of workflow states
    """
    return entity_list(entity='workflows')


def search_stories(query, page_size=25):
    # query in the form of a proper Clubhouse query string defined:
    # https://help.clubhouse.io/hc/en-us/articles/360000046646-Search-Operators
    body_params = {'page_size': page_size, 'query': query}
    entity = 'search/stories'

    data = dict()

    first_page = requests.get(url(entity=entity), body_params).json()
    stories = first_page.get('data', None)
    next_page_token = first_page.get('next', None)
    total_stories = first_page.get('total', None)
    while_count = 0

    while next_page_token:
        while_count += 1
        # if we're calling more than this we've probably made a mistake
        # picking arbitrary 1000 limit hopefully keeping us under the Clubhouse 200/min rate
        if while_count * page_size > 1000:
            data.update({'warning': 'Excessive api calls resulted in truncated data set. '
                                    f'About {page_size*while_count} of {total_stories} returned'})
            raise ValueError(data)
            break
        # so we strip off that last / because the next page token 'next' url starts with a /
        np_url = f'{CH_BASE[:-1]}{next_page_token}&token={CH_TOKEN}'
        next_page = requests.get(np_url).json()
        stories.extend(next_page['data'])
        next_page_token = next_page.get('next', None)

    data.update({'data': stories, 'total': total_stories})
    return data


def get_story(id):
    entity = 'stories/{id}'.format(id=id)
    response = requests.get(url(entity))
    return response.json()<jupyter_output><empty_output><jupyter_text>The email generatorThese next bits manage getting all the pieces, creating the email and sending it.The cells below are of no particular split points. This code was converted from another script and was pasted in in bitesized chunks simply to make testing functionality a bit easier in Jupyter Notebook.<jupyter_code>import dateutil.parser
from parse import search

workflow_states = dict()
for wf in workflow_list()[0]["states"]:
    workflow_states[wf["id"]] = wf
    
# set colours for story types (couldn't find a way to get these other than to hardwire like this)
story_types = {'feature': {'colour': '#F7F4E8',},
               'chore': {'colour': '#F1F6FE',},
               'bug': {'colour': '#FCE8E8',}}

epics = dict()
for epic in epic_list():
    epics[epic['id']] = epic

projects = dict()
for project in project_list():
    projects[project['id']] = project
# Place the stories into dictionary with key of the epic name
# this is later thought to be not quite needed as refactoring
# happened, but we use this dict to loop on later.
stories = dict()
for story in search_stories(run_params["clubhouse_query"])['data']:
    stories[story['id']] = story
def remove_clipboard_links(text_with_links):
    # the description text has some links to the embedded images
    # this removes those for our email purposes.
    t = text_with_links
    str_sentinel = {'begin':"![Clipboard ", 'end':")"}
    clipboard_strings = search(str_sentinel["begin"]+'{}'+str_sentinel["end"], t)
    if clipboard_strings:
        for clipboard_str in clipboard_strings:
            t = t.replace(f'{str_sentinel["begin"]}{clipboard_str}{str_sentinel["end"]}',"")
    return t

                      
def nl2br(text_with_br):
    return text_with_br.replace('\n','<br/>')


def clean_story(text_to_clean):
    return nl2br(remove_clipboard_links(text_to_clean))


def get_story_type_color(story_type):
    return story_types[story_type]['colour']


def epic_phrase(epic_id):
    if epic_id:
        return f"on {epics[story['epic_id']]['name']}"
    else:
        return ''

                          
def tag_format(name, colour):
    return (f"<span style=\"font-size: 11px;font-weight: 400;line-height: 13px;font-family: 'Open Sans',Helvetica,Arial,sans-serif;"
                    f"display: inline-block;"
                    f"background:#ffffff;border-radius: 5px;box-shadow: 0 1px 0 rgba(0,0,0,.08);"
                    f"box-sizing: border-box;color:#333333;font-style: normal;margin: 3px 1px 0 0;"
                    f"max-width: 100%;outline:0;padding: 3px 8px 4px 5px;vertical-align: top; border-top:3px solid {colour};\""
                    f">{name}</span>&nbsp;&nbsp;")


def email_format(story):
    # NOTE TO READER
    # this was put together by a non css, non html email loving person and needed to 
    # work quickly and look good in Gmail. The following produces a terrible looking 
    # output in Outlook, but fine in O365 Outlook, and fine on the mobile app.
                          
    rstr = list()
    
    rstr.append(f"<div style=\"padding: 8px 8px 8px 10px; display:block;background-color:{get_story_type_color(story['story_type'])};border-left:4px solid {projects[story['project_id']]['color']};\">")
                 
    rstr.append(f"<div style=\"font-family: 'Open Sans',Helvetica,Arial,sans-serif;font-size: 10px;"
                "font-weight: 700;line-height: 12px;color:#809CC0;display: inline-block;"
                f"margin: 0 0 0 0;text-transform: uppercase;\">{story['story_type']} {story['id']} {epic_phrase(story['epic_id'])} in {projects[story['project_id']]['name']}</div>")

    if story['labels']:
        rstr.append("<div style=\"margin:3px 0 2px 0;\">")
    
    if story['blocker']:
        rstr.append(tag_format("BLOCKING", "#cc5856"))
                
    for tag in story['labels']:
        rstr.append(tag_format(tag["name"], tag["color"]))

    if story['labels']:
        rstr.append("</div>")
    rstr.append(f"<div style=\"margin:0 0 8px 0;font-family: 'Open Sans',Helvetica,Arial,sans-serif;font-size: 14px;line-height: 16px;\">{story['name']}</div>")
    rstr.append(f"<div style=\"margin:0 0;font-family: 'Open Sans',Helvetica,Arial,sans-serif;font-size: 12px;line-height: 14px;\">{clean_story(story['description'])}</div>")
    if story['completed_at']:
        c_date = dateutil.parser.parse(story['completed_at'])
        rstr.append(f"<div style=\"margin:0 0;font-family: 'Open Sans',Helvetica,Arial,sans-serif;font-size:10px;font-weight:700;line-height:12px;color:#809CC0;\">Marked as completed {c_date:{ch_date_format}}</div>")
    rstr.append("</div><hr style=\"height: 1px;color:#cccccc;background-color:#cccccc;border: none;\"/>")
    return ''.join(rstr)                          
reports = dict()
include_story = True
for story_id, story in stories.items():
    state = workflow_states[story["workflow_state_id"]]["name"]
    if (state == 'Completed'): 
        # we only want recently closed stories
        c_date = dateutil.parser.parse(story['completed_at'])
        include_story = now - c_date < timedelta(days=run_params["complete_days"])
    else:
        include_story = True
    if include_story:
        story['email_html'] = email_format(story)
        try:
            reports[state].append(story)
        except KeyError:
            reports[state] = [story,] 
formatted_html_email = list()

formatted_html_email.append("<html><body>")
formatted_html_email.append(f"<div style=\"font-size:12px; line-height:14px;margin:0 0 6px 0; border-top:2px solid #291E38; font-family: 'Open Sans',Helvetica,Arial,sans-serif;display:block;font-weight: 700;\">For Query:</div>")
formatted_html_email.append(f"<div style=\"font-family: 'Open Sans',Helvetica,Arial,sans-serif;font-size: 10px;"
                "font-weight: 700;line-height:12px;color:#809CC0;display: inline-block;margin:0 0 12px 0;"
                f"text-transform: uppercase;\">")
formatted_html_email.append(f"{run_params['clubhouse_query']}<br/>")
formatted_html_email.append(f"</div>")
for rs in run_params["reporting_states"]:
    
    if reports.get(rs, None):
        formatted_html_email.append(f"<div style=\"margin:0 0 12px 0; border-top:2px solid #291E38; font-family: 'Open Sans',Helvetica,Arial,sans-serif;display:block;font-weight: 700;\">{rs}")

        if rs == "Completed":
            formatted_html_email.append(f" - Last {run_params['complete_days']} days")
        formatted_html_email.append("</div>")

        for r in sorted(reports[rs], key=lambda k: k['position']):
            formatted_html_email.append(r["email_html"])

formatted_html_email.append("</body></html>")
send_mail(send_from=run_params['send_from_addr'],
          send_to=run_params['send_to_addrs'],
          subject=run_params['email_subject'],
          html_text=''.join(formatted_html_email),
          plain_text="Update from development.",
         )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11531025880885563, 'fraction_numerical': 0.014717804801995634, 'mean_word_length': 3.6239907727797003, 'pattern_counts': {'":': 14, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 11, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '420832', 'license': 'MIT', 'n_tokens_mistral': 5066, 'n_tokens_neox': 4770, 'n_words': 1443, 'path': 'ClubHouse Story Digest.ipynb', 'repo_name': 'junctionapps/3e-notebooks', 'size': '16080'}