<jupyter_start><jupyter_text>Digging Into Discussion DataDiscussion data in Canvas might be used by a teacher for a participation grade, or to identify students who are less engaged and target interventions to them. Getting StartedInitialize access to the Canvas API and setup helper function(s).<jupyter_code>from canvasapi import Canvas, exceptions
import os
import pprint
from datetime import datetime
from dotenv import load_dotenv
load_dotenv()

API_URL = os.getenv("CANVAS_BASE_URL")
API_KEY = os.getenv("CANVAS_ACCESS_TOKEN")
OUTPUT_PATH = os.getenv("LOGIN_ACCESS_DATA_OUTPUT_PATH")

canvas = Canvas(API_URL, API_KEY)
accounts = canvas.get_accounts()

def flatten(pages):
    flat = []
    for p in pages:
        for p2 in p:
            flat.append(p2)
    return flat<jupyter_output><empty_output><jupyter_text>Get all courses so that we can look through them for discussion topics.<jupyter_code>courses = accounts[0].get_courses()
pprint.pprint([{u.name, u. id} for u in courses])<jupyter_output>[{'English I', 14}, {16, 'Algebra I'}]<jupyter_text>Discussion TopicsThe response is a nested set of pages; it is easier to work with them if they are "flattened" from a two dimensional array to a one dimensional array.<jupyter_code>topics = [c.get_discussion_topics() for c in courses]

flat_topics = flatten(topics)
pprint.pprint(flat_topics[0])<jupyter_output>DiscussionTopic(_requester=<canvasapi.requester.Requester object at 0x05482F10>, id=3, title=ಠ_ಠ, last_reply_at=2020-08-27T13:15:53Z, last_reply_at_date=2020-08-27 13:15:53+00:00, created_at=2020-08-27T13:15:53Z, created_at_date=2020-08-27 13:15:53+00:00, delayed_post_at=None, posted_at=2020-08-27T13:15:53Z, posted_at_date=2020-08-27 13:15:53+00:00, assignment_id=None, root_topic_id=None, position=None, podcast_has_student_posts=False, discussion_type=side_comment, lock_at=None, allow_rating=False, only_graders_can_rate=False, sort_by_rating=False, is_section_specific=False, user_name=<NAME>, discussion_subentry_count=0, permissions={'attach': True, 'update': True, 'reply': True, 'delete': True}, require_initial_post=None, user_can_see_posts=True, podcast_url=None, read_state=unread, unread_count=0, subscribed=False, attachments=[], published=True, can_unpublish=True, locked=False, can_lock=True, comments_disabled=False, author={'id': 14, 'display_name': '<NAME>', 'avatar_ima[...]<jupyter_text>Restructure the above responses in a JSON object that contains only the properties we care about.<jupyter_code>topic_data = [{'title': t.title, 'date': t.created_at_date, 'author_name': t.user_name, 'author_id': t.author['id'], 'message': t.message} for t in flat_topics]
[pprint.pprint(t.title) for t in flat_topics]<jupyter_output>'ಠ_ಠ'
'My Second Discussion Topic'
'Hi everyone!'<jupyter_text>Pandas: Word CountsIntroducing use of Pandas. Use it to do a count of spaces in the message, as a rough estimate of the number of words. Then remove the message from the new Pandas data frame object so that the message is not including in further output.<jupyter_code>import pandas as pd
df = pd.DataFrame(data=topic_data)

df['word_count'] = df['message'].str.count(' ') + 1

df = df.drop(['message'], axis=1)

pprint.pprint(df)<jupyter_output>title                      date  author_name  \
0                         ಠ_ಠ 2020-08-27 13:15:53+00:00  <NAME>   
1  My Second Discussion Topic 2020-08-27 13:12:28+00:00  <NAME>   
2                Hi everyone! 2020-08-27 13:11:38+00:00  <NAME>   

   author_id  word_count  
0         14           4  
1         13          99  
2         13         111<jupyter_text>Pandas: Write CSV FileWrite the data frame to a CSV file.<jupyter_code>df.to_csv(f"{OUTPUT_PATH}/discussions.csv")<jupyter_output><empty_output><jupyter_text>Open and read the file just to prove that it worked<jupyter_code>print(open(f"{OUTPUT_PATH}/discussions.csv", 'r').read())<jupyter_output>,title,date,author_name,author_id,word_count
0,à² _à² ,2020-08-27 13:15:53+00:00,<NAME>,14,4
1,My Second Discussion Topic,2020-08-27 13:12:28+00:00,<NAME>,13,99
2,Hi everyone!,2020-08-27 13:11:38+00:00,<NAME>,13,111
{'fraction_non_alphanumeric': 0.11487804878048781, 'fraction_numerical': 0.06341463414634146, 'mean_word_length': 5.603864734299517, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '438206', 'license': 'Apache-2.0', 'n_tokens_mistral': 1601, 'n_tokens_neox': 1440, 'n_words': 358, 'path': 'experimental/canvas/explorations-py/login-access-data-exploration/discussions.ipynb', 'repo_name': 'markramonDL/LMS-Toolkit', 'size': '4140'}