<jupyter_start><jupyter_text>PyConversations: A Reddit-based ExampleThe following is a tutorial notebook that demonstrates how to use `pyconversations` with Reddit data.The first step will be to obtain some data. In order to do so, you will need to configure a personal application via your Reddit account's [App Preferences](https://www.reddit.com/prefs/apps). You'll want to set up a personal usage script. See the _Getting Access_ portion of [this blog](https://towardsdatascience.com/how-to-use-the-reddit-api-in-python-5e05ddfd1e5c) for additional instructions/visualization.<jupyter_code>from pprint import pprint

from pyconversations.convo import Conversation
from pyconversations.message import RedditPost<jupyter_output><empty_output><jupyter_text>Data SampleBefore demonstating how we can use `pyconversations` for pre-processing and analysis, we need to obtain a data sample. To do so, we'll be using a package called [praw](https://github.com/praw-dev/praw).<jupyter_code>import praw
# Private information
CLIENT_ID = ''  # this should be the 'personal use script' on your App Preferences
SECRET_TOKEN = ''  # this should be the 'secret' on your App Preferences
USER_AGENT = ''  # a custom name for your application for the User-Agent parameter in the request headers; gives a brief app description to Reddit
# configure a read-only praw.Reddit instance
reddit = praw.Reddit(
    client_id=CLIENT_ID,
    client_secret=SECRET_TOKEN,
    user_agent=USER_AGENT,
)

reddit, reddit.read_only
# obtain a sub-reddit of interest
sub_name = 'Drexel'
subreddit = reddit.subreddit(sub_name)

subreddit.title  # PRAW is lazy so won't request till we ask for an attribute

pprint(vars(subreddit))
# get the top submission via 'hot'
top_submission = list(subreddit.hot(limit=1))[0]
top_submission.title

pprint(vars(top_submission))
# get all the comments on this submission
all_comments = top_submission.comments.list()

print(len(all_comments))

all_comments[0].score
pprint(vars(all_comments[0]))<jupyter_output>17
{'_fetched': True,
 '_reddit': <praw.reddit.Reddit object at 0x10a9990d0>,
 '_replies': <praw.models.comment_forest.CommentForest object at 0x11eb1e610>,
 '_submission': Submission(id='omya9w'),
 'all_awardings': [],
 'approved_at_utc': None,
 'approved_by': None,
 'archived': False,
 'associated_award': None,
 'author': Redditor(name='Fun-Obligation5729'),
 'author_flair_background_color': None,
 'author_flair_css_class': None,
 'author_flair_richtext': [],
 'author_flair_template_id': None,
 'author_flair_text': None,
 'author_flair_text_color': None,
 'author_flair_type': 'text',
 'author_fullname': 't2_883n07bs',
 'author_is_blocked': False,
 'author_patreon_flair': False,
 'author_premium': False,
 'awarders': [],
 'banned_at_utc': None,
 'banned_by': None,
 'body': 'I have a room available from September 1st- August 31st at 3406 race '
         'street. It’s 775$ . Let me know if anyone’s interested.',
 'body_html': '<div class="md"><p>I have a room available from September 1s[...]<jupyter_text>Integration with `pyconversations`All that's left to do is plug our data directly into `pyconversations`!<jupyter_code># construct a conversation container
conv = Conversation()
# parse our root submission
cons_params = {
    'lang_detect': True,  # whether to enable the language detection module on the text
    'uid': top_submission.id,  # the unique identifier of the post
    'author': top_submission.author.name if top_submission.author is not None else None,  # name of user
    'created_at': RedditPost.parse_datestr(top_submission.created),  # creation timestamp
    'text': (top_submission.title + '\n\n' + top_submission.selftext).strip()  # text of post
}
top_post = RedditPost(**cons_params)
top_post
# add our post to the conversation
conv.add_post(top_post)
# which we can easily access via the .posts property, to verify inclusion
conv.posts
# iterate through comments and add them to the conversation
for com in all_comments:
    conv.add_post(RedditPost(**{
        'lang_detect': True,  # whethher to enable the language detection module on the txt
        'uid': com.id,  # the unique identifier of the post
        'author': com.author.name if com.author is not None else None,  # name of user
        'created_at': RedditPost.parse_datestr(com.created),  # creation timestamp
        'text': com.body.strip(),  # text of post
        'reply_to': {com.parent_id.replace('t1_', '').replace('t3_', '')}  # set of IDs replied to
    }))

len(conv.posts)<jupyter_output><empty_output><jupyter_text>Sub-Conversation Segmentation<jupyter_code># seperate disjoint conversations (there is likely just the one with a full query from the site...)
segs = conv.segment()

len(segs)<jupyter_output><empty_output><jupyter_text>(Detected) Language Distribution<jupyter_code>from collections import Counter

lang_dist = Counter([post.lang for post in conv.posts.values()])
lang_dist<jupyter_output><empty_output><jupyter_text>Conversation-Level RedactionUsing `Conversation.redact()` produces a thread that is cleaned of user-specific information. This is conversationally-scoped, so all usernames are first enumerated (either from author names or from in-text reference for Reddit and Twitter) and then user mentions (and author names) are replaced by `USER{\d}` where `{\d}` is the integer assigned to that username during the enumeration stage.Here's a demonstration:<jupyter_code># pre-redaction 
names = {post.author for post in conv.posts.values()}

len(names), names
# redaction step
conv.redact()
# post-redaction
names = {post.author for post in conv.posts.values()}
len(names), names<jupyter_output><empty_output><jupyter_text>Saving and Loading from the universal format<jupyter_code>import json
# saving a conversation to disk
# alternatively: save as a JSONLine file, where each line is a conversation!
json.dump(conv.to_json(), open('reddit_conv.json', 'w+'))
# reloading directly from the JSON
conv_reloaded = Conversation.from_json(json.load(open('reddit_conv.json')))
len(conv_reloaded.posts)<jupyter_output><empty_output><jupyter_text>Feature ExtractionThe remainder of this notebook exhibits some basic vectorization of features from conversations, posts, and users within this conversation. For more information, see the documentation for PyConversations.<jupyter_code>from pyconversations.feature_extraction import ConversationVectorizer
from pyconversations.feature_extraction import PostVectorizer
from pyconversations.feature_extraction import UserVectorizer
convs = True
# convs = False

users = True
# users = False

posts = True
# posts = False

# normalization = None
# normalization = 'minmax'
# normalization = 'mean'
normalization = 'standard'

# cv = ConversationVectorizer(normalization=normalization, agg_user_fts=users, agg_post_fts=posts, include_source_user=True)
pv = PostVectorizer(normalization=normalization, include_conversation=convs, include_user=users)
# uv = UserVectorizer(normalization=normalization, agg_post_fts=posts)
# cv.fit(conv=conv_reloaded)
pv.fit(conv=conv_reloaded)
# uv.fit(conv=conv_reloaded)
# cxs = cv.transform(conv=conv_reloaded)
pxs = pv.transform(conv=conv_reloaded)
# uxs = uv.transform(conv=conv_reloaded)

# pprint(cxs.shape)
pprint(pxs.shape)
# pprint(uxs.shape)<jupyter_output>(18, 3317)
{'fraction_non_alphanumeric': 0.10071252397917238, 'fraction_numerical': 0.007673335160317895, 'mean_word_length': 5.918483412322275, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '576693', 'license': 'MIT', 'n_tokens_mistral': 2339, 'n_tokens_neox': 2281, 'n_words': 675, 'path': 'tutorial/Reddit.ipynb', 'repo_name': 'hunter-heidenreich/pyconversations', 'size': '7298'}