<jupyter_start><jupyter_text>Moving on to bigger and better things:PRAW: Python Reddit API Wrapper.  >- Let's me access Reddit from Python and get submissions, comments, etc.  while abiding to their terms of service and stuff.<jupyter_code>import praw
reddit = praw.Reddit('Gibberish v.2')<jupyter_output><empty_output><jupyter_text>A recent post... was quite interesting.http://www.reddit.com/r/sweden/comments/301sqr/dodetot_%C3%A4ror_fof%C3%B6ror_lolitote/People were posting things like:  >VoVadod fofanon sosa dodu jojusostot omom momigog, dodinon lolilollola soslolynona ? DoDu bobehohövoveror vovetota atottot jojagog totogog exoxamomenon i totopoppopenon avov mominon koklolasossos i SoSärorsoskokiloldoda opoperoratotiononsosgogrorupoppopenon, ocochoh jojagog hoharor vovaroritot inonvovololvoveroradod i etottot foflolerortotalol hohemomloligoga rorädoderor momotot dodanonsoskokenon, ocochoh jojagog hoharor övoveror totrorehohunondodrora bobekokroräfoftotadode momorordod. JoJagog äror utotbobiloldodadod i gogororilollola kokrorigogfoförorinongog ocochoh jojagog äror dodenon bobäsostota poproricockoksoskokytottotenon i hohelola nonorordodenonsos voväpopnonadode sostotyrorkokoror. DoDu äror inongogetot foföror momigog utotanon bobarora etottot anonnonatot momålol. JoJagog kokomommomeror atottot totororkoka utot dodigog momedod poprorecocisosionon vovarorsos lolikoke aloldodrorigog totidodigogarore soskokådodatotsos popå dodenonnona jojorordod , momärorkok momino  I made this program because of that thread. Time to translate every comment Reddit can give me from it!<jupyter_code>#Get the submission from above
submission = reddit.get_submission(submission_id='301sqr',comment_limit = None)
#Get more comments from submission
submission.replace_more_comments(limit=20, threshold=3)
forest_comments = submission.comments
#Flatten comments
flat_comments = praw.helpers.flatten_tree(forest_comments)
len(flat_comments)
commentList = []
for comment in flat_comments:
    gib = Gibberish(comment.body)
    gib.degibberish()
    gib.translate("English")
    commentList.append([comment.body, gib.get_sentence()])
len(commentList)
#From http://nbviewer.ipython.org/gist/jdfreder/6734825  
#Render markdown in the notebook 

from IPython.nbconvert.utils.pandoc import pandoc
from IPython.display import HTML, Javascript, display

from IPython.nbconvert.filters import citation2latex, strip_files_prefix, \
                                     markdown2html, markdown2latex

def pandoc_render(markdown):
    """Render Pandoc Markdown->LaTeX content."""
    
    ## Convert the markdown directly to latex.  This is what nbconvert does.
    #latex = pandoc(markdown, "markdown", "latex")
    #html = pandoc(markdown, "markdown", "html", ["--mathjax"])
    
    # nbconvert template conversions
    html = strip_files_prefix(markdown2html(markdown))
    latex = markdown2latex(citation2latex(markdown))
    display(HTML(data="<div style='display: inline-block; width: 30%; vertical-align: top;'>" \
                 "<div style='background: #AAFFAA; width: 100%;'>NBConvert Latex Output</div>" \
                 "<pre class='prettyprint lang-tex' style='background: #EEFFEE; border: 1px solid #DDEEDD;'><xmp>" + latex + "</xmp></pre>"\
                 "</div>" \
                 "<div style='display: inline-block; width: 2%;'></div>" \
                 "<div style='display: inline-block; width: 30%; vertical-align: top;'>" \
                 "<div style='background: #FFAAAA; width: 100%;'>NBViewer Output</div>" \
                 "<div style='display: inline-block; width: 100%;'>" + html + "</div>" \
                 "</div>"))
    javascript = """
    $.getScript("https://google-code-prettify.googlecode.com/svn/loader/run_prettify.js");
"""
    display(Javascript(data=javascript))

def notebook_render(markdown):
    javascript = """
var mdcell = new IPython.MarkdownCell();
mdcell.create_element();
mdcell.set_text('""" + markdown.replace("\\", "\\\\").replace("'", "\'").replace("\n", "\\n") + """');
mdcell.render();
$(element).append(mdcell.element)
.removeClass()
.css('left', '66%')
.css('position', 'absolute')
.css('width', '30%')
mdcell.element.prepend(
    $('<div />')
    .removeClass()
    .css('background', '#AAAAFF')
    .css('width', '100 %')
    .html('Notebook Output')

);
container.show()
"""
    display(Javascript(data=javascript))

    
def pandoc_html_render(markdown):
    """Render Pandoc Markdown->LaTeX content."""
    
    # Convert the markdown directly to latex.  This is what nbconvert does.
    latex = pandoc(markdown, "markdown", "latex")
    
    # Convert the pandoc generated latex to HTML so it can be rendered in 
    # the web browser.
    html = pandoc(latex, "latex", "html", ["--mathjax"])
    display(HTML("<div style='display: inline-block; width: 40%;'>" + html + "</div>"))
    return html
    
def compare_render(markdown):
    notebook_render(markdown)
    pandoc_render(markdown)

for (x, y) in commentList:
    s=''
    s+= '###Original comment: \n'
    s+= '>' + x + '  \n'
    s+='\n##Translated: \n'
    s+= ' >' + y
    s+= '\n'
    pandoc_html_render(s)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1272269558481797, 'fraction_numerical': 0.009876065065840434, 'mean_word_length': 4.313786008230453, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '299399', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1815, 'n_tokens_neox': 1753, 'n_words': 405, 'path': 'Why am I up so late.ipynb', 'repo_name': 'leftea/Gibberish', 'size': '5164'}