<jupyter_start><jupyter_text>This notebook implements a pre-trained sentiment analysis pipeline including a regex pre-processing step, tokenization, n-gram computation, and logistic regreression model as a RESTful API.<jupyter_code>import cPickle
import json

import pandas as pd
import sklearn
import requests<jupyter_output><empty_output><jupyter_text>Import the trained model<jupyter_code>resp = requests.get("https://raw.githubusercontent.com/crawles/gpdb_sentiment_analysis_twitter_model/master/twitter_sentiment_model.pkl")
resp.raise_for_status()
cl = cPickle.loads(resp.content)<jupyter_output><empty_output><jupyter_text>Import data pre-processing function<jupyter_code>def regex_preprocess(raw_tweets):
    pp_text = pd.Series(raw_tweets)
    
    user_pat = '(?<=^|(?<=[^a-zA-Z0-9-_\.]))@([A-Za-z]+[A-Za-z0-9]+)'
    http_pat = '(https?:\/\/(?:www\.|(?!www))[^\s\.]+\.[^\s]{2,}|www\.[^\s]+\.[^\s]{2,})'
    repeat_pat, repeat_repl = "(.)\\1\\1+",'\\1\\1'

    pp_text = pp_text.str.replace(pat = user_pat, repl = 'USERNAME')
    pp_text = pp_text.str.replace(pat = http_pat, repl = 'URL')
    pp_text.str.replace(pat = repeat_pat, repl = repeat_repl)
    return pp_text<jupyter_output><empty_output><jupyter_text>Setup the API Jupyter Kernel Gateway utilizes a global REQUEST JSON string that will be replaced on each invocation of the API.<jupyter_code>REQUEST = json.dumps({
    'path' : {},
    'args' : {}
})<jupyter_output><empty_output><jupyter_text>Compute sentiment using trained model and serve using POST Using the kernel gateway, a cell is created as an HTTP handler using a single line comment. The handler supports common HTTP verbs (GET, POST, DELETE, etc). For more information, view the docs.<jupyter_code># POST /polarity_compute
req = json.loads(REQUEST)
tweets = req['body']['data']
print(cl.predict_proba(regex_preprocess(tweets))[:][:,1])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14578947368421052, 'fraction_numerical': 0.005789473684210527, 'mean_word_length': 5.717314487632509, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '84359', 'license': 'MIT', 'n_tokens_mistral': 683, 'n_tokens_neox': 661, 'n_words': 144, 'path': 'jupyter_gateway_deployment/text-analytics-service-pcf.ipynb', 'repo_name': 'FeherBalazs/text-analytics-service', 'size': '1900'}