<jupyter_start><jupyter_text>Exploring & Explaining churn prediction modelsChurn prediction is the task of identifying users that are likely to stop using a service, product or website. In this tutorial, you will learn how to: Explore & Evaluate predictions made by the model* Explore predictions made by this model to gain confidence in the model.* Understanding why the model made the predictions that it did make.* Make a churn report by segmenting users based on the their reasons for churn.* Evaluate the model and compare it with a baseline model.  Let's get started!<jupyter_code>import graphlab as gl
import datetime
gl.canvas.set_target('ipynb') # make sure plots appear inline<jupyter_output>A newer version of GraphLab Create (v2.0.1) is available! Your current version is v2.0.

You can use pip to upgrade the graphlab-create package. For more information see https://turi.com/products/create/upgrade.
[INFO] graphlab.cython.cy_server: GraphLab Create v2.0 started. Logging: /tmp/graphlab_server_1468133144.log<jupyter_text>Load previously saved dataIn the previous notebooks, we had saved the data & models in a binary format. Let us try and load them back.<jupyter_code>interactions_ts = gl.TimeSeries("data/user_activity_data.ts/")
users = gl.SFrame("data/users.sf/")
model = gl.load_model("data/churn_model.mdl")
(train, valid) = gl.churn_predictor.random_split(interactions_ts, user_id = 'CustomerID', fraction = 0.9, seed = 12)
churn_period_oct =  datetime.datetime(year = 2011, month = 10, day = 1)<jupyter_output><empty_output><jupyter_text>Interactive view to explore the model<jupyter_code>v = model.views.overview(train, churn_period_oct, valid, user_data=users)
v.show()<jupyter_output>[WARNING] graphlab.toolkits.churn_predictor._churn_predictor: This feature is currently in beta. Please use with caution and not in mission-critical applications. For feedback or suggestions on this feature, please e-mail <EMAIL>.<jupyter_text>What are the key features that impact churn?<jupyter_code>importance = model.get_feature_importance()
print "What are the top 5 factors that impact predictions?"
print "----------------------------------------------------"
print '\n'.join(["%s. %s" % (i+1, x) for i,x in enumerate(importance['description'][0:5])])<jupyter_output>What are the top 5 factors that impact predictions?
----------------------------------------------------
1. Days since most recent event
2. Sum of "Quantity" in the last 60 days
3. Sum of "Quantity" in the last 90 days
4. Days since the first event in the last 90 days
5. 90 day trend in the number of events<jupyter_text>Segmenting groups of users with similar churn explanations<jupyter_code>report = model.get_churn_report(interactions_ts, user_data=users)
report
report['num_users'].show()<jupyter_output><empty_output><jupyter_text>What does a segment look like?<jupyter_code>segment = report[report['segment_id'] == '2'][0]
print ""
print "Segment 2"
print "---------------------------------------"
print 'Segment size      : %.2f %% of users' % segment["num_users_percentage"]
print 'Churn probability : %s' % segment["avg_probability"]
print ""
print "Characteristics of users in segment 2?"
print "-----------------------------------------------"
print "\n".join(['%s. %s' % (i + 1, x) for i, x in enumerate(segment["explanation"])])<jupyter_output>Segment 2
---------------------------------------
Segment size      : 9.01 % of users
Churn probability : 0.560969559738

Characteristics of users in segment 2?
-----------------------------------------------
1. No "Quantity" events in the last 21 days
2. Sum of "Quantity" is greater than (or equal to) 109.50 each day in the last 90 days
3. Feature 'user_type' is 'extra-heavy'
4. Less than 2.50 days with an event in the last 90 days
5. Maximum events in a day the last 90 days less than 46.50
6. No "UnitPrice" events in the last 14 days<jupyter_text>Understanding individual predictions: Why did the model make a prediction?<jupyter_code>particular_user = valid[valid['CustomerID'] == '16200']
particular_user
explanations = model.explain(particular_user, user_data=users)
print ""
print "Model explanations"
print "---------------------------------------"
print 'Customer ID       : %s' % explanations["CustomerID"]
print 'Churn probability : %s' % explanations["probability"]
print ""
print "Why did the model make this prediction?"
print "---------------------------------------"
print "\n".join(['%s. %s' % (i + 1, x) for i, x in enumerate(explanations["explanation"][0])])<jupyter_output>Model explanations
---------------------------------------
Customer ID       : ['16200']
Churn probability : [0.44427892565727234]

Why did the model make this prediction?
---------------------------------------
1. Greater than (or equal to) 89.50 events in the last 60 days
2. Less than 187.50 days since most recent event
3. Maximum events in a day the last 90 days greater than (or equal to) 46.50
4. Less than 2.50 days with an event in the last 90 days
5. Feature 'user_type' is 'extra-heavy'
6. Average number of events in the last 21 days greater than (or equal to) 106.50
{'fraction_non_alphanumeric': 0.172689978511428, 'fraction_numerical': 0.03184215667122485, 'mean_word_length': 3.3316412859560067, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '280090', 'license': 'Apache-2.0', 'n_tokens_mistral': 1566, 'n_tokens_neox': 1425, 'n_words': 568, 'path': 'dss-2016/churn_prediction/churn-tutorial-explore-explain.ipynb', 'repo_name': 'voltek62/tutorials', 'size': '5129'}