<jupyter_start><jupyter_text>Why and what is [Jupyter](http://jupyter.org/)? easy to explore from the notebook and easy to share and visualize. Can be used with R, Julia, Python [Getting raw data](raw)[Communicating with Apis](apis)[Machine Learning](ml) Data-mining and Introduction to Machine Learning using Python***<jupyter_code>from IPython.display import Image
from IPython.core.display import HTML 
import matplotlib.pyplot as plt
import numpy as np

## plotting style
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = 16, 9
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Data-mining, data-crunching, data-munging, data-wrangling. It all involves certain activities like data cleaning, data transformation, data preparation, data visualization.But it all starts with getting ***RAW DATA***   Getting raw data  To first to some validating and analysis of data, we need to get the raw data  A http request<jupyter_code>import requests

response = requests.get("http://www.blocket.se/stockholm?q=macbook air").text

sample_item = response.find("item_row")
sample_html_text = response[sample_item:sample_item+20]

print(sample_html_text)<jupyter_output>item_row item_row_fi<jupyter_text>Our research for buying a new computer Here we parse the html to a format which we can use<jupyter_code>import lxml.html

page = lxml.html.parse("http://www.blocket.se/stockholm?q=macbook")
# This is probably illegal. Do not use for reselling or redistributing. Do not sue me blocket!
items_data = []
for el in page.getroot().find_class("media item_row ptm pbm nmt"):
    links = el.find_class("item_link")
    images = el.find_class("item_image")
    prices = el.find_class("list_price")
    if links and images and prices and prices[0].text:
        items_data.append({"name": links[0].text,
                           "image": images[0].attrib['src'],
                           "price": int(prices[0].text.split(":")[0].replace(" ", ""))})
print(len(items_data))
items_data
prices = []
prices_for_retina = []
for item in items_data:
    prices.append(item['price'])
    if 'Retina' in item['name']:
        prices_for_retina.append(item['price'])    
plt.hist(prices, 5, histtype='bar')
plt.hist(prices_for_retina, 2, histtype='bar',color='blue')
plt.show()<jupyter_output><empty_output><jupyter_text>Investigate if the salse of Airbook was better in New York than in Stockholm<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('../data/raw/sample2.csv')

df.head()
# This table has 3 columns: Office, Year, Sales
df.columns

# It's really easy to query data with Pandas:
df[(df['Office'] == 'Stockholm') & (df['Sales'] > 260)]
# It's also easy to do aggregations...
aggregated_stockholm_sales = df[df.Office == 'Stockholm'].groupby('Year').sum()
aggregated_stockholm_sales

aggregated_ny_sales = df[df.Office == 'New York'].groupby('Year').sum()
# ... and generate plots
aggregated_stockholm_sales.plot(kind='bar')
aggregated_ny_sales.plot(kind='bar', color='g')<jupyter_output><empty_output><jupyter_text>Communicating with APIs<jupyter_code>import requests

response = requests.get("https://www.googleapis.com/books/v1/volumes", params={"q":"machine learning"})
raw_data = response.json()
titles = [item['volumeInfo']['title'] for item in raw_data['items']]
titles<jupyter_output><empty_output><jupyter_text>*** Apis - [reddit comments on apis](https://www.reddit.com/r/webdev/comments/3wrswc/what_are_some_fun_apis_to_play_with/)* Riot Games* [import.io](https://www.import.io/) - turns HTML pages to JSON* Spotify* Telegram* SoundCloud* Reddit* YouTube* Wunderground - Weather* Kandy - Massaging and video calling* Star Wars - All aboard the hype train* Marvel Comics* Mashape - API Market* [FoaaS](http://www.foaas.com/) - Fuck off as a Service* BreweryDB - Beer, beer, beer* Slack* Geo Names - Names of places* Common Crawl - Lots of data, petabytes of it* Programmers API - TV Show Info* FitBit - FitBit fitness tracker* JawBone - Jawbone fitness tracker* Moneypot - BitCoin Gambling* Steam - PCMR incoming* Twilio - VoiP & Text Messaging* Firebase - Build your own API* IBM Watson - Cognitive Computing with IBM* Lob - Email Postcards* Algolia - Search as a Service* Battle.net - Blizzard* Free Geo IP - Get geolocation of IP* The Counted - officer-involved killings in the US* Wolfram Alpha* IFTTT* USDA National Nutrient Database - United States Department of Agriculture* Twitter* Nutritionix - Nutrition DB* Geode Systems - lotsa data* Programmable Web API's - /thread (though keep going, this is fun)* Pokémon API* Open Weather Map - yet more weather  Download some stock data using the api from [Quandl](https://www.quandl.com/)<jupyter_code>import pandas as pd
import sys
sys.path.append('/usr/local/lib/python3.5/site-packages/quandl/')
import quandl

df = quandl.get('WIKI/GOOGL')
df.head()<jupyter_output><empty_output><jupyter_text>Taking out "relevant" features, we want to at least make the data as easy as possible without losing information<jupyter_code># We do not need any of the other columns 
# since these will have the same information only in a nicer format
df = df[['Adj. Open',  'Adj. High',  'Adj. Low',  'Adj. Close', 'Adj. Volume']]

# high - low percentage per day
df['highlow_percentage'] = (df['Adj. High'] - df['Adj. Low']) / df['Adj. Close'] * 100.0

# percentage change per day
df['percentage_change'] = (df['Adj. Close'] - df['Adj. Open']) / df['Adj. Open'] * 100.0
df = df[['Adj. Close', 'highlow_percentage', 'percentage_change', 'Adj. Volume']]
df.head()<jupyter_output><empty_output><jupyter_text>Machine LearningSubfield of artificial intelligence. ***Learning from examples and experience**** AlphaGo competing in the game Go* Similar approach is used in learning to play Mario [MarIO](https://www.youtube.com/watch?v=qv6UVOQ0F44) * A great video for an introduction on neural networks ***What is Machine Learning?***How would start by identifying a fruitDifference between orange and an apple```pythondef define_fruit():     lots of codedef detect_colors(image):    lots of codedef analyze_shapes(image):     more codedef guess_texture(image):     code.............if( fruit_is_apple() ):    then         15% success        probablity_that_apple_ad_will_work = 0.15 else if( fruit_is_orange()):    then         25% success        probablity_that_orange_AD_will_work = 0.25 ```Or how would you define characteristics for an individual. Some models are just rule based```pythondef invididual()    if (individual_is_in_high_school) && if(graduates_within_three_years):        then:         success of 10 %        probablity_for_click_on_AD = 0.10        showTheApproriateAd()        else         success of 17 %        probablity_for_click_on_AD = 0.17        showAnotherAd()```Machine Learning Problems can be seperated into a few large categories:* ***supervised learning***, which data comes from additional attributes that we can predict. Can be either: * ***classification***, discrete output variables * ***regression***, continuous output variables* ***unsupervised learning***, which training data consist of input without any target values.  Supervised LearningLabels are already in place![](supervised_learning_model.jpg "logo")We want to classify orange from appleWeight | Texture | Label--- | --- | ---150 | Bumpy | Orange170 | Bumpy | Orange140 | Soft | Apple130 | Soft | Apple... | ... | ...<jupyter_code># input to the classifier
features = [[140, 'soft'], [130,'soft'], [150,'bumpy'],[170,'bumpy']]

# output for the classifier
labels = ['apple', 'apple','orange','orange']

# We need to change the type of input for the classifier so we rename the features into ints

# 1 = 'soft', 0 = 'bumpy'
features = [[140, 1], [130,1], [150,0],[170,0]]

# 0 = 'apple', 1 = 'orange'
labels = [0,0,1,1]<jupyter_output><empty_output><jupyter_text>Now that we have the data let's make something from it<jupyter_code>from sklearn import tree

# doesnt know anything yet
clf = tree.DecisionTreeClassifier()<jupyter_output><empty_output><jupyter_text>Now we will train the data to the classifier with the training data that we have<jupyter_code>clf = clf.fit(features,labels)<jupyter_output><empty_output><jupyter_text>*** Notice that we want to predict something that is not in the training data ***<jupyter_code>clf.predict([[160,0]])<jupyter_output><empty_output><jupyter_text>Classic Machine Learning Problem [Iris flower data set](https://en.wikipedia.org/wiki/Iris_flower_data_set)Classify the type of flower based on * width* length* petal width * petal lengthscikit comes with some practices [datasets](http://scikit-learn.org/stable/datasets/)<jupyter_code>from sklearn.datasets import load_iris
iris = load_iris()

print(iris.feature_names)
print(iris.target_names)
print(iris.data[0])
print(iris.target[0])<jupyter_output>['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
['setosa' 'versicolor' 'virginica']
[ 5.1  3.5  1.4  0.2]
0<jupyter_text>Preparing DataExamples used to "test" the classifiers accurracy.will ***NOT*** be part of the training dataUsually a standard is to have ***80% training*** and ***20% testing***. But by no means a goto from the start Testing data<jupyter_code># dataset is constructed in a great way
# so we can get testing data from each example by taking 0,50,100

test_index = [0,50,100]<jupyter_output><empty_output><jupyter_text>Training data<jupyter_code># built based on c computation and mutation of arrays, i.e. its pretty fast
import numpy as np

# training data
train_target = np.delete(iris.target, test_index)
train_data = np.delete(iris.data, test_index, axis=0)

# testing data
test_target = iris.target[test_index]
test_data = iris.data[test_index]<jupyter_output><empty_output><jupyter_text>Training classifier<jupyter_code>clf = tree.DecisionTreeClassifier()
clf.fit(train_data, train_target)

# We expect the classifier to give us [0,1,2] from test_target
print(test_target)

print(clf.predict(test_data))<jupyter_output>[0 1 2]
[0 1 2]<jupyter_text>DONE! we have now made our first machine learning Classification* Naive Bayes* Decision trees / Random Forest* SVM* KNN* ... many more!Recall the features that we have, we need to evaluate them carefully.next we will visualize the decision tree to go into more in depth of what actually happensCode is taken from scikits homepage on [decision tree visualization](http://scikit-learn.org/stable/modules/tree.html) Visualize the Decision Tree Classifier<jupyter_code>import sys
sys.path.append('/usr/local/lib/python3.5/site-packages/pydot/')

import pydotplus
from sklearn.externals.six import StringIO
from IPython.display import Image  
dot_data = StringIO()  
tree.export_graphviz(clf, out_file=dot_data,  
                         feature_names=iris.feature_names,  
                         class_names=iris.target_names,  
                         filled=True, rounded=True,  
                         special_characters=True)  
graph = pydotplus.graph_from_dot_data(dot_data.getvalue()) 
graph.write_pdf("iris.pdf")
Image(graph.create_png()) 
# first testing flower
print(test_data[0], test_target[0])

print(iris.feature_names)
print("targets: ",iris.target_names)
# second testing flower
print(test_data[1], test_target[1])<jupyter_output>[ 7.   3.2  4.7  1.4] 1
{'fraction_non_alphanumeric': 0.11878648505218468, 'fraction_numerical': 0.013797983371661065, 'mean_word_length': 4.947922146238822, 'pattern_counts': {'":': 5, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 6, 'lorem ipsum': 0, 'www.': 8, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '571265', 'license': 'MIT', 'n_tokens_mistral': 3735, 'n_tokens_neox': 3525, 'n_words': 1087, 'path': 'notebooks/data-mining-intro-ML-may-2016.ipynb', 'repo_name': 'eleijonmarck/analytics-workflow-showcase', 'size': '11306'}