<jupyter_start><jupyter_text>Github Repohttps://github.com/bobhaffner/omaha_python_june_15 Anaconda  * Out of the box modules  * Environments  * Spyder  Jupyter Project<jupyter_code>import os
from IPython.display import FileLink, display, HTML
from IPython.lib.display import YouTubeVideo
from IPython.core.display import Image<jupyter_output><empty_output><jupyter_text>Markdown cells are nice to have* Bullet    * Sub bullet<jupyter_code>#running cells (ctrl-enter and shift-enter)
print (os.path)
#output
os.path 
#code completion
HTML("hello.txt")
#displaying a video
YouTubeVideo("dQw4w9WgXcQ",width=600, height=400,start=43,autoplay=True)
#displaying an image
Image("rr.jpg")
#bash commands
! ls
#magic functions
% timeit range(1000) #and so many more
#open web pages.  the reason for the name change
url="https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages"
HTML(url)<jupyter_output><empty_output><jupyter_text>Misc Navigation* File    * Save and Checkpoint    * Close and Halt* Edit    * Cut    * Merge* Insert    * Insert Cell Above - Control then a    * Insert Cell Below - Control then b* Cell    * Run - Control-Enter    * Run and Select Below - Shift-Enter* Kernel    * Interrupt aka Ctrl-C    * Restart* Help    * Keyboard Shortcuts  Pandas* Inspired by dataframes in R * Built on the popular numpy* In memory* Under active development  Why* Data scrubbing and formatting through the power of python* Data exploration* Hooks to data visualization* Hooks to machine learning and other modules* Combined with Jupyter, allows you to build a forgiving and dynamic workflow* If you want to do work with data and python, I would highly recommend learning pandas<jupyter_code>import pandas as pd
#reading in a csv file. One of many ways to read in data
df = pd.read_csv("bike_data_sub.csv",parse_dates=["start_date","end_date"])
#data
df #head shows the top 5 by default.  There's also a tail()
#Series and DataFrames
type(df.trip_id), type(df)
#The dates in the file are really datetimes so we rename columns post import
d = {"start_date" : "start_date_time", "end_date" : "end_date_time"}
df.rename(columns=d, inplace = True)
df.head()
#basic stats for every numeric value in your dataframe.  Whether you want it or not :-)
df.describe()
#even more stuff
df.info()
#doing some math
df["double_duration"] = df["duration"] * 2
df[["duration","double_duration"]].head()
(df['start_station'] == 'San Jose City Hall')
#filtering by way boolean indexing
filter = (df['start_station'] == 'San Jose City Hall')
df[filter].head(5)
#groupby and sorting
df[['start_station','trip_id']].groupby('start_station').count().sort(
                                'trip_id',ascending=False).head()
pd.pivot_table(df,'duration','subscription_type','start_station',aggfunc=len,fill_value=0)
dfLatLongs = pd.read_csv("latlongs.csv")
dfLatLongs.head()
#merging
df = pd.merge(df,dfLatLongs,left_on="start_terminal",right_on="terminal", how="left")
df.head()<jupyter_output><empty_output><jupyter_text>Functional aspect of Pandas<jupyter_code>#plain old function
def getDate(x):
    return x.date()
#creating a new column and updating it to the date of a datetime.  WITHOUT a for loop
df["start_date"] = df["start_date_time"].apply(getDate)
df[["start_date_time","start_date"]].head()
#Getting the day of the week as an integer using a Lambda
df["start_dow"] = df["start_date_time"].apply(lambda x : x.weekday())
df[["start_date_time","start_dow"]].head()
#dict keys = integers representing days and values = days of the week
d = {0 : 'Monday', 1 : 'Tuesday', 2 : 'Wednesday', 3 : 'Thursday', 4 : 'Friday',  5 : 'Saturday', 6 : 'Sunday'}
d
#map the values to our dataframe
df['start_dow_name'] = df['start_dow'].map(d)
df[["start_dow","start_dow_name"]].head()<jupyter_output><empty_output><jupyter_text>What didn't we cover* Indexes* Time Series* In addition read_csv() there are many other ways to get data in (and out)* In addition to pivot_table() check out other reshaping methods* In addtion to merge check out join and concat* Panels (multiple related dataframes)* And a ton of other things ResouresUsing SQL as a comparisonhttp://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/http://www.gregreda.com/2013/10/26/using-pandas-on-the-movielens-dataset/Videoshttps://www.youtube.com/watch?v=MxRMXhjXZoshttps://www.youtube.com/watch?v=w26x-z-BdWQhttps://www.youtube.com/watch?v=rEalbu8UGeoBookshttp://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1449319793/ref=sr_1_1?ie=UTF8&qid=1414785088&sr=8-1&keywords=python+for+data+analysis  BokehThey wrote javascript so you don't have to<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
from bokeh.charts import Bar, Line, TimeSeries 
from bokeh.plotting import show, output_notebook, figure, output_file, ColumnDataSource
from bokeh.models import Text, Rect, Triangle, Circle, HoverTool, Plot, Range1d
output_notebook()
#housekeeping
d = {"lat" : "start_lat", "long" : "start_long"}
df.rename(columns=d, inplace = True)
dow = df[["start_dow_name","duration"]].groupby("start_dow_name").sum()
dow.head()
#pandas integration with matplotlib is pretty good and works for most things
dow.plot(kind="bar");
#but bokeh gives you some extra interaction and polish
p = Bar(dow)
show(p)
#prepping some data by grouping and cutting
grp = df[["start_terminal","start_station","trip_id","start_long","start_lat"]].groupby(
        ["start_terminal","start_station","start_long","start_lat"]).count().reset_index()
grp["cuts"] = pd.cut(grp.trip_id,4,labels=[10,20,30,40]).astype("int")
grp.tail(10)
TOOLS='box_zoom,pan,box_select,crosshair,resize,reset,hover'

p = figure(plot_width=800, plot_height=600, title=None, tools=TOOLS)

#output_file("bike.html")  

source = ColumnDataSource(data=dict(x=grp["start_long"],y=grp["start_lat"],id=grp["start_terminal"],
                                    name=grp["start_station"],trips=grp["trip_id"]))

p.circle(grp["start_long"], grp["start_lat"],size=grp["cuts"] + 20, color="blue", source=source)
hover = p.select(dict(type=HoverTool))
hover.tooltips = [("(x,y)", "($x, $y)"),("id", "@id"),("name", "@name"),("trips", "@trips")]

show(p)<jupyter_output><empty_output><jupyter_text>Show bike.htmlfile:///Users/tbmh1/jupyter_notebooks/omaha_python_june_15/bike.html Galleryhttp://bokeh.pydata.org/en/latest/docs/user_guide/interaction.htmlhttp://bokeh.pydata.org/en/latest/docs/gallery.htmlstatic-examples Resourceshttp://bokeh.pydata.org/en/latest/https://www.youtube.com/watch?v=O5OvOLK-xqQhttps://www.youtube.com/watch?v=S11GfFlQgtw  Blaze Familiar python interface for interacting with a variety of data sources using the resources of those data sources http://blaze.pydata.org/en/latest/_static/presentations/blaze.html/0/4<jupyter_code>import blaze as bz
#loading a csv into a postgres table using the postgres Copy command 
bz.into("postgresql:///bike_data::bike_trips","bike_data_sub_no_headers.csv")
#doing a "groupby" using the postgres engine
bike_db = bz.Data('postgresql:///bike_data')
bz.by(bike_db.bike_trips.start_terminal, count=bike_db.bike_trips.trip_id.count()).sort("start_terminal")
#what's happening behind the scenes
print (bz.compute(bz.by(bike_db.bike_trips.start_terminal, 
                        count=bike_db.bike_trips.trip_id.count())))
#and with a pandas dataframe
bdf = bz.Data(df)
bz.by(bdf.start_terminal,count=bdf.trip_id.count())
#and with a csv file
bike_csv = bz.Data("bike_data_sub.csv")
bz.by(bike_csv.start_terminal, count=bike_csv.trip_id.count())<jupyter_output><empty_output><jupyter_text>Resourceshttps://www.youtube.com/watch?v=x0svOPW6DdEhttp://blaze.pydata.org/en/latest/ Failed attempt with Spark and Blazehttp://stackoverflow.com/questions/30897932/error-accessing-spark-with-blaze<jupyter_code>p = sc.parallelize(range(100))
p.min(), p.max(), p.mean()
sc
rdd = bz.into(sc,"/Users/tbmh1/jupyter_notebooks/omaha_python_june_15/bike_data_sub_spark.csv")
bike_spark = bz.Data(rdd)
bike_spark.count()  #yea
bz.by(bike_spark.start_terminal, count=bike_spark.trip_id.count()) #nay
#IPYTHON=1 pyspark
#IPYTHON_OPTS="notebook" pyspark
#import blaze as bz<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13729782317888847, 'fraction_numerical': 0.015079654627264988, 'mean_word_length': 6.011082693947144, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 8, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '551735', 'license': 'MIT', 'n_tokens_mistral': 3052, 'n_tokens_neox': 2887, 'n_words': 652, 'path': 'omaha_python_june_15.ipynb', 'repo_name': 'bobhaffner/opug_pydata_2015_06_17', 'size': '8223'}