<jupyter_start><jupyter_text>How Does a Bike-Share Navigate Speedy Success ![](ghtop_images/header2.png)  IntroductionSome quick analysis on Cycle bike-share data from ChicagoWelcome to the Cyclistic bike-share analysis case study! In this case study, you will perform many real-world tasks of a juniordata analyst. You will work for a fictional company, Cyclistic, and meet different characters and team members. In order toanswer the key business questions, you will follow the steps of the data analysis process: ask, prepare, process, analyze,share, and act. Along the way, the Case Study Roadmap tables — including guiding questions and key tasks — will help youstay on the right path.By the end of this lesson, you will have a portfolio-ready case study. Download the packet and reference the details of thiscase study anytime. Then, when you begin your job hunt, your case study will be a tangible way to demonstrate yourknowledge and skills to potential employers.  Some imports<jupyter_code>import pandas as pd
from bs4 import BeautifulSoup
import numpy as np
import requests
# from datetime import datetime, timezone
import os
import geopandas as gpd

#get the current working directory
owd=os.getcwd()<jupyter_output><empty_output><jupyter_text>A function to extract web zip files and save to /dat folder We'll just look at Q1 for 2020 dataThe data for 2020 is split into Q1 and the rest is for each month. So we need to download and convert each to a data frame and then combine them together<jupyter_code>def extractStuff(url):
    import requests, zipfile, io
    import os
    owd=os.getcwd()
    r = requests.get(url)
    z = zipfile.ZipFile(io.BytesIO(r.content))
    z.extractall(owd+"/dat/")
URL="https://divvy-tripdata.s3.amazonaws.com/"
noma =[['202004-divvy-tripdata'],
['202005-divvy-tripdata'],
['202006-divvy-tripdata'],
['202007-divvy-tripdata'],
['202008-divvy-tripdata'],
['202009-divvy-tripdata'],
['202010-divvy-tripdata'],
['202011-divvy-tripdata'],
['202012-divvy-tripdata']]
    
for nom in noma:
    extractStuff(URL+ nom[0]+".zip")<jupyter_output><empty_output><jupyter_text>Load files, put in pandas data frame and have a look Lets load each csv and combine them<jupyter_code>for i,nom in enumerate(noma):
    if i>0:
        df=pd.read_csv(owd+"/dat/"+nom[0]+'.csv')
        dfAll=pd.concat([df,dfAll])    
        print('1 ',nom[0])
    else:
        dfAll=pd.read_csv(owd+"/dat/"+nom[0]+'.csv')
        print('2',nom[0])<jupyter_output>2 202004-divvy-tripdata
1  202005-divvy-tripdata
1  202006-divvy-tripdata
1  202007-divvy-tripdata
1  202008-divvy-tripdata
1  202009-divvy-tripdata
1  202010-divvy-tripdata
1  202011-divvy-tripdata
1  202012-divvy-tripdata<jupyter_text>How many NaN stations?<jupyter_code>print('The percentage start stations NaN = {}'.format(100*np.shape(dfAll[dfAll['start_station_id'].isna()])[0] / np.shape(dfAll)[0]) )#95 282 3 114 796

print('The percentage end stations NaN = {}'.format(100*np.shape(dfAll[dfAll['end_station_id'].isna()])[0] / np.shape(dfAll)[0]) )#95 282 3 114 796

bothNa=dfAll[dfAll['start_station_id'].isna() | dfAll['end_station_id'].isna()] 
print('The percentage start stations NaN = {}'.format(100*np.shape(bothNa)[0] / np.shape(dfAll)[0]) )#95 282 3 114 796
import copy
dfUse=copy.copy(dfAll[dfAll['start_station_id'].notnull() & dfAll['end_station_id'].notnull()])
dfUse.describe(include='all')<jupyter_output><empty_output><jupyter_text>Now we need to convert the dates from object (i.e. string) to date format next add a new column as time for hire in hours<jupyter_code>dfUse.loc[:,'started_at']=pd.to_datetime(dfUse['started_at'],infer_datetime_format=True)
dfUse.loc[:,'ended_at']=pd.to_datetime(dfUse['ended_at'],infer_datetime_format=True)

delta=dfUse.iloc[:,3]-dfUse.iloc[:,2]
dd=delta.dt.total_seconds()/(60*60)
dfUse.insert(2,"hire_time_h",dd)
dfUse.head()<jupyter_output><empty_output><jupyter_text>Maybe we want the day of the week?The day of the week with Monday=0, Sunday=6.<jupyter_code>dfUse.insert(3,'day_week',dfUse.loc[:,'started_at'].dt.dayofweek)
dfUse.head()<jupyter_output><empty_output><jupyter_text>Lets also get the time on its own<jupyter_code>dfUse.insert(4,'time_day',dfUse.loc[:,'started_at'].dt.hour + dfUse.loc[:,'started_at'].dt.minute/60)
dfUse.head()<jupyter_output><empty_output><jupyter_text>And the distance travelled<jupyter_code>def distanceLatLong(lat1,lon1,lat2,lon2):
    
    import numpy as np

    def deg2rad(deg):
        return deg * np.pi/180

    R = 6371; # Radius of the earth in km
    dLat = deg2rad(lat2-lat1)  # deg2rad below
    dLon = deg2rad(lon2-lon1)
    a = np.sin(dLat/2) * np.sin(dLat/2) + \
    np.cos(deg2rad(lat1)) * np.cos(deg2rad(lat2)) * \
    np.sin(dLon/2) * np.sin(dLon/2)

    c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a))
    d = R * c ## Distance in km
 
    return d
d=distanceLatLong(dfUse["start_lat"].values,dfUse["start_lng"].values,dfUse["end_lat"].values,dfUse["end_lng"].values)
dfUse.insert(11,'distance',d)
dfUse.describe()<jupyter_output><empty_output><jupyter_text>Some issues arose above- hire_time_h max and min valuesLooks like the error is there from the start, so lets delete themdo the same for long times<jupyter_code>dfUse=dfUse[dfUse.hire_time_h>=0]

dfUse=dfUse[dfUse.hire_time_h<24]<jupyter_output><empty_output><jupyter_text>Lets drop some columns for space<jupyter_code>dfUse.drop(columns=["ride_id", "started_at","ended_at","start_station_name","end_station_name"],inplace=True)
#,"start_station_id","end_station_id"]<jupyter_output><empty_output><jupyter_text>save<jupyter_code>dfUse.to_csv('/data/df_2020.csv')
df=copy.copy(dfUse)
df = pd.read_csv('/data/df_2020.csv')
df<jupyter_output>C:\Users\44781\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:3165: DtypeWarning: Columns (5,6) have mixed types.Specify dtype option on import or set low_memory=False.
  has_raised = await self.run_ast_nodes(code_ast.body, cell_name,<jupyter_text>Put frequency location onto a map<jupyter_code>import folium
from folium import plugins
from folium.plugins import HeatMap

lat=df['start_lat'].values
lon=df['start_lng'].values
latlon = [lat, lon]

maps = folium.Map(location=[lat[0],lon[0]],
                    zoom_start = 11)

latlon=np.transpose(latlon)

# Plot it on the map
HeatMap(latlon).add_to(maps)

# Display the map
maps<jupyter_output><empty_output><jupyter_text>![](ghtop_images/ChiHeat.png) Bit of a mess, grouping by region may be better  Plot some choroplotsopen and modify the geojson file- seems to make life easier later<jupyter_code>import geopandas as gpd
fname='Chicago.geojson'
chicago = gpd.read_file(fname)

# neighborhoods aren't unique so lets use the index and call it ID
chicago.reset_index(inplace=True)
chicago.rename(columns={'index':'ID'},inplace=True)

chicago.drop(columns=['sec_neigh','shape_area','shape_len'],inplace=True)
chicago.to_file("Chi_.json", driver="GeoJSON")
chicago.head()<jupyter_output><empty_output><jupyter_text>Now we want to convert each station to a region within the json file First let's create a variable for each station, with location and station_id<jupyter_code>#use mean here just in case some slight differences- big ones lets hope not!
dfStat=df.groupby(by=['start_station_id']).mean()

dfStat=dfStat.drop(columns=['Unnamed: 0','hire_time_h','day_week','time_day','distance','end_lat','end_lng'])
dfStat.reset_index(inplace=True)
dfStat<jupyter_output><empty_output><jupyter_text>Now for each station we want a JSON-area codeThis is *slightly* convoluted- scroll through each station- for each station find if it's inside a Chigao_JSON region- if not we assign if a value 1000- for those with no region find the nearest station that has a JSON-region (done in tab after this)whatChoro = json IDwhatwhat = station ID<jupyter_code>from shapely.geometry import shape, Point


whatChoro=[]
whatwhat=[]
# check each polygon to see if it contains the point
i=0
# scroll through each station
for istat in range(np.shape(dfStat)[0]):
    i=0
    #create a point for the station
    point=Point(dfStat.loc[istat,'start_lng'],dfStat.loc[istat,'start_lat'])
    
    #scroll through each geometery
    for feature in chicago.ID:
        polygon = shape(chicago.loc[i,'geometry'])
        if polygon.contains(point):
            #this gives the json region ID
            whatChoro.append(chicago.loc[i,'ID'])
            #this give the station id
            whatwhat.append(dfStat.loc[istat,'start_station_id'])
            break 
        
        #if we don't get a match!!
        if feature==chicago['ID'].iloc[-1]:
            
            import copy
            # find distances lat2/lng2 this location
            # lat1/lng1 all locations
            lat1_=copy.copy(dfStat['start_lat'])
            lon1_=copy.copy(dfStat['start_lng'])
            lat2_=dfStat.loc[istat,'start_lat']
            lon2_=dfStat.loc[istat,'start_lng']

            
            #this gives the json region ID
            whatChoro.append(1000)
            #this give the station id
            whatwhat.append(dfStat.loc[istat,'start_station_id'])
            
            
        i=i+1<jupyter_output><empty_output><jupyter_text>This does the cleaning up if they don't have a json idThis will handle when we don't get a match---> a reuse of the distance function with slight mods<jupyter_code>def distanceLatLong_v2(lat1_,lon1_,lat2_,lon2_):
    
    import numpy as np
    import math
    def deg2rad(deg):
        return deg * np.pi/180
    def inner(lat1,lon1,lat2,lon2):
        R = 6371; # Radius of the earth in km
        dLat = deg2rad(lat2-lat1)  # deg2rad below
        dLon = deg2rad(lon2-lon1)
        a = np.sin(dLat/2) * np.sin(dLat/2) + \
        np.cos(deg2rad(lat1)) * np.cos(deg2rad(lat2)) * \
        np.sin(dLon/2) * np.sin(dLon/2)

        c = 2 * math.atan2(np.sqrt(a), np.sqrt(1-a))
        d = R * c ## Distance in km
        if d==0:
            d=1000
        return d
    if np.shape(lat1_)[0]>1:
        d=[]
        for i in range(np.shape(lat1_)[0]):
            d.append(inner(lat1_[i],lon1_[i],lat2_,lon2_))
        
    else:
        d=inner(lat1_,lon1_,lat2_,lon2_)   
    
    return d<jupyter_output><empty_output><jupyter_text>this scrolls through ones we didn't match and finds nearest JSON-id we did match<jupyter_code>for i in range(np.shape(dfStat)[0]):
    if whatChoro[i]==1000:
             
        #find distances lat2/lng2 this location
        # lat1/lng1 all locations
        lat1_=copy.copy(dfStat['start_lat'])
        lon1_=copy.copy(dfStat['start_lng'])
        lat2_=dfStat.loc[i,'start_lat']
        lon2_=dfStat.loc[i,'start_lng']
        ind=[idx for idx, element in enumerate(whatChoro) if element==1000]
        lat1_[ind]=0
        lon1_[ind]=0
        d=distanceLatLong_v2(lat1_,lon1_,lat2_,lon2_)
        indamin=d.index(min(d))
#         whatwhat[i]=whatwhat[indamin]
        whatChoro[i]=whatChoro[indamin]
        
        print(i,indamin,whatChoro[i],whatChoro[indamin],min(d))<jupyter_output>554 474 48 48 3.218616605981318
555 474 48 48 1.5436541433245226
556 474 48 48 0.7244340745224377
557 554 48 48 1.618989106320433
558 557 48 48 0.9589393666816851
559 558 48 48 1.703490590200107
560 559 48 48 1.0394756465095663
561 554 48 48 0.5321339843549502
562 561 48 48 0.8611490298165934
563 561 48 48 0.4535879672825213
570 554 48 48 0.7620712283526319
603 562 48 48 0.7657516801324228
604 1355 55 55 0.9398603094768803
605 559 48 48 1.3757808765327746
1441 554 48 48 0.0009191040374285816
1442 1441 48 48 0.003706303019684898
1443 555 48 48 0.006456249041029345
1444 556 48 48 4.9908562509422157e-05
1445 1444 48 48 0.002230404415936471
1446 557 48 48 0.0012372240336437155
1447 1446 48 48 0.003412919018808641
1450 558 48 48 0.005285184511373336
1451 1450 48 48 0.011408782497315317
1452 559 48 48 0.027327480291422585
1453 560 48 48 0.0012042612006966005
1454 561 48 48 0.0010429138707679156
1455 562 48 48 0.0017205826064828574
1456 1455 48 48 0.0034819520850311483
1457 563 48 48 0.002580[...]<jupyter_text>Now we can insert a new column in df with the json ID<jupyter_code>chicID=[]
for stat in df['start_station_id']:
#     chicID.append(stat)
    ind=[idx for idx, element in enumerate(whatwhat) if element==stat]
    try:
        chicID.append(whatChoro[ind[0]])
    except:
        continue
df.insert(0,'ID',chicID)
df.head()<jupyter_output><empty_output><jupyter_text>And represent each JSON region by how many times they're usedWe'll take the count and divide it by the total- and because of the distribution we'll also take the log- basically hires are highly focussed on a few regions with many having low %<jupyter_code>dfG=df.groupby('ID').count()
dfG.reset_index(inplace=True)
dfG=dfG[['ID','rideable_type']]
dfG.rename(columns={'rideable_type':'Frequency'})
dfG.rideable_type=np.log(dfG.rideable_type/sum(dfG.rideable_type))
dfG.head()
df.to_csv('/data/dfChoro_2020.csv')
df = pd.read_csv('/data/dfChoro_2020.csv')<jupyter_output>C:\Users\44781\anaconda3\lib\site-packages\IPython\core\interactiveshell.py:3165: DtypeWarning: Columns (7,8) have mixed types.Specify dtype option on import or set low_memory=False.
  has_raised = await self.run_ast_nodes(code_ast.body, cell_name,<jupyter_text>Now the plottingWe first read in the json file, add the df with our frequency values to it then we can plot the data  Lets put this in a function to look at differences<jupyter_code>def bigChoro(dfIN,colname,choi):
    import folium 
    
    LEGNOM=colname
    if choi=='count':
        dfG=dfIN.groupby('ID').count()
        dfG[colname]=dfG[colname]/(100*274/7)
        myscale = (dfG[colname].quantile((0,0.25,0.5,0.75,0.9,0.95,.97,1))).tolist()
#         np.linspace(dfG[colname].min(),dfG[colname].max(),10)
        LEGNOM='Number of journeys 100s per week'
    elif choi=='mean':
        dfG=dfIN.groupby('ID').mean()
        myscale = np.linspace(dfG[colname].min(),dfG[colname].max(),10)
    elif choi=='sum':
        dfG=dfIN.groupby('ID').sum()
        myscale = np.linspace(dfG[colname].min(),dfG[colname].max(),10)
    elif choi=='dayofweek':
        dfIN=dfIN[['ID',colname]]
        dfG=dfIN.groupby(['ID']).agg(lambda x:x.value_counts().index[0])
        dfG[dfG[colname]>4]=5-dfG[dfG[colname]>4]
        myscale = np.array([-2.,0.,1.,2.,3.,4.])
        LEGNOM='Day of week (-2 to -1 weekend, 0-4 Monday to Friday)'
    elif choi=='mode':
        dfIN=dfIN[['ID',colname]]
        dfIN[colname].astype('int32')
        dfG=dfIN.groupby(['ID']).agg(lambda x:x.value_counts().index[0])
        myscale = np.linspace(dfG[colname].min(),dfG[colname].max(),10)
        
    dfG.reset_index(inplace=True)
    dfG=dfG[['ID',colname]]


    nil=gpd.read_file("Chi_.json")
    nil=nil[['ID','geometry']]

    # merge data frames
    nilpop=nil.merge(dfG,on="ID")

    #initial map
    m = folium.Map(location=[41.884,-87.6247], zoom_start=10,\
       control_scale=True,tiles="Stamen Toner")#,tiles = t_list[1])

    folium.TileLayer('CartoDB positron',name="Light Map",control=False).add_to(m)

   
    # (dfG['rideable_type'].quantile((0,.02,0.1,.25,0.5,0.75,0.9,0.95,0.98,1))).tolist()
    choropleth =folium.Choropleth(
        geo_data="Chi_.json", 
        data=nilpop,
        threshold_scale=myscale,
        columns=['ID',colname],
        name='choropleth',
        fill_color='BuPu',#PuBuGn YlGn PuBuGn YlGnBu RdYlBu
        key_on= "feature.properties.ID",
        fill_opacity=0.7,
        line_opacity=0.2,
        nan_fill_color='gray',
        legend_name=LEGNOM, 
        nan_fill_opacity =.5,
        ).add_to(m)
    folium.LayerControl().add_to(m)
    choropleth.geojson.add_child(
        folium.features.GeoJsonTooltip(['pri_neigh'],labels=False)
        )
    return m
dfIN.columns
colname='distance'#end_station_id
dfIN= df[df.member_casual=='member']
m=bigChoro(dfIN,colname,'mean')
m<jupyter_output><empty_output><jupyter_text>![](ghtop_images/ChiDist.png)<jupyter_code>colname='day_week'#end_station_id
dfIN= df[df.member_casual=='member']
m=bigChoro(dfIN,colname,'dayofweek')
m<jupyter_output><empty_output><jupyter_text>![](ghtop_images/ChiDay.png)<jupyter_code>colname='time_day'#end_station_id
dfIN= df[df.member_casual=='member']
m=bigChoro(dfIN,colname,'mode')
m<jupyter_output><empty_output><jupyter_text>![](ghtop_images/ChiDayTime.png)<jupyter_code>colname='time_day'#end_station_id
dfIN= df[df.member_casual=='member']
m=bigChoro(dfIN,colname,'count')
m<jupyter_output><empty_output><jupyter_text>![](ghtop_images/ChiCountMember.png)<jupyter_code>colname='time_day'#end_station_id
dfIN= df[df.member_casual=='casual']
m=bigChoro(dfIN,colname,'count')
m<jupyter_output><empty_output><jupyter_text>![](ghtop_images/ChiCountCasual.png)<jupyter_code>np.linspace(-2,4,7)
np.array([-2.,0.,1.,2.,3.,4.])
import seaborn as sns
import matplotlib as mpl
import matplotlib.pyplot as plt
sns.set_theme(style="ticks")

f, ax = plt.subplots(figsize=(7, 5))
sns.despine(f)

# Draw a nested boxplot to show bills by day and time
sns.histplot(df,hue="member_casual", x="time_day",
            multiple="stack",
            palette="dark:b_r",
            edgecolor=".3",
            linewidth=.5,)
sns.despine(offset=10, trim=True)
from matplotlib.ticker import PercentFormatter
df_=df[df.hire_time_h<5]
df_=df_[df_.hire_time_h>0]
sns.set_theme(style="ticks")

binwidth = 5

f, ax1 = plt.subplots(figsize=(10, 7))
sns.despine(f)
df2_=df_[df_.member_casual=='member']
# Draw a nested boxplot to show bills by day and time
sns.histplot(df2_, x="hire_time_h",
            multiple="stack",
            palette="light:m_r",
            edgecolor=".3",
            linewidth=.5,
            stat='probability',
            log_scale=True,
            ax=ax1,
            label='member')

ax2=ax1.twinx()

df2_=df_[df_.member_casual=='casual']
# Draw a nested boxplot to show bills by day and time
sns.histplot(df2_, x="hire_time_h",
             element="step",fill=False,
             color='red',
            linewidth=.8,
            stat='probability',
            log_scale=True,
            ax=ax2,
            label='casual')

ax1.legend(loc='upper left')
ax2.legend(loc='upper right')
sns.despine(offset=10, trim=True)
# ax.legend('Member','Casual')
sns.set_theme(style="ticks")

f, ax = plt.subplots(figsize=(10, 8))
sns.despine(f)
df_=df[ df["distance"]>0.1 ]
df_= df_[df_["distance"]<20]
df_=df_[df_.member_casual=='member']
# Draw a nested boxplot to show bills by day and time
sns.histplot(data=df_, x="distance",
            edgecolor=".3",
            linewidth=.5,
            stat='probability',
             label='member')
sns.despine(offset=10, trim=True)
df_=df[df["distance"]>0.1]
df_= df_[df_["distance"]<20]
df_=df_[df_.member_casual=='casual']
sns.histplot(data=df_, x="distance",
            linewidth=.8,
            color='r',
            stat='probability',
             label='casual',
            fill=False,
            element='step')

ax.legend(loc='upper right')

# f, ax = plt.subplots(figsize=(7, 5))
sns.set_theme(style="whitegrid")

# Draw a nested boxplot to show bills by day and time
ax=sns.histplot(df,hue="member_casual", 
             x="day_week",palette="dark:b_r",
             multiple="dodge",
             bins=[0 ,1 ,2 ,3, 4, 5, 6,7],
             shrink=.9
            )
sns.despine(offset=20, trim=True)
aa=np.array([0,1,2,3,4,5,6])+.5
ax.set_xticks(aa)
ax.set_xlim([0, 7.5])
lab=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday']
ax.set_xticklabels(lab,rotation='vertical')
ax.set_xlabel('Day of the week')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1352456920795623, 'fraction_numerical': 0.07046070460704607, 'mean_word_length': 4.17270563343031, 'pattern_counts': {'":': 0, '<': 101, '<?xml version=': 0, '>': 106, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '418565', 'license': 'Apache-2.0', 'n_tokens_mistral': 8114, 'n_tokens_neox': 7149, 'n_words': 1498, 'path': '_notebooks/2021-11-16-ChicagoBikes.ipynb', 'repo_name': 'ThomasHSimm/THS', 'size': '19557'}