<jupyter_start><jupyter_text><NAME>, Data Scientist - Who has fun LEARNING, EXPLORING & GROWING Week Five Lists__Sequential Ordered Collection__ Creating lists<jupyter_code>x = [4,2,6,3] # Create a list with values
y = list()
z = []
print(x,y,z,sep='\n')<jupyter_output>[4, 2, 6, 3]
[]
[]<jupyter_text>Adding items to a list<jupyter_code>x=list()
print(x)
x.append('One') #Adds 'One' to the back of the empty list
print(x)
x.append('Two') #Adds 'Two' to the back of the list ['One']
print(x)
x.insert(0,'Half')
print(x)
x = list()
x.extend([1,2,3])
print(x)
x.extend([4,5,6])
x<jupyter_output><empty_output><jupyter_text>Indexing and slicing<jupyter_code>x=[1,7,2,5,3,5,67,32]
print(len(x))
print(x[3])
print(x[2:5])
print(x[-1])
print(x[::-1])<jupyter_output>8
5
[2, 5, 3]
32
[32, 67, 5, 3, 5, 2, 7, 1]<jupyter_text>Removing item from a list<jupyter_code>x = [1,7,2,5,3,5,6,7,32]
x.pop() # Removes the last element from a list
print(x)
x.pop(3)
x
#### x.remove(7) # Removes first 7 from the list
x<jupyter_output><empty_output><jupyter_text>Mutability of lists<jupyter_code>y=['a','b']
x = [1,y,3]
print(x)
print(y)
y[1] = 4
print(y)
print(x)
x="Hello"
print(x,id(x))
x+=" You!"
print(x,id(x)) #x is not the same object it was
y=["Hello"]
print(y,id(y))
y+=["You!"] 
print(y,id(y)) #y is still the same object. Lists are mutable. Strings are immutable

def eggs(item,total=0):
    total+=item
    return total
def spam(elem,some_list=[]):
    some_list.append(elem)
    return some_list
print(eggs(1))
print(eggs(2))
print(spam(1))
print(spam(2))<jupyter_output>1
2
[1]
[1, 2]<jupyter_text>Iteration Range iteration<jupyter_code>#The for loop creates a new variable (e.g., index below)
#range(len(x)) generates values from 0 to len(x) 
x=[1,7,2,5,3,5,67,32]
for index in range(len(x)):
    print(x[index])
list(range(len(x)))<jupyter_output><empty_output><jupyter_text>List element iteration<jupyter_code>x=[1,7,2,5,3,5,67,32]
#The for draws elements - sequentially - from the list x and uses the variable "element" to store values
for element in x: 
    print(element)<jupyter_output>1
7
2
5
3
5
67
32<jupyter_text>Practice problemWrite a function search_list() that searches a list of tuple pairs and returns the value assosiated with the first element of the pair<jupyter_code>def search_list(list_of_tuples,value):
    for t in prices:
        if t[0] == value:
            return t[1]
prices = [('AAPL',96.43),('IONS',39.28),('GS',159.53)]
ticker = 'IONS'
print(search_list(prices,ticker))<jupyter_output>39.28<jupyter_text>Dictionaries<jupyter_code>mktcaps = {'AAPL':538.7,'GOOG':68.7,'IONS':4.6}

mktcaps['AAPL'] #Returns the value associated with the key "AAPL"
mktcaps['GS'] #Error because GS is not in mktcaps
print(mktcaps.get('GS')) # Returns None because GS is not in mktcaps
mktcaps['GS'] = 88.65 #Adds GS to the dictionary
print(mktcaps) 
del mktcaps['GOOG'] #Removes GOOG from mktcaps
print(mktcaps)
mktcaps.keys()
mktcaps.values()
mktcaps.values()[1]
numbers = [1, 2, 3, 4] 
numbers.append([5, 6, 7, 8])
print(len(numbers))
numbers
dict1 = {"john":40, "peter":45} 
dict2 = {"john":466, "peter":45}
dict1 > dict2<jupyter_output><empty_output><jupyter_text>datetime library<jupyter_code>import datetime
d1 = datetime.date(2016,11,24)
d2 = datetime.date(2017,10,24)
max(d1,d2)
print(d2-d1)
import datetime

century_start = datetime.date(2000,1,1)
today = datetime.date.today()
print(century_start,today)
print('\nWe are',today - century_start,'days into this century')<jupyter_output>2000-01-01 2018-03-21

We are 6654 days, 0:00:00 days into this century<jupyter_text>For a cleaner output<jupyter_code>print('\nWe are',(today - century_start).days,'days into this century')<jupyter_output>We are 6654 days into this century<jupyter_text>datetime.datetime<jupyter_code>century_start = datetime.datetime(2000,1,1,0,0,0)
time_now = datetime.datetime.now()
print(century_start,time_now)
print("we are",time_now - century_start,"days, \
hour, minutes and seconds into this century")<jupyter_output>2000-01-01 00:00:00 2018-03-21 16:21:03.830647
we are 6654 days, 16:21:03.830647 days, hour, minutes and seconds into this century<jupyter_text>datetime objects can check validityA ValueError exception is raised if the object is invalid<jupyter_code>some_date=datetime.date(2015,2,29)
#some_date =datetime.date(2016,2,29)
#some_time=datetime.datetime(2015,2,28,23,60,0)<jupyter_output><empty_output><jupyter_text>datetime.timedeltaUsed to store the duration between two points in time<jupyter_code>century_start = datetime.datetime(2000,1,1,0,0,0)
time_now = datetime.datetime.now()
time_since_century_start = time_now - century_start
print('days since century start',time_since_century_start.days)
print('seconds since century start',time_since_century_start.total_seconds())
print("minutes since century start",time_since_century_start.total_seconds()/60)
print("hours since century start",time_since_century_start.total_seconds()/60/60)<jupyter_output>days since century start 6654
seconds since century start 574965031.088364
minutes since century start 9582750.5181394
hours since century start 159712.50863565665<jupyter_text>datetime.time<jupyter_code>date_and_time_now = datetime.datetime.now()
time_now = date_and_time_now.time()
print(time_now)<jupyter_output>16:31:43.729316<jupyter_text>We can do arithmetic operations on datetime objectsWe can use timedelta objects to calculate new dates or times a given date<jupyter_code>today = datetime.date.today()
five_days_later = today + datetime.timedelta(days=5)
print(five_days_later)
now=datetime.datetime.today()
five_minutes_and_five_seconds_later = now + datetime.timedelta(minutes=5,seconds=5)
print(five_minutes_and_five_seconds_later)
now=datetime.datetime.today()
five_minutes_and_five_seconds_earlier = now+datetime.timedelta(minutes=-5,seconds=-5)
print(five_minutes_and_five_seconds_earlier)<jupyter_output>2018-03-21 16:32:45.135673<jupyter_text>But you can't use timedelta on time objects. If you do, you'll get a TypeError exception<jupyter_code>time_now=datetime.datetime.now().time() #Returns the time component (drops the day)
print(time_now)
thirty_seconds=datetime.timedelta(seconds=30)
time_later=time_now+thirty_seconds
#Bug or feature?
#But this is Python
#And we can always get around something by writing a new function!
#Let's write a small function to get around this problem
def add_to_time(time_object,time_delta):
    import datetime
    temp_datetime_object = datetime.datetime(500,1,1,
                                             time_object.hour,
                                             time_object.minute,
                                             time_object.second)
    #print(temp_datetime_object)
    return (temp_datetime_object+time_delta).time()
#And test it
time_now=datetime.datetime.now().time()
thirty_seconds=datetime.timedelta(seconds=30)
print(time_now,add_to_time(time_now,thirty_seconds))<jupyter_output>16:44:59.301277 16:45:29<jupyter_text>datetime and strings More often than not, the program will need to get the date or time from a string:    From a website (bus/train timings)    From a file (date or datetime associated with a stock price)    From the user (from the input statement)Python needs to parse the string so that it correctly creates a date or time object__datetime.strptime__- datetime.strptime(): grabs time from a string and creates a date or datetime or time object- The programmer needs to tell the function what format the string is using- See [this](http://pubs.opengroup.org/onlinepubs/009695399/functions/strptime.html) for how to specify the format<jupyter_code>date='01-Apr-03'
date_object=datetime.datetime.strptime(date,'%d-%b-%y')
print(date_object)
#Unfortunately, there is no similar thing for time delta
#So we have to be creative!
bus_travel_time='2:15:30'
hours,minutes,seconds=bus_travel_time.split(':')
x=datetime.timedelta(hours=int(hours),minutes=int(minutes),seconds=int(seconds))
print(x)
#Or write a function that will do this for a particular format
def get_timedelta(time_string):
    hours,minutes,seconds = time_string.split(':')
    import datetime
    return datetime.timedelta(hours=int(hours),
                              minutes=int(minutes),
                              seconds=int(seconds))<jupyter_output><empty_output><jupyter_text>datetime.strftime- the strftime function flips the strptime function. It converts a datetime object to a string with specified format<jupyter_code>now = datetime.datetime.now()
string_now = datetime.datetime.strftime(now,'%m/%d/%y %H:%M:%S')
print(now,string_now)
print(str(now)) #Or you can use the default conversion<jupyter_output>2018-03-21 16:57:39.180239 03/21/18 16:57:39
2018-03-21 16:57:39.180239<jupyter_text>Datetime examples Bucketing timeThe file "sample_data.csv" contains start times and processing times for all complaints registered with New York City's 311 complaint hotline on 01/01/2016. Our goal is to compute the average processing time for each hourly bucket. __Let's take a quick look at the data__<jupyter_code>!head sample_data.csv<jupyter_output>2016-01-01 00:00:09,0.0815162037037037
2016-01-01 00:00:40,0.1334837962962963
2016-01-01 00:01:09,20.388726851851853
2016-01-01 00:02:59,0.9811458333333334
2016-01-01 00:03:03,7.048576388888889
2016-01-01 00:03:03,0.1400810185185185
2016-01-01 00:03:29,0.11086805555555555
2016-01-01 00:04:06,0.016967592592592593
2016-01-01 00:04:37,0.1597222222222222
2016-01-01 00:04:56,2.996585648148148<jupyter_text>Step One: Read the data<jupyter_code>data_time_pairs = list()

with open('sample_data.csv','r') as f:
    for line in f:
        data_time_pairs.append(line.strip().split(','))
        
print(len(data_time_pairs))
data_tuples[0:5]
print(type(data_time_pairs[0][0]))
print(type(data_time_pairs[0][1]))
# Let's convert strings into data ans double formats
import datetime
#Figure out the format string
# http://pubs.opengroup.org/onlinepubs/009695399/functions/strptime.html 
x='2016-01-01 00:00:09'
format_str = '%Y-%m-%d %H:%M:%S'
datetime.datetime.strptime(x,format_str)
data_time_pairs = list()

with open('sample_data.csv','r') as f:
    for line in f:
        data_time_pairs.append(line.strip().split(','))
        
import datetime

for i in range(0,len(data_time_pairs)):
    data_time_pairs[i][0] = datetime.datetime. \
    strptime(data_time_pairs[i][0],format_str)
    data_time_pairs[i][1] = float(data_time_pairs[i][1])
data_time_pairs[0:5]<jupyter_output><empty_output><jupyter_text>Now, replace the __datetime__ by __hourly buckets__<jupyter_code># Extract the hour from a datetime object
x = data_time_pairs[0][0]
x.hour
data_time_pairs = [(x[0].hour,x[1] )for x in data_time_pairs]
data_time_pairs[0:5]<jupyter_output><empty_output><jupyter_text>Create function that returns data<jupyter_code>def get_data(filename):
    data_time_pairs = list()

    with open('sample_data.csv','r') as f:
        for line in f:
            data_time_pairs.append(line.strip().split(','))
        
    import datetime
    
    format_str = "%Y-%m-%d %H:%M:%S"

    data_time_pairs = [(datetime.datetime.strptime(x[0],format_str).hour,
                        float(x[1])) for x in data_time_pairs]
    
    return data_time_pairs
get_data('sample_data.csv')<jupyter_output><empty_output><jupyter_text>Step Two: Accumulate counts and sums for each bucket<jupyter_code>hour_buckets = {}

for item in get_data('sample_data.csv'):
    if item[0] in hour_buckets:
        hour_buckets[item[0]][0] += 1
        hour_buckets[item[0]][1] += item[1]
    else:
        hour_buckets[item[0]] = [1,item[1]]
print(type(hour_buckets.items()))
hour_buckets.items()<jupyter_output><class 'dict_items'><jupyter_text>Let's print them to see what sort of pattern is there in the data.Bear in mind that this is just one day's data!<jupyter_code>for key, value in hour_buckets.items():
    print("Hour: ",key,"\tAverage: ",value[1]/value[0])<jupyter_output>Hour:  0 	Average:  0.6570511564469035
Hour:  1 	Average:  2.9613477328431377
Hour:  2 	Average:  2.334965452261305
Hour:  3 	Average:  3.0839338759007866
Hour:  4 	Average:  4.663183017810805
Hour:  5 	Average:  2.550054976851854
Hour:  6 	Average:  5.344349826388891
Hour:  7 	Average:  2.5844597678664565
Hour:  8 	Average:  6.0724520669659565
Hour:  9 	Average:  8.564869090207626
Hour:  10 	Average:  12.671294691132733
Hour:  11 	Average:  5.901653566341063
Hour:  12 	Average:  13.66402543540564
Hour:  13 	Average:  8.593492462013293
Hour:  14 	Average:  8.100135135135135
Hour:  15 	Average:  12.776634463154863
Hour:  16 	Average:  10.943701434277418
Hour:  17 	Average:  6.634365784623489
Hour:  18 	Average:  7.324956692612944
Hour:  19 	Average:  9.098796085858586
Hour:  20 	Average:  5.199433822667603
Hour:  21 	Average:  4.74319171267541
Hour:  22 	Average:  8.449229102956167
Hour:  23 	Average:  5.184938602292768<jupyter_text>Put everything into a function.This way, we can easily test other similar datasets<jupyter_code>def get_hour_bucket_averages(filename):
    def get_data(filename):
        data_tuples = list()
        with open(filename,'r') as f:
            for line in f:
                data_tuples.append(line.strip().split(','))
                
        import datetime
        format_str = "%Y-%m-%d %H:%M:%S"
        data_tuples = [(datetime.datetime.strptime(x[0],format_str).hour,
                        float(x[1])) for x in data_tuples]
        return data_tuples
    
    hour_buckets = dict()
    for item in get_data(filename):
        if item[0] in hour_buckets:
            hour_buckets[item[0]][0] += 1
            hour_buckets[item[0]][1] += item[1]
        else:
            hour_buckets[item[0]] = [1,item[1]]
    return [(key,value[1]/value[0]) for key,value in hour_buckets.items()]
get_hour_bucket_averages('sample_data.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12464773466290917, 'fraction_numerical': 0.09545487390707422, 'mean_word_length': 5.209062359802602, 'pattern_counts': {'":': 4, '<': 91, '<?xml version=': 0, '>': 92, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '101282', 'license': 'MIT', 'n_tokens_mistral': 5949, 'n_tokens_neox': 5127, 'n_words': 1145, 'path': 'Analytics & Python/All iPython Notebooks/_2_BAMM_101_W_2_Lists-Dicts-daytime.ipynb', 'repo_name': '2series/DataScience-Courses', 'size': '13846'}