<jupyter_start><jupyter_text>Chapter 3  Built-in Data Structures, Functions and files  3.1 Data Structures and Sequences  Tuple - a fixed-length, immutable sequence of Python objects<jupyter_code>tup = 4, 5, 6
tup
nested_tup = (4, 5, 6), (7, 8)
nested_tup
tuple([4, 0, 2])
tup = tuple('string')
tup
tup[0]
tup = tuple(['foo', [1, 2], True])
## tup[2] = false will cause an eror as tup[2] cannot be changes
tup[1].append(3)
tup
(4, None, 'foo') + (6, 0) + ('Bar',)
(4, None, 'foo') * 4<jupyter_output><empty_output><jupyter_text>Unpacking tuples<jupyter_code>tup = (4, 5, 6)
a, b, c = tup
b
tup = 4, 5, (6,7)
a, b, (c, d) = tup
d<jupyter_output><empty_output><jupyter_text>Typical swap<jupyter_code>tmp = a
a = b
b = tmp<jupyter_output><empty_output><jupyter_text>swap a and b<jupyter_code>a, b = 1, 2
b, a = a, b<jupyter_output><empty_output><jupyter_text>Unpacking - iterating over sequences of tuples or lists<jupyter_code>seq = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]
for a, b, c in seq:
    print('a={0}, b={1}, c={2}'.format(a, b, c))<jupyter_output>a=1, b=2, c=3
a=4, b=5, c=6
a=7, b=8, c=9<jupyter_text>pluck a few elements form the beginning of a tuplet<jupyter_code>values = 1, 2, 3, 4, 5
a, b, *rest = values
a,b
rest
a = (1, 2, 2, 2, 3, 4, 2)
a.count(2)<jupyter_output><empty_output><jupyter_text>List - In contrast to trupes, lists are variable-length and their contents can be modified in-place<jupyter_code>a_list = [2, 3, 7, None]
tup = ('foo', 'bar', 'baz')
b_list = list(tup)
b_list
b_list[1] = 'peekaboo'
b_list<jupyter_output><empty_output><jupyter_text>The list function is frequently used as a way to materalize an iterator or generator expression<jupyter_code>gen = range(10)
gen
list(gen)
b_list.append('dwarf')
b_list
b_list.insert(1, 'red')
b_list
b_list.pop(2)
b_list
b_list.append('foo')
b_list
b_list.remove('foo')
b_list
'dwarf' in b_list
'dwarf' not in b_list
[4, None, 'foo'] + [7, 8, (2, 3)]
x = [4, None, 'foo']<jupyter_output><empty_output><jupyter_text>extend is less expensive than concatenation<jupyter_code>x.extend([7, 8, (2, 3)])
x<jupyter_output><empty_output><jupyter_text>Sorting<jupyter_code>a = [7, 2, 5, 1, 3]
a.sort()
a
b=['saw', 'small', 'He', 'foxes', 'six']
b.sort(key=len)
b<jupyter_output><empty_output><jupyter_text>Binary search and maintaining a sorted list<jupyter_code>import bisect
c = [1, 2, 2, 2, 3, 4, 7]
bisect.bisect(c, 2)
bisect.bisect(c, 5)
bisect.insort(c, 6)
c<jupyter_output><empty_output><jupyter_text>Slicing<jupyter_code>seq = [7, 2, 3, 7, 5, 6, 0, 1]
seq[1:5]
seq[3:4] = [6, 3]
seq
seq[:5]
seq[3:]
seq[-4:]
seq[-6:-2]
seq
seq[::2] # take every other element
seq[::-1] # reversing a list<jupyter_output><empty_output><jupyter_text>Built-in Sequence Functions enumerate<jupyter_code>some_list = ['foo', 'bar', 'baz']
mapping = {}
for i, v in enumerate(some_list):
    mapping[v] = i
mapping<jupyter_output><empty_output><jupyter_text>sorted<jupyter_code>sorted([7, 1, 2, 6, 0, 3, 2])
sorted('horse race')<jupyter_output><empty_output><jupyter_text>zip<jupyter_code>seq1 = ['foo', 'bar', 'baz']
seq2 = ['one', 'two', 'three']
zipped = zip(seq1, seq2)
list(zipped)
seq3 = [False, True]
list(zip(seq1, seq2, seq3))<jupyter_output><empty_output><jupyter_text>A very common use of zip is similtaneously iterating over multiple sequences, possibly also combine with enumerate<jupyter_code>for i, (a, b) in enumerate(zip(seq1, seq2)):
    print('{0}: {1}, {2}'.format(i, a, b))
pitchers = [('Nolan', 'Ryan'), ('Roger', 'Clemens'), ('Curt', 'Shilling')]
first_name, last_name = zip(*pitchers)
first_name
last_name<jupyter_output><empty_output><jupyter_text>Reversed<jupyter_code>list(reversed(range(10)))<jupyter_output><empty_output><jupyter_text>dict - more common name is hash map or associated array  It is a flexible sized collection of key-value pairs, where key and value are Python objects<jupyter_code>empty_dict = {}
d1 = {'a' : 'some value', 'b' : [1, 2, 3, 4]}
d1
d1[7] = 'an integer'
d1
d1['b']
'b' in d1 # her we check if dict contains a key the same way as whether a list contains a value
d1[5] = 'some value'
d1
d1['dummy'] = 'another value'
d1
del d1[5]
d1
ret = d1.pop('dummy')
ret
d1
list(d1.keys())
list(d1.values())
d1.update({'b' : 'foo', 'c' : 12})
d1<jupyter_output><empty_output><jupyter_text>Creating dicts from sequences<jupyter_code>mapping = dict(zip(range(5), reversed(range(5))))
mapping<jupyter_output><empty_output><jupyter_text>Default Values<jupyter_code>words = ['apple', 'bat', 'bar', 'atom', 'book']
by_letter = {}
for word in words:
    letter = word[0]
    if letter not in by_letter: # I think this not in is on the key not the value
        by_letter[letter] = [word]
    else:
        by_letter[letter].append(word)
by_letter<jupyter_output><empty_output><jupyter_text>Valid dict key type - dict keys generally have to be immutable objects like scalar types (int, float, string) or tuples(all the objects in a tuple need to be immutable, too)<jupyter_code>hash('string')
hash((1, 2, (2, 3)))
# hash((1, 2 [2, 3])) # fails because lists are mutable<jupyter_output><empty_output><jupyter_text>To use a list as a key<jupyter_code>d = {}
d[tuple([1, 2, 3])] = 5
d<jupyter_output><empty_output><jupyter_text>Set - an unordered collection of unique elements, like dicts, but keys only, no values<jupyter_code>set([2, 2, 2, 1, 3, 3])
{2, 2, 2, 1, 3, 3}
a = {1, 2, 3, 4, 5}
b = {3, 4, 5, 6, 7, 8}
a.union(b)
a | b
a.intersection(b)
a & b
c = a.copy()
c |= b
c
d = a.copy()
d &= b
d<jupyter_output><empty_output><jupyter_text>Like dicts, set elements generally must be immutable. To have a list like element, you must convert it to a tuple<jupyter_code>my_data = [1, 2, 3, 4]
my_set = {tuple(my_data)}
my_set
a_set = {1, 2, 3, 4, 5}
{1, 2, 3}.issubset(a_set)
a_set.issuperset({1, 2, 3})
{1, 2, 3} == {3, 2, 1}<jupyter_output><empty_output><jupyter_text>List, Set, and Dict Comprehensions  List Comprehensions all you to form a new list by filtering the elements of a collection Lists - use []<jupyter_code>strings = ['a', 'as', 'bat', 'car', 'dove', 'python']
[x.upper() for x in strings if len(x) > 2]<jupyter_output><empty_output><jupyter_text>Set and Dict comprehensions - use {} Set Comprehension<jupyter_code>unique_length = {len(x) for x in strings}
unique_length
set(map(len, strings))<jupyter_output><empty_output><jupyter_text>Dict comprehension<jupyter_code>loc_mapping = {val : index for index, val in enumerate(strings)}
loc_mapping<jupyter_output><empty_output><jupyter_text>Nested list comprehensions<jupyter_code>all_data = [['John', 'Emily', 'Michael', 'Mary', 'Steven'], ['Maria', 'Juan', 'Javier', 'Natalia', 'Pilar']]
result = [name for names in all_data for name in names if name.count('e') >= 2]
result
some_tuples = [(1, 2, 3), (4, 5, 6), (7, 8, 9)]
flattened = [x for tup in some_tuples for x in tup]

# flattened[]
#
# for tup in some_tuple:
#   for x in tup:
#       flattened.append(x)
        
flattened
[[x for x in tup] for tup in some_tuples]<jupyter_output><empty_output><jupyter_text>3.2 Functions code organisation and reuse<jupyter_code>def my_function(x, y, z=1.5):
    if z > 1:
        return z * (x + y)
    else:
        return z / (x + y)
    
my_function(5, 6, z=0.7)
my_function(3.14, 7, 3.5)
my_function(10,20)
my_function(x=5, y=6, z=7)
my_function(y=6, x=5, z=7)<jupyter_output><empty_output><jupyter_text>Namespaces, Scope, and Function<jupyter_code>a = []
def func():
    a = []
    for i in range(5):
        a.append(i)
func()
a
a = []
def func():
    for i in range(5):
        a.append(i)
func()
a
func()
a<jupyter_output><empty_output><jupyter_text>global<jupyter_code>a = None
def bind_a_variable():
    global a
    a = []
bind_a_variable()
print(a)<jupyter_output>[]<jupyter_text>Returning multiple values<jupyter_code>def f():
    a = 5
    b = 6
    c = 7
    return a, b, c
a, b, c = f()
print(a, b, c)<jupyter_output>5 6 7<jupyter_text>return_value can be a 3-tuple<jupyter_code>return_value = f()
return_value<jupyter_output><empty_output><jupyter_text>Or return a dict instead<jupyter_code>def f():
    a = 5
    b = 6
    c = 7
    return {'a': a, 'b' : b, 'c' : c}
return_value = f()
return_value<jupyter_output><empty_output><jupyter_text>Functions are objectsData cleaning and transformations<jupyter_code>states = ['    Alabama', 'Georgia!', 'Georgia', 'georgia', 'FlOrIda', 'south carolina##', 'West virginia?']
import re
def clean_strings(strings):
    result = []
    for value in strings:
        value = value.strip()
        value = re.sub('[!#?]', '', value)
        value = value.title()
        result.append(value)
    return result    
clean_strings(states)<jupyter_output><empty_output><jupyter_text>An alternative approach - make a list of the operations you want to apply<jupyter_code>states = ['    Alabama', 'Georgia!', 'Georgia', 'georgia', 'FlOrIda', 'south carolina##', 'West virginia?']
def remove_punctuation(value):
    return re.sub('[!#]', '', value)
clean_ops = [str.strip, remove_punctuation, str.title]
def clean_strings(strings, ops):
    result = []
    for value in strings:
        for function in ops:
            value = function(value)
        result.append(value)
    return result
clean_strings(states, clean_ops)<jupyter_output><empty_output><jupyter_text>You can use functions as arguments to other functions like the built in map function<jupyter_code>for x in map(remove_punctuation, states):
    print(x)<jupyter_output>Alabama
Georgia
Georgia
georgia
FlOrIda
south carolina
West virginia?<jupyter_text>Anonymous (Lambda) FunctionsWriting functions consisting of a single statement<jupyter_code>def short_function(x):
    return x * 2
short_function(3)
equiv_anon = lambda x: x * 2
equiv_anon(3)<jupyter_output><empty_output><jupyter_text>An advantabe is lambda functions can be used as arguements to many data transformation functions<jupyter_code>def apply_to_list(some_list, f):
    return [f(x) for x in some_list]
ints = [4, 0, 1, 5, 6]
apply_to_list(ints, lambda x: x * 2)<jupyter_output><empty_output><jupyter_text>Another example - sort a collection of strings by the number of distinct letters in each string<jupyter_code>strings = ['foo', 'card', 'bar', 'aaaa', 'abab']
strings.sort(key=lambda x: len(set(list(x))))
strings<jupyter_output><empty_output><jupyter_text>Currying: Partial Argument ApplicationDeriving new functions from existing ones by partial argument application<jupyter_code>def add_numbers(x, y):
    return x + y
add_five = lambda y: add_numbers(5, y)
add_five(3)<jupyter_output><empty_output><jupyter_text>Alternativly using built-in function functools module / partial function<jupyter_code>from functools import partial
add_five = partial(add_numbers, 5)
add_five(5)<jupyter_output><empty_output><jupyter_text>GeneratorsA generator is a concise way to construct a new iterable object Example iterating over a dict yields the dict key<jupyter_code>some_dict = {'a': 1, 'b': 2, 'c': 3}
for key in some_dict:
    print(key)
dict_iterator = iter(some_dict)
dict_iterator
list(dict_iterator)<jupyter_output><empty_output><jupyter_text>Generator example - to create a generator use the yield keyword instead of return in a function<jupyter_code>def squares(n=10):
    print('Generating squares from 1 to {0}'.format(n ** 2))
    for i in range(1, n+1):
        yield i ** 2
gen = squares()
gen
for x in gen:
    print(x, end=' ')<jupyter_output>Generating squares from 1 to 100
1 4 9 16 25 36 49 64 81 100<jupyter_text>Generator expressions<jupyter_code>gen = (x ** 2 for x in range(100))
gen
sum(x ** 2 for x in range(100))
dict((i, i ** 2) for i in range(5))<jupyter_output><empty_output><jupyter_text>itertools module - has a collection of generators for many common data algorithems<jupyter_code>import itertools
first_letter = lambda x: x[0]
first_letter('Alan')
names = ['Alan', 'Adam', 'Wes', 'Will', 'Albert', 'Steven']
for letter, names in itertools.groupby(names, first_letter):
    print (letter, list(names)) # names is a generator<jupyter_output>A ['Alan', 'Adam']
W ['Wes', 'Will']
A ['Albert']
S ['Steven']<jupyter_text>Errors and Exception Handling<jupyter_code>float('1.2345')
## float('something')  # This will cause an error
def attempt_float(x):
    try:
        return float(x)
    except:
        return x
attempt_float('1.2345')
attempt_float('something')
def attempt_float1(x):
    try:
        return float(x)
    except (TypeError, ValueError):
        return x
attempt_float1('something')
attempt_float1((1, 2))<jupyter_output><empty_output><jupyter_text>sample code about using finally: as part of the if, else statement<jupyter_code># f = open(path, 'w')
# 
# try:
#     write_to_file(f)
# except:
#     print('Failed')
# else:
#     print('Succeeded')
# finally:
#     f.close()<jupyter_output><empty_output><jupyter_text>Exceptions in Ipython<jupyter_code>%run "C://Users/MartinVM1/projects/python-for-data-analysis/pydata-book-2nd-edition/examples/ipython_bug.py"<jupyter_output><empty_output><jupyter_text>3.3 Files and the Operating System<jupyter_code>## path = "C://Users/MartinVM1/projects/python-for-data-analysis/pydata-book-2nd-edition/examples/segismundo.txt"
path = "../pydata-book-2nd-edition/examples/segismundo.txt"
f = open(path)
for line in f: # we can treat the file handle f like a list and iterate over the lines like this
    pass
lines = [x.rstrip() for x in open(path)]
lines
f.close()<jupyter_output><empty_output><jupyter_text>Use with statement to make it easier to clean up open files<jupyter_code>with open(path) as f:
    lines = [x.rstrip() for x in f]<jupyter_output><empty_output><jupyter_text>read, seek and tell<jupyter_code>f = open(path)
f.read(10)
f2 = open(path, 'rb') # Binary mode
f2.read(10)
f.tell()
f2.tell()<jupyter_output><empty_output><jupyter_text>To check the default encoding in the sys module<jupyter_code>import sys
sys.getdefaultencoding()<jupyter_output><empty_output><jupyter_text>seek changes the file position to the indicated byte in the file<jupyter_code>f.seek(3)
f.read(1)
f.close()
f2.close()
path
with open('../pydata-book-2nd-edition/examples/Scratch/tmp.txt', 'w') as handle:
    handle.writelines(x for x in open(path) if len(x) > 1)
with open('../pydata-book-2nd-edition/examples/Scratch/tmp.txt') as f:
    lines = f.readlines()
lines<jupyter_output><empty_output><jupyter_text>Bytes and Unicode with Filesdefault is text mode - that is you intend to work with Python strings(i.e., Unicode)<jupyter_code>with open(path) as f:
    chars = f.read(10)
chars
with open(path, 'rb') as f:
    data = f.read(10)
data
data.decode('utf8')
# data[:4].decode('utf8') # this causes error as some Unicode characters may not have been fully formed
sink_path = '../pydata-book-2nd-edition/examples/Scratch/sink.txt'
with open(path) as source:
    with open(sink_path, 'xt', encoding='iso-8859-1') as sink:
        sink.write(source.read())
with open(sink_path, encoding='iso-8859-1') as f:
    print(f.read(10))<jupyter_output>SueÃ±a el<jupyter_text>Beware using seek when opening files<jupyter_code>f = open(path)
f.read(5)
f.seek(4)
f.read(1)
f.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14593412419643448, 'fraction_numerical': 0.02617801047120419, 'mean_word_length': 5.476394849785407, 'pattern_counts': {'":': 0, '<': 218, '<?xml version=': 0, '>': 222, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '56', 'dir': 'jupyter-structured-clean-dedup', 'id': '633751', 'license': 'MIT', 'n_tokens_mistral': 6138, 'n_tokens_neox': 5822, 'n_words': 1542, 'path': 'C3-data-structures-functions-and-files.ipynb', 'repo_name': 'data-architect/python-for-data-analysis', 'size': '15089'}