<jupyter_start><jupyter_text>Data Structures<jupyter_code>import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>For most of this course we are going to work with `pandas` DataFrames. However, it's important to start with an introduction to the different types of data structures available in Python, their characteristics, their differences and their comparative advantages.The most important data structures in Python are:- lists- tuples- sets- dictionaries- numpy arrays- pandas DataFrames- pyspark DataFrames  Lists A **list** is a mutable array data structure in Python with the following characteristics:- can hold any type of data- can hold different types of data at the same time- can be modified We can **generate** lists using square brackets.<jupyter_code>l = [12, "world", [3,4,5]]
print(l)<jupyter_output>[12, 'world', [3, 4, 5]]<jupyter_text>Since lists are *ordered*, we can **access** their element by calling the position of the element in the list.<jupyter_code>l[0]<jupyter_output><empty_output><jupyter_text>Since lists are *mutable*, we can **modify** their elements.<jupyter_code>l[0] = 'hello'
print(l)<jupyter_output>['hello', 'world', [3, 4, 5]]<jupyter_text>We can **add** elements to a list using `.append()`.<jupyter_code>l.append(23)
l<jupyter_output><empty_output><jupyter_text>We can **remove** elements by calling `del` on it.<jupyter_code>del l[0]
print(l)<jupyter_output>['world', [3, 4, 5], 23]<jupyter_text>We can **combine** two lists using `+`. Note that this operation does not modify the list but generates a new one.<jupyter_code>l + [23]<jupyter_output><empty_output><jupyter_text>We can also generate lists using **comprehensions**.<jupyter_code>l = [n for n in range(3)]
print(l)<jupyter_output>[0, 1, 2]<jupyter_text>Comprehensions are a powerful tool!<jupyter_code>l = [n+10 for n in range(10) if (n%2==0) and (n>4)]
print(l)<jupyter_output>[16, 18]<jupyter_text>Tuples A **tuple** is an immutable array data structure in Python with the following characteristics:- can hold any type of data- can hold different types of data at the same time- **can not be modified** We can **generate** tuples using curve brackets.<jupyter_code># A list of different data types
t = (12, "world", [3,4,5])
print(t)<jupyter_output>(12, 'world', [3, 4, 5])<jupyter_text>Since tuples are *ordered*, we can **access** their element by calling the position of the element in the list.<jupyter_code>t[0]<jupyter_output><empty_output><jupyter_text>Since tuples are *unmutable*, we cannot **modify** their elements.<jupyter_code># Try to modify element
try:
    t[0] = 'hello'
except Exception as e:
    print(e)
# Try to add element
try:
    t.append('hello')
except Exception as e:
    print(e)
# Try to remove element
try:
    del t[0]
except Exception as e:
    print(e)<jupyter_output>'tuple' object doesn't support item deletion<jupyter_text>We can **combine** two tuples using `+`. Note that this operation does not modify the tuple but generates a new one. Also note that to generate a 1-element tuple we need to insert a comma.<jupyter_code>t + (23,)<jupyter_output><empty_output><jupyter_text>We can generate tuples using **comprehensions**, but we need to specify it's a `tuple`.<jupyter_code>t = tuple(n for n in range(3))
print(t)<jupyter_output>(0, 1, 2)<jupyter_text>Sets A **set** is a mutable array data structure in Python with the following characteristics:- **can only hold hashable types**- can hold different types of data at the same time- **cannot be modified**- **cannot contain duplicates** We can **generate** using curly brackets.<jupyter_code>s = {12, "world", (3,4,5)}
print(s)<jupyter_output>{(3, 4, 5), 12, 'world'}<jupyter_text>Since sets are *unordered* and *unindexed*, we cannot **access** single elements by calling their position.<jupyter_code># Try to access element by position
try:
    s[0]
except Exception as e:
    print(e)<jupyter_output>'set' object is not subscriptable<jupyter_text>Since sets are *unordered*, we cannot **modify** their elements by specifying the position.<jupyter_code># Try to modify element
try:
    s[0] = 'hello'
except Exception as e:
    print(e)<jupyter_output>'set' object does not support item assignment<jupyter_text>However, since sets are *mutable*, we can **add** elements using `.add()`.<jupyter_code>s.add('hello')
print(s)<jupyter_output>{'hello', (3, 4, 5), 12, 'world'}<jupyter_text>However, we cannot add **duplicates**.<jupyter_code>s.add('hello')
print(s)<jupyter_output>{'hello', (3, 4, 5), 12, 'world'}<jupyter_text>We can **delete** elements of a set using `.remove()`.<jupyter_code>s.remove('hello')
print(s)<jupyter_output>{(3, 4, 5), 12, 'world'}<jupyter_text>We can also generate sets using **comprehensions**.<jupyter_code>s = {n for n in range(3)}
print(s)<jupyter_output>{0, 1, 2}<jupyter_text>Dictionaries A **dictionary** is a mutable array data structure in Python with the following characteristics:- can hold any type- can hold different types at the same time- can be modified- **items are named** We can **generate** dictionaries using curly brackets. Since elements are indexed by keys, we have to provide one for each element.Dictionary keys can be of any **hashable type**. A hashable object has a hash value that never changes during its lifetime, and it can be compared to other objects. Hashable objects that compare as equal must have the same hash value.Immutable types like strings and numbers are hashable and work well as dictionary keys. You can also use tuple objects as dictionary keys as long as they contain only hashable types themselves.<jupyter_code>d = {"first": 12, 2: "world", (3,): [3,4,5]}
print(d)<jupyter_output>{'first': 12, 2: 'world', (3,): [3, 4, 5]}<jupyter_text>Since dictionaries are indexed but not ordered, we can only **access** elements by the corresponding key. If the corresponding key does not exist, we do not access any element.<jupyter_code>try:
    d[0]
except Exception as e:
    print(e)<jupyter_output>0<jupyter_text>We can **access all values** of the dictionary using `.values()`.<jupyter_code>d.values()<jupyter_output><empty_output><jupyter_text>We can **access all keys** of the dictionary using `.keys()`.<jupyter_code>d.keys()<jupyter_output><empty_output><jupyter_text>We can **access both values and keys** using `.items()`.<jupyter_code>d.items()<jupyter_output><empty_output><jupyter_text>This gives us a list of tuples which we can iterate on.<jupyter_code>[f"{key}: {value}" for key, value in d.items()]<jupyter_output><empty_output><jupyter_text>Since dictionaries are mutable, we can **modify** their elements.<jupyter_code>d["first"] = 'hello'
print(d)<jupyter_output>{'first': 'hello', 2: 'world', (3,): [3, 4, 5]}<jupyter_text>If we try to modify an element that does not exist, the element is added to the dictionary.<jupyter_code>d[0] = 'hello'
print(d)<jupyter_output>{'first': 'hello', 2: 'world', (3,): [3, 4, 5], 0: 'hello'}<jupyter_text>We can **remove** elements using `del`.<jupyter_code>del d[0]
print(d)<jupyter_output>{'first': 'hello', 2: 'world', (3,): [3, 4, 5]}<jupyter_text>We can cannot **combine** two dictionaries using `+`. We can only add one element at the time.<jupyter_code>try:
    d + {"fourth": (1,2)}
except Exception as e:
    print(e)<jupyter_output>unsupported operand type(s) for +: 'dict' and 'dict'<jupyter_text>We can also generate dictionaries using **comprehensions**.<jupyter_code>d = {f"k{n}": n+1 for n in range(3)}
print(d)<jupyter_output>{'k0': 1, 'k1': 2, 'k2': 3}<jupyter_text>Numpy Arrays A **numpy array** is a mutable array data structure in Python with the following characteristics:- can hold any type of data- **can only hold one type at the same time**- can be modified- **can be multi-dimensional**- **support matrix operations** We can **generate** numpy arrays are generated the `np.array()` function on a list.<jupyter_code>a = np.array([1,2,3])
print(a)<jupyter_output>[1 2 3]<jupyter_text>We can make 2-dimensional arrays (a matrix) as lists of lists.<jupyter_code>m = np.array([[1,2,3] , [4,5,6]])
print(m)<jupyter_output>[[1 2 3]
 [4 5 6]]<jupyter_text>Since numpy arrays are mutable, we can **modify** elements by calling the index of the numpy array.<jupyter_code>m[0,0] = 89
print(m)<jupyter_output>[[89  2  3]
 [ 4  5  6]]<jupyter_text>We can check the **shape** of a numpy array using `.shape`<jupyter_code>m.shape<jupyter_output><empty_output><jupyter_text>We can **expand dimensions** of a numpy array using `.expand_dims()`.<jupyter_code>a = np.expand_dims(a,1)
print(a)<jupyter_output>[[1]
 [2]
 [3]]<jupyter_text>We can **transpose** matrices using `.T`.<jupyter_code>a = a.T
print(a)<jupyter_output>[[1 2 3]]<jupyter_text>We **add** elements to a numpy array using `np.concatenate()`. All elements must have the same number of dimensions and must have the same number of elements along the concatenation axis.<jupyter_code>m = np.concatenate((m, a), axis=0)
print(m)<jupyter_output>[[89  2  3]
 [ 4  5  6]
 [ 1  2  3]]<jupyter_text>We cannot **remove** elements of numpy arrays.<jupyter_code>try:
    del a[0]
except Exception as e:
    print(e)<jupyter_output>cannot delete array elements<jupyter_text>We can do **matrix operations** between numpy arrays. For example, we can do multiplication with `@`.<jupyter_code>a @ m<jupyter_output><empty_output><jupyter_text>If we use `*` we get dot (or element-wise) multiplication instead.<jupyter_code>a * m<jupyter_output><empty_output><jupyter_text>There is a wide array of functions available in numpy. For example `np.invert()` inverts a squared matrix.<jupyter_code>np.invert(m)<jupyter_output><empty_output><jupyter_text>Comparison Which data structure should you use and why? Let's compare different data types<jupyter_code>K = 100_000
l = [n for n in range(K)]
t = tuple(n for n in range(K))
s = {n for n in range(K)}
a = np.array([n for n in range(K)])<jupyter_output><empty_output><jupyter_text>**Size**Which data type is more efficient?<jupyter_code>import sys

def compare_size(list_objects):
    for o in list_objects:
        print(f"Size of {type(o)}: {sys.getsizeof(o)}")
compare_size([l, t, s, a])<jupyter_output>Size of <class 'list'>: 800984
Size of <class 'tuple'>: 800040
Size of <class 'set'>: 4194520
Size of <class 'numpy.ndarray'>: 800104<jupyter_text>Size is very similar for lists, tuples and numpy arrays. **Speed**Which data type is faster?<jupyter_code>import time

def compare_time(list_objects):
    for o in list_objects:
        start = time.time()
        [x**2 for x in o]
        end = time.time()
        print(f"Time of {type(o)}: {end - start}")
compare_time([l, t, s, a])<jupyter_output>Time of <class 'list'>: 0.021067142486572266
Time of <class 'tuple'>: 0.02071404457092285
Time of <class 'set'>: 0.021012067794799805
Time of <class 'numpy.ndarray'>: 0.010493993759155273
{'fraction_non_alphanumeric': 0.13299539170506913, 'fraction_numerical': 0.026820276497695854, 'mean_word_length': 6.032404406999352, 'pattern_counts': {'":': 3, '<': 162, '<?xml version=': 0, '>': 163, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '517841', 'license': 'MIT', 'n_tokens_mistral': 3936, 'n_tokens_neox': 3700, 'n_words': 1250, 'path': 'notebooks/01_data_structures.ipynb', 'repo_name': 'matteocourthoud/Data-Science-Python', 'size': '10850'}