<jupyter_start><jupyter_text>Python Lists and ArraysThis notebook contains the code and most of the text for the article [Python Lists vs. Arrays:  How to Choose Between Them](https://codesolid.com/python-lists-vs-arrays/).  See the article for further details and analysis.When newcomers think of arrays, they may have an idea that this is implemented in Python as the built in list type.  However, as in other languages, Python lists and arrays are actually quite different.  Even experienced Python developers often reach for the list type first when they need a sequence type, but there are advantages to how arrays are implemented.  As with most things in Python, the language's consistency means that you can often easily replace the one with the other, or at least, thir APIs are very similar.   Let's look at arrays and lists in turn.<jupyter_code>import sys
sys.version

from datetime import datetime

# A list initialized with a literal
short_list = ["first", 2, datetime.now()]

# Define a list using a list comprehension
long_list = [x for x in range(1,101)]


# Lists grow dynamically.
long_list_2 = list()
for x in range(1,101):
    long_list_2.append(x)
    
print(long_list == long_list_2)<jupyter_output>True<jupyter_text>From the first example above (```short_list```), we see that lists in Python are able to hold elements of disparate types.  The second example (```long_list```) shows a list comprehension to generate a list.  The third example creates a list identical to the one in the second example, using a simple loop as one might do in another language.  The other thing this demonstrates is that the list is created dynamically, and mutated in place.  We call append to add each element in turn, and the list grows in size to accommodate this.Despite the name and the ability to grow dynamically, Python lists are not implemented internally as a linked list.  Rather, they are an array of references to objects, the memory for which is over-allocated as appends are done to be more efficient.  Because they are an array, indexing any object in the list is efficient no matter where in the array the object is located. Next we turn to Python arrays.There are a few differences between lists and arrays that we should go over up front:* The list type is always available as part of Python's built-ins.  With arrays, you have to import the array class from the array module.* Arrays do not have a literal syntax like the ```[item1, item2, ..., itemN]``` syntax that can be used to initialize a list.  For arrays, the array constructor must be used.* Arrays must always contain objects of the same type, and the type must be specified when the object is initialized.Some examples will make this clear.<jupyter_code>b1 = b"123\n4"
print(f"Bytes of length: {len(b1)}:  {b1}")

b2 = br"123\n4"
print(f"Raw bytes of length: {len(b2)}:  {b2}")

b1 = b"Hello, Python fans!"
print(b1[7:13])
print(b1[7])

# This code would give raise an exception:
# TypeError: 'bytes' object does not support item assignment
# b1[7] = 'J'
bytes_object = b'Python'
print(type(bytes_object[0]))
bytearray_object = bytearray(100)
print(bytearray_object[0:5])
bytearray_object[0:5] = b'Hello'
print(bytearray_object[0:5])
class FileIterator:
    def __init__(self, filename):
        self.filename = filename
        self.file_handle = open(filename, 'r')
        
    def readline(self):
        while line := self.file_handle.readline():
            yield line
        self.file_handle.close()
        
requirements = FileIterator('requirements.txt')
for line in requirements.readline():
    print(line)

print(sum((x for x in range(10) if x %2 == 1)))
from array import array

# Build an array of unicode characters (strings) from a single larger string.
string_elements = array("u", "Hello, array lovers everywhere!")
print(len(string_elements))
print(string_elements[0])
print(string_elements[0:5])
print(type(string_elements[0]))
print(string_elements.itemsize)<jupyter_output>31
H
array('u', 'Hello')
<class 'str'>
4<jupyter_text>In the example above, the first parameter to the array constructor is a type code, in this case saying we're dealing with Unicode characters.  The second parameter is a string, which is converted into an array of type ```wchar_t``` internally in C. However, when we access an element of the list, Python converts the character to the Python string type, as we see in the last line.  It does the same thing for slices in this case.We get a very different result if we change the type to an array of bytes.<jupyter_code>from array import array

# Build an array of unicode characters (strings) from a single larger string.
string_elements = array('B', b"Hello, array lovers everywhere!")

print(len(string_elements))
print(string_elements[0])
print(string_elements[0:5])
print(type(string_elements[0]))
print(string_elements.itemsize)
# Build an array from the list
long_list = [x for x in range(1,101)]

# Array of signed short integers
short_array = array("h", long_list)

# Array of unsigned long integers
long_array = array("L", long_list)<jupyter_output><empty_output><jupyter_text>Here, the "b" prefix before the string means to treat the string as an array of bytes, and it's represented internally as an unsigned char.  Because the string is in English, the length Python returns is the same in both cases, because it's the length of the array, not the underlying byte representation.  This works out because the first 128 ASCII characters in UTF-8 use the same 1-byte value that ASCII uses.<jupyter_code>double_array = array("d", [42.42])
print(double_array.itemsize)<jupyter_output>8<jupyter_text>In addition to the space savings (in the case of more compact data types like short integers or chars), it's possible that Python may also be gaining some runtime efficiency from when it resolves the data to a Python type, since in the list case the code to look up and render the type in Python may be less efficient than it is in the case of arrays, or the looping code may be less efficient in some other way.To test this out, we can use the timeit function to see if the iteration time differs significantly between arrays and lists.  In each case we'll iterate an array of 10,000 numbers 10,000 times.<jupyter_code>from timeit import timeit

result = timeit(setup="num_list = [x for x in range(0,10000)]", stmt="for i in num_list: z = i", number=10000)
print(result)
from timeit import timeit

result = timeit(setup="from array import array; num_array = array('i', [x for x in range(0,10000)])", stmt="for i in num_array: z = i", number=10000)
print(result)
from timeit import timeit

result = timeit(setup="import numpy as np; np_array = np.array([x for x in range(0,10000)])", stmt="for i in np_array: z = i", number=10000)
print(result)
# Do a sum on three different types of lists.  First let's make a long list (approximately 256,000)  
# numeric values that can fit in a byte

from timeit import timeit

iterations = 500

def display(elapsed_time, message, rounding=3):
    print(f"{round(elapsed_time, rounding)} seconds -- {message}.")

result = timeit(setup="num_list = [n for n in range(0,256)] * 1000", 
                stmt="sum(num_list)", number=iterations)
display(result, "Python sum function on a list of numbers")

result = timeit(setup="l = [n for n in range(0,256)] * 1000; num_array = bytearray(l)", 
                stmt="sum(num_array)", number=iterations)
display(result, "Python sum function on a bytearray of numbers")

result = timeit(setup="import numpy as np; l = [n for n in range(0,256)] * 1000; np_array = np.array(l)", 
                stmt="sum(np_array)", number=iterations)
display(result, "Python sum function on a NumPy array", rounding=4)

result = timeit(setup="import numpy as np; l = [n for n in range(0,256)] * 1000; np_array = np.array(l)", 
                stmt="np_array.sum()", number=iterations)
display(result, "NumPy sum method on a NumPy array")

list_numbers = [0, 0, 0, 0, 0]
print(list_numbers)
first_three = list_numbers[0:2]
first_three[0] = 100
first_three[1] = 99
print(list_numbers)
list_numbers = [0, 0, 0, 0, 0]
print(list_numbers)
list_numbers[0:1]  = [5,6]
print(list_numbers)
memoryview_numbers = memoryview(bytearray(5))
print(memoryview_numbers.tolist())
first_three = memoryview_numbers[0:2]
first_three[0] = 100
first_three[1] = 99
print(memoryview_numbers.tolist())<jupyter_output>[0, 0, 0, 0, 0]
[100, 99, 0, 0, 0]
{'fraction_non_alphanumeric': 0.09063945901055878, 'fraction_numerical': 0.024676711353659984, 'mean_word_length': 4.770020533880904, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '247832', 'license': 'Apache-2.0', 'n_tokens_mistral': 2633, 'n_tokens_neox': 2431, 'n_words': 1104, 'path': 'day-15/PythonArraysvsLists.ipynb', 'repo_name': 'JohnLockwood/100-days-of-python', 'size': '8429'}