<jupyter_start><jupyter_text>Python Internals  Asking for an address is not the same as asking for a house<jupyter_code>davids_address = "32 Lark Lane"

davids_address
# I've given you my address, and you've written it down in your address book...

address_book = {
    'david': davids_address
}

address_book
class House:
    def __init__(self, address, occupants, contents):
        self.address = address
        self.occupants = occupants
        self.contents = contents
        
    def __repr__(self):
        return self.address + ' : ' + str(self.occupants)
        
# I live with a mouse
davids_house = House(davids_address, ["David", "Jeremy (the mouse)"], ['A bronze horse sculpture', 'A beachball'])
davids_house
# Your address book contains my address.  Not my house.  It wouldn't fit, and it's hard to copy houses.

address_book['david']<jupyter_output><empty_output><jupyter_text>A brief primer on CPython shares many syntactic similarities to C, and the reference implementation of Python(ie what you get when you download and install Python) is CPython - Python written in C.The C language has basic datatypes similar to Python (int, float etc).  There are differences, but they represent conceptually the same kinds of values.In addition, C has "pointers", indicated by the * symbol - these do _not_ contain the underlying data, but rather the _address_ of the dataYou must manually specify all types (including pointer types) in CUnderstanding a little about pointers explains a lot of what happens in Python<jupyter_code>/*   Pass by value only
* Does effectively nothing - creates a local variable x, that contains the value of the first
* argument, then sets that local variable to 1.1
* Nothing is returned - nothing changes outside of the function (there are no side effects)
*/

void
value_function(float x) {
    x = 1.1;
}

/*   Pass a pointer
* A pointer to a float is passed in as an argument
* This pointer is dereferenced (ie its address is looked up, so we are now
* operating on the underlying data), and the data pointed to by x is changed
* This change is visible outside of the function (there are side effects)
*/

void
pointer_function(float *x) {
    *x = 1.1;
}

/*
* Pointers also represent arrays.  In C, arrays are stored contiguously in memory,
* so offsets are equivalent to indexes into the array
*/

void
pointer_array_function(float *x) {

    /* Does the same thing as the function above */
    x[0] = 1.1;
    
    /* Indexes into the array from the pointer (the address at 'x + sizeof(float) * index') */
    x[5] = 2.2
}<jupyter_output><empty_output><jupyter_text>Back to Python<jupyter_code># Nearly everything in Python is really a _pointer_, not a value
# That is, we are not moving data around in memory, or making copies - 
# we are simply giving out addresses
a = ['shoes', 'bats', 'wheels']
b = ['red', 'green', 'shocking pink']
# The names a and b are 'identifiers'
# Identifiers are really just entries in an address book, which Python neatly has a builtin datatype for; the dict
# At the top level, these are stored as 'globals', which can be seen anywhere
# At other scope (inside functions), they are stored as locals

globals()
def see_locals(x):
    y = 123
    print(locals())

see_locals(a)
# When a line of code is parsed by the Python interpreter (right to left) it looks up entries in the address book, and 
# possibly creates new ones

a = 5  # Take the integer literal 5, add an 'a' entry to the address book whose value is 5
a = [a] # Create a new list; look up 'a', and place it in the list;  update 'a' to point to this new list

b = a # Look up a (which is a list);  update 'b' to refer to this list

a,b
# These are equal to other (which is what you'd expect)

a == b
# But they're not just equal... a _is_ b

a is b
# Things can be equal, but not be the same object

c = [0,1,2]
d = [0,1,2]

c == d, c is d
d.append(3)
c == d
a, b
# In this example, we might say we're appending to a...

a.append(7)

#... but really, we're appending to the object that is referred to by a
# (ie the object we get when we look up a in the address book)
# This is the same object we'd get when we look up b; 'a' and 'b' really contain the addresses of the objects (pointers)
# not the objects themselves

a, b
# What will happen here?

a.append(a)

a
a[2] is a
# There are some inconsistencies... Python _nearly_ always behaves like it's using pointers, but basic datatypes like
# int, float and str don't quite work like this...  they really do store the _value_
# This is largely for efficiency
f = 5.1
g = f

# Looks up f, which really is a floating point number, adds 1.0, updates the entry for f in the symbol table
f += 1.0

f,g
h = [1]
i = h

# Calls the append method directly on our list
h += [2]


# h and i both still point to the same object
h, i
# If you really want a copy of something (instead of pointer to it), you can call... copy
# Most standard python objects have a copy method, and there is a copy module for cases where this is not the case

j = h.copy()

j == h, j is h
# However, copy only goes one layer down - consider the following

a = dict(x = 12.1, y = 17.2)
b = [a]

c = b.copy()

c.append(3)

print(b, c)

a["z"] = 31.1 

print(b, c)
# you can use deepcopy for these situations
# from copy import deepcopy<jupyter_output><empty_output><jupyter_text>Numpy arrays (hint: they're C arrays)<jupyter_code># Numpy is much more like pure C than the Python standard libraries.
# This is why it's fast!
import numpy as np

x = np.linspace(0,1.0, 11)

x
# Numpy arrays are Python wrappers around actual C arrays
# That means they are stored in order, can be passed to C functions etc

x.flags
# We can look at the real underlying address in memory:

x.ctypes.data
# Numpy tries really really hard not to copy data, only pointers (and some indexing information)

# y is a pointer into x
y = x[5:]

y[:] = 3.0

x
x.ctypes.data
# Remember, while arrays are views into memory, basic python datatypes still operate as values, not real pointers

z = x[0]
print(z)
z = 10.0  # The z identifier is updated, but it is not referring to any part of our array
print(z, x)
# However, operating on a length one array subset is still referring to the actual array

z = x[0:1]
print(z)
z[0] = 10.0
print(x)<jupyter_output><empty_output><jupyter_text>A brief divergence - why is numpy fast and standard Python slow? Python (historically) hasn't had a native array type, and has used List as the basic ordered container Lists work fundamentally differently ; they are not stored in order, but rather the order is a property of each item; it knows about it's connections to the next and previous itemsa = [0, 1] Internally, it looks something a bit like this - a.head = listitem(0, None, None)   The first member of a, contains 0, knows about nothinga.head.tail = listitem(1, a.head, None)  the 'tail' - the next item to be iterated;                                           it contains 1, and also knows that its previous listitem is a.head                                         These are 'linked'; ie they know about each other This means that (for example), removing (or inserting) a single item of a list is very, very fast - we only need to update the immediately neightbouring items However looking up and item in a list takes linear time - you have to go through every item, whereas in C arrays, it's just arithmetic on the address<jupyter_code># Copies (or new data) will be created using stanard parsing logic...

# Get y - add 3.0, but not in place, add a new identifier entry called z, 
z = y + 3.0
print(z)
z[:] = 0.0
print(z)
print(y) # Unchanged<jupyter_output><empty_output><jupyter_text>Garbage Collection<jupyter_code># Some final notes about memory, identifiers/symbol tables
# In C (and C++), you need to manually manage memory
# If you don't delete something, it lives forever, and you run out of memory
# In Python this is automatic... but how?

class Thing:
    def __init__(self, name):
        self.name = name
        
    def __repr__(self):
        return self.name

# It's a small universe...
things_that_exist = [Thing('dogs'), Thing('hats'), Thing('the future')]

# I only care about hats
things_that_i_care_about = things_that_exist[1]

# Let's clean up the universe
things_that_exist = None 

# Because there are things still referenced in this list, they still exist
print(things_that_i_care_about)

# However, the others are gone!  There is no way to look up their addresses, and so no possible meaningful way to interact
# with them - they are garbage, and it is collected

# This is called 'reference counting'; every time an object is bound (e.g placed in a container, assigned to an identifier)
# it has 1 added to it's reference count.  If this disappear, then 1 is subtracted
# When the count reaches 0, they are deleted (garbage collected)

# This is easy and possible because of the 'everything is a pointer' approach<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0785889707524001, 'fraction_numerical': 0.009042196918955124, 'mean_word_length': 3.6832200731834814, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '479554', 'license': 'BSD-2-Clause-FreeBSD', 'n_tokens_mistral': 2733, 'n_tokens_neox': 2624, 'n_words': 1357, 'path': 'notebooks/workshops/02PythonInternals.ipynb', 'repo_name': 'emmamcbryde/AuTuMN-1', 'size': '8958'}