<jupyter_start><jupyter_text>Associations and dictionariesIn mathematics, we group numbers or other elements in parentheses, thus, forming a *tuple*. For example, to represent a three-dimensional Euclidean point, we'd use 3-tuple notation like `(32,9,9732)`.  Python uses the same mathematical notation:<jupyter_code>p = (32,9,9732)
print type(p)
print p<jupyter_output><type 'tuple'>
(32, 9, 9732)<jupyter_text>Because Python also uses parentheses for grouping subexpressions like `(1+2)*3`, there is an ambiguity in the language. Does `(5)` represent a single element tuple containing 5 or is it just the integer 5? It turns out that Python considers it an integer so we use the slightly awkward notation `(5,)` instead to mean a 1-tuple.Tuples are ordered and so we access the elements using array indexing notation.<jupyter_code>print p[0]
print p[1]
print p[2]<jupyter_output>32
9
9732<jupyter_text>It's really implemented as a list:<jupyter_code>from lolviz import *
listviz(p)<jupyter_output><empty_output><jupyter_text>Tuples are **immutable**, meaning you can't change the elements.  For example, `p[0]=34` gives you an error:`TypeError: 'tuple' object does not support item assignment`. In the computational boot camp, we saw one use of tuples in [Manipulating and Visualizing Data](data.ipynb) where the shape of the table is reported using a tuple:<jupyter_code>import pandas
prices = pandas.read_csv('data/SampleSuperstoreSales.csv')
print prices.shape<jupyter_output>(8399, 21)<jupyter_text>Because tuples are ordered, we could also use list notation to group the elements of a tuple:<jupyter_code>q = [32,9,9732]
print type(q)
print q
listviz(q)<jupyter_output><type 'list'>
[32, 9, 9732]<jupyter_text>When I first started programming in Python and encountered tuples, they seemed redundant and an unnecessary complexity. But, I've come to appreciate the distinction between tuples and lists. I often think of tuples as **unordered** to group some data, though, I still have to access the elements using array index notation. In other words, I view tuples as *associations* or grouping of elements. It's often the case that the elements of a single tuple have different types.Relevant to our topic of document analysis, we'll associate a word (string) with the frequency (integer) with which it occurs in the document. For example, if the word "cat" appears 10 times, we'd create a tuple like this:<jupyter_code>a = ('cat', 10)
print type(a)
print a<jupyter_output><type 'tuple'>
('cat', 10)<jupyter_text>The tuple notation works even when the values are variables not literals:<jupyter_code>word = 'cat'
freq = 10
a = (word, freq)
listviz(a)<jupyter_output><empty_output><jupyter_text>We could define `class`es to access the elements by name, such as `a.word` instead of `a[0]`, but that is beyond what we can cover in this course.  Bag of words representationA document is a sequence of words that we can represent simply as a list of strings. For example, let's split apart a simple document into words:<jupyter_code>doc = 'the cat sat on the hat'
words = doc.split(' ')
listviz(words)<jupyter_output><empty_output><jupyter_text>Using a list of words preserves the order, which is sometimes important, but often it is not.  When order is not important, we can use the [bag of words](https://en.wikipedia.org/wiki/Bag-of-words_model) representation of a document that records the words and their frequencies. This not only compresses the space needed to represent the document but it also tells us something about the meaning of the document.  For example,   examining the most common words is often useful for searching and machine learning problems. One representation for bag of words is just a list of associations (order of tuples doesn't matter):<jupyter_code>from lolviz import *
bag = [('the',2), ('cat',1), ('sat',1), ('hat',1)]
listviz(bag, showassoc=False)<jupyter_output><empty_output><jupyter_text>Because we are really using the tuples as associations, I prefer to think of the data structure like this:<jupyter_code>listviz(bag)<jupyter_output><empty_output><jupyter_text>That representation is a faithful representation of a bag of words, but looking up word frequencies is not efficient. To find a word, we must linearly scan the list of tuples looking for the word and then plucking out the frequency.  DictionariesA list of tuples representing a list of associations is a perfectly fine way to represent a bag of words.  It implies an order because it's in a list but we could ignore that. The biggest problem is that lists are slow to search when they get big. It turns out that if we use a dictionary, we can look up words very quickly using a hash table implementation as you build in your project. To create a dictionary from a list of associations is easy:<jupyter_code>d = dict(bag)
print d
dictviz(d)<jupyter_output>{'the': 2, 'sat': 1, 'hat': 1, 'cat': 1}<jupyter_text>Python prints dictionaries out using `dict` literal notation, which we can use to define dictionaries directly:<jupyter_code>e = {'the': 2, 'sat': 1, 'hat': 1, 'cat': 1}
print e
dictviz(e)<jupyter_output>{'the': 2, 'cat': 1, 'hat': 1, 'sat': 1}<jupyter_text>Given a dictionary, you can get a list of the associations out as a list of tuples:<jupyter_code>tuples = e.items()
print tuples
listviz(tuples)<jupyter_output>[('the', 2), ('cat', 1), ('hat', 1), ('sat', 1)]<jupyter_text>In implementation, however, dictionaries are actually more complicated than lists of associations in order to get the speed.Accessing elements of the dictionary looks like array indexing except that the index is an arbitrary object, such as a string in our case:<jupyter_code>print d['the']
print d['hat']
d['hat'] = 99    # Replace the value for key hat with 99
print d['hat']
dictviz(d)
q = {'ronald':{3,4}, 'reagan':{19}}
dictviz(q)<jupyter_output><empty_output><jupyter_text>Trying to access a key that does not exist in the dictionary causes a `KeyError` so it's best to check if the key exists first:<jupyter_code># print d['foo']     # This would cause a KeyError!
if 'cat' in d:       # hat is indeed in dictionary d
    print d['cat']
if 'foo' in d:       # does not exist so we don't get an error on the next line    
    print d['foo']   # does not execute<jupyter_output>1<jupyter_text>Pointer variablesExcept for the atomic elements like integers and floating-point numbers, variables in Python refer or *point* at the data element in memory. For example, here is a visual representation of how Python holds a few numeric values in memory:<jupyter_code>i = 1
price = 99.34<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0862590799031477, 'fraction_numerical': 0.016041162227602906, 'mean_word_length': 5.524185587364265, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '545290', 'license': 'MIT', 'n_tokens_mistral': 2047, 'n_tokens_neox': 1919, 'n_words': 898, 'path': 'notes/dict.ipynb', 'repo_name': 'akkiittiwari/msan692', 'size': '6608'}