<jupyter_start><jupyter_text>Cultural AnalyticsENGL64.05 Fall 2019---- Lists and Metadata Processing Rev: 08/23/2019<jupyter_code># We can create a simple list with square brackets:
var = [1,2,3,4,5]
# Again, to identify the datatype of a variable, we can use the type() function:
type(var)
# We can access list items by requesting the item in sequence.
# This works much like a string. To get the first character of
# a string, we use square brackets and 0 for the zero-th element:
# var[0]
# We use the same method with lists. This should return '2':

var[1]
# We can iterate through a list with a 'for loop'. We name the local variable
# that will be assigned a value for each list item.

# If we've not noticed this before, a for loop has three important features:
#
# 1) we name the local variable and the object to iterate through
# 2) follow the object to iterate through with a colon ':'
# 3) the instructions that will be iterated are indented (all to the same point)
#

for number in var:
    print(number,type(number))
# Note how the above shows that Python has autotyped each item in the list
# as an integer.

# Now, we can create a list for a book. Note that we can break lines easily
# here to make this readable.

metadata = ["<NAME>",
            "Narrative of the Life of <NAME>, an American Slave. Written by Himself"]

# This provides us with a semi-computable list of strings.
# But what if we want to add the year of publication?
# We can append a list with the append method. It operates directly on the list
# and modifies the list.
#
# Strings are immutable while lists are mutable.

# If we add just the year, it will be autotyped as an integer:
metadata.append(1845)
print(metadata)
# But what if we have multiple books? We can create what is called a list-of-lists.
# We can start be creating an empty list to hold all the items:

book_metadata = list()
book_metadata.append(metadata)
print(book_metadata)
# Now we can easily add another entry:
book_metadata.append(['<NAME>',
                      'Life and Times of <NAME>, Written by Himself. His Early Life as a Slave, His Escape from Bondage, and His Complete History to the Present Time, Including His Connection with the Anti-slavery Movement; His Labors in Great Britain as Well as in His Own Country; His Experience in the Conduct of an Influential Newspaper; His Connection with the Underground Railroad; His Relations with <NAME> and the Harpers Ferry Raid;  His Recruiting the 54th and 55th Mass. Colored Regiments; His Interviews with Presidents Lincoln and Johnson; His Appointment by Gen. Grant to Accompany the Santo Domingo Commission&#x2014;Also to a Seat in the Council of the District of Columbia; His Appointment as United States Marshal by President <NAME>; Also His Appointment to Be Recorder of Deeds in Washington by President <NAME>; with Many Other Interesting and Important Events of His Most Eventful Life; With an Introduction by Mr. <NAME>, of Boston',
                      1892])
# We can now count the number of entries with the len() function:
len(book_metadata)<jupyter_output><empty_output><jupyter_text>Now we'll setup a more complicated approach to examining book metadata---<jupyter_code># This lines tell the Python interpreter to load some additional functions.
# We need functions to parse Comma Separated Value (CSV) data.

import csv

# we'll set a variable to hold the number of records read from this CSV file.
ln = 0

# as above, we'll create a metadata variable (overwritting the previous variable)
metadata=list()

# Now we open the CSV file and read each line, appending to the metadata list:
with open('data/Underwood_ch1/allgenremeta.csv', encoding = 'utf-8') as f:
    reader = csv.reader(f, delimiter = ',')
    for row in reader:
        metadata.append(row)
        
        # increment our counter
        ln += 1

# tell us how many entries we've read
print("read %s lines" % ln)
# The first line will give us some data about our metadata:
print(metadata[0])
# Let's remove the header (our "metadata") for easier processing:
metadata = metadata[1:]
# This metadata file contains 5,751 entries about a sample of books from 
# the 18th to 20th century. Some of the entries have uncertain publication 
# dates and there are three different genres:
#
# poe = poetry
# fic = fiction
# bio = biography

cleaned_up=list()
for book in metadata:

    # if the date just contains digits (i.e., just a year-of-publication)
    if book[2].isdigit() == True:
       
        # only select for fiction
        if book[5] == "fic":
            cleaned_up.append(book)
# How many books now?
print(len(cleaned_up))
# sort and print first and last entry
cleaned_metadata = sorted(cleaned_up, key = lambda x: x[2])
print(cleaned_metadata[:1])
print(cleaned_metadata[-1:])<jupyter_output><empty_output><jupyter_text>Title LengthsUsing our knowledge of conditional tests (if statements), we can get the average title length for each century of data.---<jupyter_code># Eighteenth Century
average = 0
i = 0
for book in cleaned_metadata:
    pub_year = int(book[2])
    if pub_year >= 1700 and pub_year <= 1799:
        i = i + 1
        average = average + len(book[10])
print(average/i)
# Nineteenth Century
average = 0
i = 0
for book in cleaned_metadata:
    pub_year = int(book[2])
    if pub_year >= 1800 and pub_year <= 1899:
        i = i + 1
        average = average + len(book[10])
print(average/i)
# Twentieth Century
average = 0
i = 0
for book in cleaned_metadata:
    pub_year = int(book[2])
    if pub_year >= 1900 and pub_year <= 1999:
        i = i + 1
        average = average + len(book[10])
print(average/i)<jupyter_output><empty_output><jupyter_text>Question: Now run the same for the other three genres. Any differences?<jupyter_code># We can iterate through the title list and search for some of the examples
# invoked by Moretti.

i = 0
for book in cleaned_metadata:
    title = book[10]
    if title.startswith("The"):
        i += 1
print(i)
# But was that all of them?
i = 0
for book in cleaned_metadata:
    title = book[10]
    if title.lower().startswith("the"):
        i += 1
print(i)
# What other patterns might we search for?<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07932924862947437, 'fraction_numerical': 0.018219929055143504, 'mean_word_length': 3.8010835913312695, 'pattern_counts': {'":': 1, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '101134', 'license': 'MIT', 'n_tokens_mistral': 1905, 'n_tokens_neox': 1773, 'n_words': 854, 'path': 'Python-Lab-02-Lists-Metadata.ipynb', 'repo_name': 'jeddobson/ENGL64.05-19F', 'size': '6277'}