<jupyter_start><jupyter_text>Analzye Data using Straight PythonThis example analyzes one of our test data sets using just straight python listelements (not pandas) but strives to avoid loops as much as possible and thinkabout the data as a data set. We begin by loading the file into a list of items using the "csv" package.<jupyter_code>import csv

file = "datasets/lunch_10.csv"
rawrows = []
with open( file, newline='') as csvfile:
    rd = csv.reader( csvfile )
    for row in rd:
        rawrows.append( row )

labels = rawrows[ 0 ]
rows = rawrows[ 1: ]<jupyter_output><empty_output><jupyter_text>Alright lets get some basic feel for the data. What are the labels? How many rows are there?<jupyter_code>labels
len( rows )<jupyter_output><empty_output><jupyter_text>And now we want to ask some questions. How many unique schools are in my dataset? How manydays does my dataset cover?<jupyter_code># So lets create a little useful dictionary to allow us to talk about columns
# by name
lDict = {}
for i,n in enumerate( labels ):
    lDict[ n ] = i
# We can find the unique elements by casting to a set.
schools = set( [ r[lDict["School"]] for r in rows ] )
[ schools, len( schools ) ]
# Similarly we can find the unique days
len( set ( [ r[ lDict["Date"] ] for r in rows ] ))<jupyter_output><empty_output><jupyter_text>OK so now we need to do something harder. For each school we have the number of studentsand the number in attendance on a given day. So we can figure out the average daily attendance? That's a bit tricky because the data is all in one set and we want to derive an answer by 'grouping' the data into each school. There are lots of ways to do this and toolkits like 'pandas' make it really easy butsince we are working just with a list of elements here, we need to recreate that grouping a bit.<jupyter_code>attendance = dict()
for s in schools:
    attendance[ s ] = [ 0, 0 ]

for r in rows:
    school = r[ lDict[ "School" ]]
    enroll = int(r[ lDict[ "Enrollment" ] ])
    attend = int(r[ lDict[ "Attendance" ] ])
    
    prior = attendance[ school ]
    newval = [ prior[ 0 ] + enroll, prior[ 1 ] + attend ]
    attendance[ school ] = newval

result = dict()
for k,v in attendance.items():
    result[ k ] = v[ 1 ] / v[ 0 ] * 100

result<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08765808983863933, 'fraction_numerical': 0.0056694286960314, 'mean_word_length': 3.9439655172413794, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '166209', 'license': 'Apache-2.0', 'n_tokens_mistral': 731, 'n_tokens_neox': 709, 'n_words': 328, 'path': 'Analyze_Lists.ipynb', 'repo_name': 'baconpaul/dataset_exercises', 'size': '2293'}