<jupyter_start><jupyter_text>Arrays An array has to have elements of the same type.<jupyter_code>my_arr = make_array(42.5, 5.11, 777)
my_arr<jupyter_output><empty_output><jupyter_text>If we create an array of mixed types, Python will automatically force them all to be the same type. If an array contains even a single string, the entire resulting array will contain strings.<jupyter_code>my_arr = make_array(42, 5, 11, '8')
my_arr<jupyter_output><empty_output><jupyter_text>Adding a single number to an array adds it to every element in that array. If an array contains a single element, it is as if that element is just a single value. We can verify it by creating a new array of integers and another array with a single integer.<jupyter_code>arr2 = make_array(1, 5, 10, 0)
arr2
5 + arr2
arr1 = make_array(5)
arr1 + arr2<jupyter_output><empty_output><jupyter_text>Tables `Table()` will return an empty table. Since we rarely want just an empty table, we usually use it together with the `with_columns` function to add column names ("labels") and corresponding values to our table.<jupyter_code>empty_tbl = Table()
empty_tbl
type(empty_tbl)  # the type is supposed to be datascience.tables.Table<jupyter_output><empty_output><jupyter_text>We can either use `Table()` again to create a new empty table or we can start with an empty table that we just stored in `empty_tbl`. Notice that if we use an existing table, we do not need to use the `Table()` function.<jupyter_code>Table().with_columns('Tasks', 
                     make_array("Finish INT5 lab", 
                                "Do the dishes", 
                                "Buy snacks"))
empty_tbl.with_columns('Tasks', 
                     make_array("Finish INT5 lab", 
                                "Do the dishes", 
                                "Buy snacks"))<jupyter_output><empty_output><jupyter_text>Let's save this table into `my_todo` and add a couple more columns to it to keep track of tasks' priorities and whether we've done them. Notice that we can either use `empty_tbl`, which we already created or `Table()` to create an empty table.<jupyter_code>my_todo = Table().with_columns(
                    'Tasks', make_array("Finish INT5 lab", "Do the dishes", "Buy snacks"), # first column
                    'Priority', make_array(5, 3, 2), # second column
                    'Done', make_array(0, 1, 0) # third column
)
my_todo<jupyter_output><empty_output><jupyter_text>We could have alternatively, created the arrays separately, and then added them to the table.<jupyter_code>tasks = make_array("Finish INT5 lab", 
                   "Do the dishes", 
                   "Buy snacks")
tasks<jupyter_output><empty_output><jupyter_text>For each task above, let's add its priority: the higher the number, the more important the task.<jupyter_code>priority_values = make_array(10, 4, 1)
priority_values<jupyter_output><empty_output><jupyter_text>Let's indicate if a task is done `1` or not `0`.<jupyter_code>doneness = make_array(0, 1, 0)
doneness<jupyter_output><empty_output><jupyter_text>The table below looks just like the one we've created above.<jupyter_code>empty_tbl.with_columns(
    "Tasks", tasks, # first column
    "Priority", priority_values, # second column
    "Done", doneness # third column
)<jupyter_output><empty_output><jupyter_text>We also could have added the columns one at a time, instead of all at once. Notice, that we'll need to save the intermediate tables to be able to add more columns to them later. We'll start with the empty `todo` table but we also could have directly used the command that created it (`Table()`).<jupyter_code>empty_tbl # Table() empty table we created above
todo_tasks = Table().with_columns("Tasks", tasks) # first column
todo_tasks<jupyter_output><empty_output><jupyter_text>We are now going to add `priority_values` to the new table we just created (`todo_tasks`). Notice that instead of using the empty `todo` table to which we are adding columns, we are using `todo_tasks` to add the new column to it.<jupyter_code>todo_tasks_pr = todo_tasks.with_column(
    "Priority", priority_values
) # second column
todo_tasks_pr
empty_tbl<jupyter_output><empty_output><jupyter_text>If we used just `todo` or `Table()` with the above command, we would have been adding a new column to an **empty table** and would have gotten back a table with just one column.<jupyter_code>empty_tbl.with_columns('Priority', priority_values) 
# same as
Table().with_columns('Priority', priority_values) # preferred, since it is more explicit<jupyter_output><empty_output><jupyter_text>Now, let's add the last column to our table to indicate if a task is done `1` or not `0`.<jupyter_code>todo_all = todo_tasks_pr.with_columns(
    'Done', doneness) # third column
todo_all<jupyter_output><empty_output><jupyter_text>Since the commands work with the result of the previous commands (i.e., adding to the table that was previously created), we could also chain them together to achieve the same effect but without the need for the names of intermediate tables.<jupyter_code>Table().with_columns('Tasks', tasks).with_columns('Priority', priority_values)<jupyter_output><empty_output><jupyter_text>Note that `.num_rows`, `.num_columns`, and `.labels` are the only commands that we use that do not have `()` after them.<jupyter_code>todo_all.num_rows
todo_all.labels<jupyter_output><empty_output><jupyter_text>`relabeled` function leaves the original table intact and returns a new table with the specified column relabeled to a new name. To save that table, store it in a new variable name.<jupyter_code>todo_all.relabeled('Priority', 'Stars')
todo_all.labels
todo = todo_all.relabeled('Priority', 'Stars')
todo
task_stars = todo.select("Tasks")  # what happens if you mispsell the column name?
task_stars = todo.select("Tasks", "Stars")
task_stars<jupyter_output><empty_output><jupyter_text>Average priority of the tasks on my list.<jupyter_code>sum( todo.column("Stars") ) / todo.num_rows<jupyter_output><empty_output><jupyter_text>Instead of manually computing the average, we can use a `numpy` method `average` or `mean`. We need to make sure we prefix the function with `np.` and give it **an array** as an input.<jupyter_code>np.mean(todo.column("Stars"))
np.average(...)
todo.column("Stars").max()
max(todo.column("Stars"))<jupyter_output><empty_output><jupyter_text>Select the top 2 most important tasks... They are not in any order in the table, so we first should sort them.Let's put the most important tasks at the top of the table.<jupyter_code>todo.sort("Stars")
todo_sorted = todo.sort("Stars", descending=True)
todo_sorted
todo_sorted.take([0, 1])<jupyter_output><empty_output><jupyter_text>Notice that we have a task that we have finished is on the list. What if we want to get only the most important unfinished tasks? We'll need to exclude the "done" tasks first, then take the first two rows from the _sorted_ table.<jupyter_code>unfinished_tasks = todo.where("Done", are.equal_to(0))
unfinished_tasks
todo.where("Tasks", are.equal_to("Do the dishes"))<jupyter_output><empty_output><jupyter_text>What good is a to-do list if we cannot add more tasks to it?<jupyter_code>new_task = ['Practice table functions', 9, 0]
todo.with_row(new_task)<jupyter_output><empty_output><jupyter_text>Perhaps, we decided that the priority column is not necessary for us and everything we add to the list is important. We can remove that column by using a `drop` function, which needs to know which column you want to drop.<jupyter_code>todo.drop("Stars")
# todo.drop("Stars", "Done") # Can we drop more than one column?<jupyter_output><empty_output><jupyter_text>As usual, the command doesn't modify the original table, so we need to save the resulting table with a new name if we want to keep using it.<jupyter_code>todo<jupyter_output><empty_output><jupyter_text>Reading a table from a CSV file<jupyter_code>minard = Table().read_table("data/minard.csv")
minard
minard.select('Survivors') # returns a TABLE
minard.column('Survivors') # returns a array, since everything inside the column has to have the same type<jupyter_output><empty_output><jupyter_text>Now, you shouldn't be surprised to see that the line below results in an error.<jupyter_code>sum(minard.select('Survivors'))/minard.num_rows 
sum(minard.column('Survivors'))/minard.num_rows<jupyter_output><empty_output><jupyter_text>Visualizing Data<jupyter_code>import matplotlib
matplotlib.use('Agg', warn=False)
%matplotlib inline
import matplotlib.pyplot as plots
plots.style.use('fivethirtyeight')<jupyter_output><empty_output><jupyter_text>Let's look at the table and sort it by latitudes. How big is the difference between them?<jupyter_code>minard.sort("Latitude")<jupyter_output><empty_output><jupyter_text>Let's get all the latitudes into an array.<jupyter_code>lat = minard.column("Latitude")
lat
min_lat = lat.min()
# same as min(lat)
min_lat
max_lat = lat.max()
# same as max(lat)
max_lat
max_lat - min_lat<jupyter_output><empty_output><jupyter_text>Do the same for the Longitude to see which range is greater. We can also visualize them on a scatterplot, showing us the number of survivors at each recorded latitude.<jupyter_code>minard.scatter("Latitude", "Survivors")
minard.scatter("Longitude", "Survivors")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09424252171116114, 'fraction_numerical': 0.005896858582609628, 'mean_word_length': 5.311231393775373, 'pattern_counts': {'":': 0, '<': 121, '<?xml version=': 0, '>': 121, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '635851', 'license': 'MIT', 'n_tokens_mistral': 2965, 'n_tokens_neox': 2860, 'n_words': 1011, 'path': 'demo/lec07.ipynb', 'repo_name': 'ucsb-int5/int5-f19-notebooks', 'size': '9327'}