<jupyter_start><jupyter_text>Unit 2:  Homework Option III -- Working With Large Files This homework provides students with the option to get practice opening, sampling, and cleaning data when it has file sizes that might stretch your available RAM, or even exceed it.  **What you'll turn in:**You'll make a pull request with a folder (titled as your name) that contains the following: - for section I, a file called `titanic`, stored in a binary `feather` or `parquet` file format, that reduces the `titanic` dataset to its smallest possible memory amount without tampering any of its values. - for section II, simply turn in this notebook with the questions answered. - for section III, a script called `chunking.py` that, when run, will connect to an s3 bucket, and stream a file through memory, clean it, and output it to a `.csv` file, even though the entire file was never entirely loaded into RAM.  Section I: Downcasting the Titanic Dataset **What You'll Learn:** The basics of managing memory within files, and how to use advanced file formats such as the `feather` or `parquet`, to make it easier to maintain persistent data types when a file isn't loaded.**What You'll Turn In:** A file called `titanic`, which contains the memory-reduced form of the file. Downcasting`Downcasting` is the task of reducing the memory footprint of different columns in your dataset so they take up less RAM when you load them in.  Most software used for handling data makes use of your available RAM to process its tasks.  If the size of your file neatly fits into the available RAM that your computer has then this is fine.  If it's significantly larger (no laptop is going to have 1TB of RAM for example), then you won't be able to load in the file and work with it.Pandas works this way, and therefore the amount of working RAM you have available to use is going to function as a limit for what file sizes you can work with.    A Quick Intro to Data Types in Pandas & Numpy Numbers come in different flavors in pandas and numpy.  At the simplest level you have integers (whole numbers) and floating point numbers (numbers with decimals).  However, numbers use different sized containers to store their values.  They are as follows: - **64 bit:** Can store values as large as 264, which is 18446744073709551616 - **32 bit:** Can store values as large as 232, which is 4294967296 - **16 bit:** Good for values up to 216, 65536 - **8 bit:**  28, or up to 264 **Integers** typically have the above range, while **floats** typically can only go down to 32 bits.You can see the whole range of numeric data types here:  https://docs.scipy.org/doc/numpy/user/basics.types.htmlThe important detail here, is that if a number is encoded as being a 64 bit number, it will *always* use the same amount of memory to store it, even if the value itself is much smaller.  So, if you have a column of 0's and 1's, an 8 bit encoding will work perfectly fine (since the values are less than 28), and a 64 bit encoding will take up 8x as much memory as it needs to.An important detail about how Pandas works is that *all numbers are automatically encoded as 64 bit numbers*.  This is good for making sure values aren't tampered with, but bad for optimizing memory with large files.**Methods Used For Managing Memory In Pandas:** - `df.memory_usage()`, returns the memory usage, in bytes, of whatever is selected. - `pd.Series.astype()`, allows you to change the data type of 1 variable to another. - `df.info()`, returns the data type and memory usage of every column selected - `df.dtypes()/pd.Series.dtype()`, returns the data type of everything selected Let's take a look at how these items work.  Run the following cells to get a quick demonstration of what they do.<jupyter_code># load in the titanic dataset here -- use a different url if need be to
# load it in
import pandas as pd
import numpy as np
df = pd.read_csv('../data/titanic.csv')
# check the info of your dataset -- notice the 64 bit numbers
df.info()
# We can also check the memory usage of each column
df.memory_usage()
# now let's see what happens when we adjust a columns data type
df['Survived'].astype(np.int8).memory_usage()<jupyter_output><empty_output><jupyter_text>As you can see, an 8 bit number takes up about 1/8 as much memory as a 64 bit number.  Permanently changing something's data type is simple.<jupyter_code># this will change the data type of a column to something else
df['Survived'] = df['Survived'].astype(np.int8)
# and now we can see its memory footprint is permanently smaller
df.memory_usage()<jupyter_output><empty_output><jupyter_text>Notice however, that if you make a columns data type *smaller* than what it is, the original values will be tampered.  For example, the Passenger ID column has values as large as 891.....which is more than 28.  Notice what happens when you make the change:<jupyter_code># the values at the end of the series should be 889, 890, 891, etc
df['PassengerId'].astype(np.int8)<jupyter_output><empty_output><jupyter_text>So clearly, getting things 'just right' is important.  Notice also the difference between **signed** and **unsigned** data types.  If it's signed, that means they can accept negative values.  So, a datatype of `np.uint8` can accept ranges of 0 - 255, whereas `np.int8` accepts values from -128 to 127.  Categorical Data in Pandas and Numpy Text based data in Pandas and Numpy has two different varieties: - **np.object**: this is the default numpy way of treating and handling data.  Pretty close to a python string, and is used to store data that doesn't have any other characteristic (integer, float, bool, etc) - **category**: this is a special data type built specifically for Pandas, to handle text data that has a small number of repeating values.  Like the `sex` column in our dataset.  When appropriately used, it can drastically reduce the memory footprint of text based data.  You can read more about them here:  https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html See below for a demonstration.<jupyter_code># the Sex column currently uses the same memory as a 64 bit number
df['Sex'].memory_usage()
# if you change it to a category, it uses about the same amount
# as an 8 bit number
df['Sex'].astype('category').memory_usage()<jupyter_output><empty_output><jupyter_text>Please note that using categories only has the desired effect when there are repeating values.  Make sure to check the memory footprint before making the change!  Section I Task:  Clean Up the Titanic Dataset For the first part of your homework assignment, your task is to reduce the memory footprint of the titanic dataset as much as possible, while not tampering any values, and export it to a binary file format called `feather`(preferably), or `parquet` which will maintain information about all of its data types when it's loaded back in. You can read more about the feather file format here:  https://blog.rstudio.com/2016/03/29/feather/.The file `titanic` should be turned in inside a folder with your name on it, and it will be inspected to make sure it was downcasted in the most appropriate manner.You can use the `to_feather()`, and `read_feather()` methods to load and export your files.  Or the `to_parquet()` and `read_parquet()` methods, respectively.**Note:** It is best to use feather files if you can.  They are the only file format that can maintain persistent information about the `category` data type after it's been saved.However, you will likely need some additional libraries to get it working.  Often pandas does not work well with `feather` files right out of the box.  The typical library used to work with feather files is, appropriately called......feather.  To install the `feather` library, simply go to Anaconda Prompt/Terminal and type in `conda install feather-format`.The library's homepage can be found here:  https://pypi.org/project/feather-format/This library isn't always well supported, and there's a possibility that you might have an exceedingly difficult time getting it to work.  The purpose of this homework is not to have you go down rabbit holes getting an obscure library to work.  If after 30 minutes - 1 hour you still don't have the file installed, feel free to use the `parquet` file format, which can be a little easier to work with.You can find more about it here:  https://arrow.apache.org/docs/python/parquet.html  Section II: Working With Larger File Sizes Section I was your warmup to get the hang of how to reduce the memory of your data and get it loaded into a more advanced file format that can be reused for other projects.This section will extend what you just did, but add in two additional wrinkles: - The file will be much larger -- approximately 2 million rows - You'll find out what types of data types you should be working with.....*without* reading in the entire file in the first place. Imagine this hypothetical scenario:  there's a 10 gb file that you need to work on, and loading it into pandas is returning a `MemoryError`.  You're almost certain you could reduce it to something much smaller......except you can't even read it in to figure out what to do next.These types of chicken & egg problems are fairly common, and a popular antidote to them is to sample in a portion of your dataframe.  There are two primary ways to do this if you're reading in `.csv` files: - `nrows`   : tells you how many rows to read in from the original file - `skiprows`: tells you which rows to *skip* from the original file For both of these you can just manually read in x number of rows relatively easily, but doing so right from the beginning or end has some problems.  Mainly, many datasets don't have consistent values from beginning to end.  For example, if you have a dataset with 10 years of sales info, it's very possible values being recorded are very different at varying time segments.  Often new columns are added to datasets in the middle of their collection, and all values for all times before that are simply `null`, so just reading in the first 5000 or 10000 might not give you a consistent picture of what to expect.For this reason, it's good to randomly sample in dataframes before you want to read them in entirely.To do this, you can mix `lambda` functions (remember those?) with the `skiprows` argument, which can accept a function as values.Here's the basic idea:<jupyter_code>import random

# random.random() will generate a random value between 0 and 1
# so this will return True 50% of the time
random.random() > .50<jupyter_output><empty_output><jupyter_text>Using this same logic, we can pass this into a lambda function like so, to read in 30% of the titanic dataset.<jupyter_code># use read_csv with a lambda function
df = pd.read_csv('../data/titanic.csv', skiprows=lambda x: x > 0 and random.random() > 0.3)
# and notice the size of the dataframe that was read in
df.shape<jupyter_output><empty_output><jupyter_text>Here, inside the `lambda` function, `x` represents the index value of the row being read in.  `x > 0` is used assuming the first row are headers, and of course `random.random() > 0.3` will return `True` 70% of the time, hence the results that we get. The idea is that you can read in a very small fraction of a very large file if you want to investigate its most important properties.  Now, this begs the question.....how do you specify the data types you want a column to be before you read it in?  There is a very useful argument in `read_csv` called `dtype` that accepts a dictionary, where you can list column labels as keys, and their corresponding data type as a value.So for example, if we wanted to change the `Embarked` column and the `Survived` column to `category` and `np.int8`, we could do so in the following way.<jupyter_code># this dictionary contains the columns we want to change
dtypes = {
    # column label: new data type
    'Embarked': 'category',
    'Survived': np.int8
}

# and now we'll re-read in the .csv file, and use the dtypes dict
df = pd.read_csv('../data/titanic.csv', dtype=dtypes)
# and we can see that the data type of the columns are in fact changed
df.info()<jupyter_output><empty_output><jupyter_text>Section II Task: Sample, Clean Up, And Read in the taxi.csv fileThis portion of section II contains the task that you will be graded on.  You can simply answer the prompts inside this notebook and turn it in.  No additional files are necessary.  The file is located in an S3 bucket at this location:  `https://dat-data.s3.amazonaws.com/taxi.csv`It records information about every taxi ride given by a particular company for approximately 1 year. **Part I:** Randomly sample in 10% the taxi.csv file<jupyter_code># your answer here<jupyter_output><empty_output><jupyter_text>**Part II:** Go ahead and do the appropriate exploratory data analysis to figure out the most appropriate data type for each column.<jupyter_code># your answer here<jupyter_output><empty_output><jupyter_text>**Part III**: Create a dictionary that contains the key/value pairs for each column that needs to be changed to a different data type, and then read in the file.<jupyter_code># your answer here<jupyter_output><empty_output><jupyter_text>**Part IV:** Confirm that each column has the appropriate data type<jupyter_code># your answer here<jupyter_output><empty_output><jupyter_text>Section III (Optional): File Streaming File streaming is a way to get around memory limitations in pandas.  When you stream in a file, you spoon feed a portion of it into memory, and when you're finished, load in the next portion, and so on until there's nothing left. It's less convenient than regular file I/O, but it removes any sort of memory limit you might face when working with a file because it's never loaded at the same time. In this section of the homework assignment, you'll be tasked with performing basic cleaning operations on a file......without ever having to load it into memory.  This means what you accomplish in this section you could perform on a file of any size.  Basic Introduction to File StreamsWhen it's available, you can specify a file stream by using the `chunksize` argument, which specifies that you only read in so many lines at a time.Notice how it works:<jupyter_code># when we read in the csv file, we'll set chunksize to 200
df = pd.read_csv('../data/titanic.csv', chunksize=250)

# notice that df is NOT a df.....it's a file stream
type(df)
# and now if we want, we can 'chunk' in 250 rows at a time
df.get_chunk()
# and this would be the next 250 rows
df.get_chunk()<jupyter_output><empty_output><jupyter_text>With this basic syntax, you can lazily read in bits and pieces of a file at a time, without having to worry about being able to fit the entire file into RAM at once.  This is useful for doing some basic exploratory data analysis, and peek into a file, even if it's unreasonably large.However, what if you wanted some way to go through the *entire* file, and make bulk changes to it?  You can do that as well, by looping through the file stream. Here's a simple example, that prints off the total memory footprint for each chunk being read in:<jupyter_code># this basically reads: for every chunk in the file stream
for chunk in pd.read_csv('../data/titanic.csv', chunksize=200):
    # do this to each chunk
    print(chunk.memory_usage().sum())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06351272206676645, 'fraction_numerical': 0.01145311381531854, 'mean_word_length': 4.461265103056148, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 58, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '267544', 'license': 'MIT', 'n_tokens_mistral': 4187, 'n_tokens_neox': 3925, 'n_words': 2389, 'path': 'DAT-02-14-main/Homework/Unit1/project-eda-downcasting-streaming.ipynb', 'repo_name': 'bweiss415/my-ga-repo', 'size': '15367'}