<jupyter_start><jupyter_text>Data Manipulation with Pandas- [Download the lecture notes](https://philchodrow.github.io/PIC16A/content/pd/pd_1.ipynb). The `pandas` package for Python offers a set of powerful tools for working with tabular data. The name `pandas` is originally derived from the common term "panel data." While different programmers pronounce the package name in different ways, I prefer to pronounce it like the plural of "panda [bears]". I then imagine a small army of adorable bears performing my computations for me.     This panda is upside down because it is sorting data in reverse order.The first step when working with `pandas` is always to `import` it:<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>CSV Data, RevisitedA few weeks ago, we discussed some tools for reading CSV data from files using the `csv` module. `pandas` is usually a better choice for reading (and working with) CSV data. To read a CSV file  from data, we use the function `pd.read_csv()`. First, the following code block will place a copy of our data into the current working directory.<jupyter_code>import urllib

url = "https://philchodrow.github.io/PIC16A/datasets/palmer_penguins.csv"
filedata = urllib.request.urlopen(url)
to_write = filedata.read()

with open("palmer_penguins.csv", "wb") as f:
    f.write(to_write)<jupyter_output><empty_output><jupyter_text>Next, we can read in the the file as a DataFrame:<jupyter_code>penguins = pd.read_csv("palmer_penguins.csv")
type(penguins)<jupyter_output><empty_output><jupyter_text>Now let's inspect our new DataFrame:<jupyter_code>penguins
penguins.shape # (rows, columns)
penguins.dtypes # data type of each column.<jupyter_output><empty_output><jupyter_text>A data type of `object` means that `pandas` isn't sure what kind of data are in the corresponding columns. This is very common when the columns contain strings. [It is possible](https://pandas.pydata.org/pandas-docs/stable/user_guide/text.html) to give string columns a dedicated data type, but we won't focus on that here.  A pleasant way to get a quick overview of the numerical columns in your data set is the `describe` method.<jupyter_code>penguins.describe()<jupyter_output><empty_output><jupyter_text>Parts of a Data Frame When working with data frames, it's important to get comfortable with their different parts. These are: 1. The index. The index is used to refer to **rows**. In many cases, you can think of the index as a unique numerical label for a row. 2. The column names. These tell you what kinds of data appear in each row. It is important to be able to comfortably grab columns from the data frame for use in computations. 3. The data itself. You can think of the data as a set of different arrays, one for each column name. Each array has the same length. Many of the methods of these arrays will be familiar from `np.array`s.     The parts of a data frame.Let's now begin to look at how to obtain different parts of a data frame. Selecting ColumnsThe easiest way to select a column of a data frame is to pass the name of the column to the DataFrame with `[]` brackets. In this way, you can think of a data frame as being similar to a dictionary whose keys are the column names.<jupyter_code>penguins['Region']<jupyter_output><empty_output><jupyter_text>The result is no longer a data frame, but rather a `pd.Series` object, which is similar to a `np.array`.  To select multiple columns, pass a list of column names:<jupyter_code>penguins[['Species', 'Region', 'Island']]<jupyter_output><empty_output><jupyter_text>This time, the result is a data frame containing the specified columns.<jupyter_code># L1
# intro, parts of a df, select
# indexing and filter


# L2
# mutate
# group_by, summarise

# L3
# code patterns with visualization<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07398692810457516, 'fraction_numerical': 0.002875816993464052, 'mean_word_length': 5.161030595813204, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '84931', 'license': 'MIT', 'n_tokens_mistral': 1143, 'n_tokens_neox': 1110, 'n_words': 510, 'path': 'content/pd/pd_1.ipynb', 'repo_name': 'mm5110/PIC16A', 'size': '3825'}