<jupyter_start><jupyter_text>Starting With Data Working With Pandas DataFrames in PythonWe can automate the process of performing data manipulations in Python. It's efficient to spend timebuilding the code to perform these tasks because once it's built, we can use itover and over on different datasets that use a similar format. This makes ourmethods easily reproducible. We can also easily share our code with colleaguesand they can replicate the same analysis. Starting in the same spotTo help the lesson run smoothly, let's ensure everyone is in the same directory.This should help us avoid path and file name issues. At this time pleasenavigate to the workshop directory. If you working in IPython Notebook be surethat you start your notebook in the workshop directory.A quick aside that there are Python libraries like [OSLibrary](https://docs.python.org/3/library/os.html) that can work with ourdirectory structure, however, that is not our focus today. Our DataFor this lesson, we will be using the Portal Teaching data, a subset of the datafrom Ernst et al[Long-term monitoring and experimental manipulation of a Chihuahuan Desert ecosystem near Portal, Arizona, USA](http://www.esapubs.org/archive/ecol/E090/118/default.htm)We will be using files from the [Portal Project Teaching Database](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459).This section will use the `surveys.csv` file that can be downloaded here:[https://ndownloader.figshare.com/files/2292172](https://ndownloader.figshare.com/files/2292172)We are studying the species and weight of animals caught in plots in our studyarea. The dataset is stored as a `.csv` file: each row holds information for asingle animal, and the columns represent:| Column           | Description                        ||------------------|------------------------------------|| record_id        | Unique id for the observation      || month            | month of observation               || day              | day of observation                 || year             | year of observation                || plot_id          | ID of a particular plot            || species_id       | 2-letter code                      || sex              | sex of animal ("M", "F")           || hindfoot_length  | length of the hindfoot in mm       || weight           | weight of the animal in grams      |The first few rows of our first file look like this:<jupyter_code>record_id,month,day,year,plot_id,species_id,sex,hindfoot_length,weight
1,7,16,1977,2,NL,M,32,
2,7,16,1977,3,NL,M,33,
3,7,16,1977,2,DM,F,37,
4,7,16,1977,7,DM,M,36,
5,7,16,1977,3,DM,M,35,
6,7,16,1977,1,PF,M,14,
7,7,16,1977,2,PE,F,,
8,7,16,1977,1,DM,M,37,
9,7,16,1977,1,DM,F,34,<jupyter_output><empty_output><jupyter_text>About LibrariesA library in Python contains a set of tools (called functions) that performtasks on our data. Importing a library is like getting a piece of lab equipmentout of a storage locker and setting it up on the bench for use in a project.Once a library is set up, it can be used or called to perform many tasks. Pandas in PythonOne of the best options for working with tabular data in Python is to use the[Python Data Analysis Library](http://pandas.pydata.org/) (a.k.a. Pandas). ThePandas library provides data structures, produces high quality plots with[matplotlib](http://matplotlib.org/) and integrates nicely with other librariesthat use [NumPy](http://www.numpy.org/) (which is another Python library) arrays.Python doesn't load all of the libraries available to it by default. We have toadd an `import` statement to our code in order to use library functions. To importa library, we use the syntax `import libraryName`. If we want to give thelibrary a nickname to shorten the command, we can add `as nickNameHere`.  Anexample of importing the pandas library using the common nickname `pd` is below. Each time we call a function that's in a library, we use the syntax`LibraryName.FunctionName`. Adding the library name with a `.` before thefunction name tells Python where to find the function. In the example above, wehave imported Pandas as `pd`. This means we don't have to type out `pandas` eachtime we call a Pandas function. Reading CSV Data Using PandasWe will begin by locating and reading our survey data which are in CSV format.We can use Pandas' `read_csv` function to pull the file directly into a[DataFrame](http://pandas.pydata.org/pandas-docs/stable/dsintro.htmldataframe). So What's a DataFrame?A DataFrame is a 2-dimensional data structure that can store data of differenttypes (including characters, integers, floating point values, factors and more)in columns. It is similar to a spreadsheet or an SQL table or the `data.frame` inR. A DataFrame always has an index (0-based). An index refers to the position ofan element in the data structure. Notice when you assign the imported DataFrame to a variable, Python does notproduce any output on the screen. We can view the value of the `surveys_df`object by typing its name into the Python command prompt. which prints contents like above Exploring Our Species Survey DataAgain, we can use the `type` function to see what kind of thing `surveys_df` is: As expected, it's a DataFrame (or, to use the full name that Python uses to referto it internally, a `pandas.core.frame.DataFrame`).What kind of things does `surveys_df` contain? DataFrames have an attributecalled `dtypes` that answers this: All the values in a column have the same type. For example, months have type`int64`, which is a kind of integer. Cells in the month column cannot havefractional values, but the weight and hindfoot_length columns can, because theyhave type `float64`. The `object` type doesn't have a very helpful name, but inthis case it represents strings (such as 'M' and 'F' in the case of sex).We'll talk a bit more about what the different formats mean in a different lesson. Useful Ways to View DataFrame objects in PythonThere are many ways to summarize and access the data stored in DataFrames,using attributes and methods provided by the DataFrame object.To access an attribute, use the DataFrame object name followed by the attributename `df_object.attribute`. Using the DataFrame `surveys_df` and attribute`columns`, an index of all the column names in the DataFrame can be accessedwith `surveys_df.columns`.Methods are called in a similar fashion using the syntax `df_object.method()`.As an example, `surveys_df.head()` gets the first few rows in the DataFrame`surveys_df` using **the `head()` method**. With a method, we can supply extrainformation in the parens to control behaviour.Let's look at the data using these.>  Challenge - DataFrames>> Using our DataFrame `surveys_df`, try out the attributes & methods below to see> what they return.>> 1. `surveys_df.columns`> 2. `surveys_df.shape` Take note of the output of `shape` - what format does it>    return the shape of the DataFrame in?>    >    HINT: [More on tuples, here](https://docs.python.org/3/tutorial/datastructures.htmltuples-and-sequences).> 3. `surveys_df.head()` Also, what does `surveys_df.head(15)` do?> 4. `surveys_df.tail()`{: .challenge}  Calculating Statistics From Data In A Pandas DataFrameWe've read our data into Python. Next, let's perform some quick summarystatistics to learn more about the data that we're working with. We might wantto know how many animals were collected in each plot, or how many of eachspecies were caught. We can perform summary stats quickly using groups. Butfirst we need to figure out what we want to group by.Let's begin by exploring our data: Let's get a list of all the species. The `pd.unique` function tells us all ofthe unique values in the `species_id` column. >  Challenge - Statistics>> 1. Create a list of unique plot ID's found in the surveys data. Call it>   `plot_names`. How many unique plots are there in the data? How many unique>   species are in the data?>> 2. What is the difference between `len(plot_names)` and `surveys_df['plot_id'].nunique()`? Groups in PandasWe often want to calculate summary statistics grouped by subsets or attributeswithin fields of our data. For example, we might want to calculate the averageweight of all individuals per plot.We can calculate basic statistics for all records in a single column using thesyntax below: We can also extract one specific metric if we wish: But if we want to summarize by one or more variables, for example sex, we canuse **Pandas' `.groupby` method**. Once we've created a groupby DataFrame, wecan quickly calculate summary statistics by a group of our choice.<jupyter_code># Group data by sex<jupyter_output><empty_output><jupyter_text>The **pandas function `describe`** will return descriptive stats including: mean,median, max, min, std and count for a particular column in the data. Pandas'`describe` function will only return summary values for columns containingnumeric data.<jupyter_code># summary statistics for all numeric columns by sex

# provide the mean for each numeric column by sex<jupyter_output><empty_output><jupyter_text>The `groupby` command is powerful in that it allows us to quickly generatesummary stats.summary stats.>  Challenge - Summary Data>> 1. How many recorded individuals are female `F` and how many male `M`> 2. What happens when you group by two columns using the following syntax and>    then grab mean values:>	- `grouped_data2 = surveys_df.groupby(['plot_id','sex'])`>	- `grouped_data2.mean()`> 3. Summarize weight values for each plot in your data. HINT: you can use the>   following syntax to only create summary statistics for one column in your data>   `by_plot['weight'].describe()` Quickly Creating Summary Counts in PandasLet's next count the number of samples for each species. We can do this in a fewways, but we'll use `groupby` combined with a **`count()` method**.<jupyter_code># count the number of samples by species<jupyter_output><empty_output><jupyter_text>Or, we can also count just the rows that have the species "DO": >  Challenge - Make a list>>  What's another way to create a list of species and associated `count` of the>  records in the data? Hint: you can perform `count`, `min`, etc functions on>  groupby DataFrames in the same way you can perform them on regular DataFrames. Basic Math FunctionsIf we wanted to, we could perform math on an entire column of our data. Forexample let's multiply all weight values by 2. A more practical use of this mightbe to normalize the data according to a mean, area, or some other valuecalculated from our data.<jupyter_code># multiply all weight values by 2<jupyter_output><empty_output><jupyter_text>Quick & Easy Plotting Data Using PandasWe can plot our summary stats using Pandas, too.<jupyter_code># make sure figures appear inline in Ipython Notebook

# create a quick bar chart<jupyter_output><empty_output><jupyter_text>We can also look at how many animals were captured in each plot: >  Challenge - Plots>> 1. Create a plot of average weight across all species per plot.> 2. Create a plot of total males versus total females for the entire dataset.{: .challenge}>  Summary Plotting Challenge>> Create a stacked bar plot, with weight on the Y axis, and the stacked variable> being sex. The plot should show total weight by sex for each plot. Some> tips are below to help you solve this challenge:>> * [For more on Pandas plots, visit this link.](http://pandas.pydata.org/pandas-docs/stable/visualization.htmlbasic-plotting-plot)> * You can use the code that follows to create a stacked bar plot but the data to stack>  need to be in individual columns.  Here's a simple example with some data where>  'a', 'b', and 'c' are the groups, and 'one' and 'two' are the subgroups.> > We can plot the above with<jupyter_code># plot stacked data so columns 'one' and 'two' are stacked<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07382663847780127, 'fraction_numerical': 0.012684989429175475, 'mean_word_length': 4.4, 'pattern_counts': {'":': 1, '<': 29, '<?xml version=': 0, '>': 79, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '382669', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3268, 'n_tokens_neox': 3069, 'n_words': 1632, 'path': '_episodes_pynb/01-Starting_with_data_clean.ipynb', 'repo_name': 'scw-ss/-2018-06-27-cfmehu-python-ecology-lesson', 'size': '11825'}