<jupyter_start><jupyter_text>This guide describes how to use pandas and Jupyter notebook to analyze a Socrata dataset. It will cover how to do basic analysis of a dataset using pandas functions and how to transform a dataset by mapping functions.  Contents1. [Installing Python and Jupyter](installing-python-and-jupyter)2. [Importing a Dataset Into Jupyter](importing-a-dataset-into-jupyter)3. [Basic Analysis of a Dataset](basic-analysis-of-a-dataset)4. [Mapping Functions to Transform Data](mapping-functions-to-transform-data) Installing Python and JupyterYou can run Jupyter notebook in the cloud using a service like https://try.jupyter.org/ or install and run it locally. To install it locally follow the instructions here: http://jupyter.readthedocs.org/en/latest/install.html. You will need to have or install Python 3.3 or greater or Python 4.   Import a Dataset Into Jupyter Before we import our sample dataset into the notebook we will import the pandas library. pandas is an open source Python library that provides "high-performance, easy-to-use data structures and data analysis tools." - http://pandas.pydata.org/<jupyter_code>import pandas as pd
print(pd.__version__)
pd.read_json("/g")<jupyter_output>1.3.0<jupyter_text>Next, we will read the following dataset from the Open San Mateo County site:https://data.smcgov.org/Government/Educational-Attainment/mb6a-xn89pandas provides several methods for reading data in differet formats. Here we'll read it in as json but you can read in csv and Excel files as well. Note that you can get the help for any method by adding a "!" to the end and running the cell. For example:<jupyter_code>pd.read_json?<jupyter_output><empty_output><jupyter_text>The data is returned as a "DataFrame" which is a 2 dimensional spreadsheet-like datastructure with columns of different types. pandas has two main data structures - the DataFrame and Series. A Series is a one-dimensional array that can hold any value type - This is not necessarily the case but a DataFrame column may be treated as a Series.Displayed below are the first 5 rows of the DataFrame we imported (to see the last n rows use .tail(n)).<jupyter_code>df = pd.read_json("https://data.smcgov.org/resource/mb6a-xn89.json")
df.head(5)<jupyter_output><empty_output><jupyter_text>Basic Analysis of Dataset pandas has several methods that allow you to quickly analyze a dataset and get an idea of the type and amount of data you are dealing with along with some important statistics. - .shape - returns the row and column count of a dataset- .describe() - returns statistics about the numerical columns in a dataset - .dtypes returns the data type of each column<jupyter_code>df.shape
df.describe()<jupyter_output><empty_output><jupyter_text>You can also run the .describe method with the "include='all'" flag to get statistics on the non-numeric column types. In this example we have to drop the "location_1" column because the .describe method doesn't accept dictionary objects.<jupyter_code>df.drop("location_1", axis=1).describe()
df.dtypes<jupyter_output><empty_output><jupyter_text>Here are some additional methods that can give you statistics of a DataFrame or particular column in a DataFrame.- .mean(axis=0 [will give you the calculated value per column]) - returns the statistical mean - .median(axis=0 [will give you the calculated value per column]) - returns the statistical median - .mode(axis=0 [will give you the calculated value per column]) - returns the statistical mode- .count() - gives number of total values in column- .unique() - returns array of all unique values in that column- .value_counts() - returns object containing counts of unique values<jupyter_code>df.bachelor_s_degree_or_higher.mean()
df.geography.count()
df.geography_type.unique()
df.less_than_high_school_graduate.value_counts()<jupyter_output><empty_output><jupyter_text>Mapping Functions to Transform Data Often times we need to apply a function to a column in a dataset to transform it. pandas makes it easy to do with the .apply() method. In this example, we will map the values in the "geography_type" column to either a "1" or "0" depending on the value. We will append this information to the DataFrame in a new column.<jupyter_code>def mapGeography(x):
    if x == "City":
        return 1
    else:
        return 0
df['geography_mapped_value'] = df.geography_type.apply(mapGeography)
df.geography_mapped_value.value_counts()<jupyter_output><empty_output><jupyter_text>We could have also accomplished the same thing in a lambda function in the following way<jupyter_code>df['geography_mapped_value_lambda'] = df.geography_type.apply(lambda y: 1 if y == "City" else 0)
df.geography_mapped_value_lambda.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07380151387720774, 'fraction_numerical': 0.006518082422203532, 'mean_word_length': 5.202086049543676, 'pattern_counts': {'":': 1, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '377232', 'license': 'MIT', 'n_tokens_mistral': 1378, 'n_tokens_neox': 1331, 'n_words': 574, 'path': 'notebooks/pandas-and-jupyter-notebook2.ipynb', 'repo_name': 'bozhnyukAlex/notebooks', 'size': '4756'}