<jupyter_start><jupyter_text>Data Analysis with Python and Pandas Tutorial Part 1: Loading and Exploring data  Tutorial ObjectivesIn this tutorial, you will learn:* Create Pandas dataframes directly from JSON data* Create dataframes by loading data from CSV, Excel, and over the internet* Explore data using shape, columns, info and describe* Explore data by plotting plots  What is a DataFrame / Series?<jupyter_code>### Series and Dataframes in Pandas
from IPython.display import Image
from IPython.core.display import HTML
Image(url='https://storage.googleapis.com/lds-media/images/series-and-dataframe.width-1200.png', width=600)<jupyter_output><empty_output><jupyter_text>Common libraries<jupyter_code># import the Pandas library
import pandas as pd<jupyter_output><empty_output><jupyter_text>Creating a DateFrame (from JSON)<jupyter_code># create some data in JSON format
weather_data = { 
  "City": ["Cantho", "Danang", "Haiphong", "Hanoi", "Ho Chi Minh City"],
  "Temperature": [83, 64, 72, 81, 70],
  "Humidity": [86, 65, 90, 75, 96]
}
# create a Pandas dataframe from the JSON data
df1 = pd.DataFrame(weather_data)
# output the dataframe
df1
# output the shape of the dataframe
df1.shape
# output the list of columns (Series)
df1.columns
# output detailed dataframe info
df1.info()
# describe the values in the dataframe
df1.describe()
# set the index to City
# output it again to see the difference (look at the City column)
df1.set_index(keys='City', inplace=True)
df1
# plot the dataframe, as a 'bar' or 'barh'
df1.plot(kind='bar', figsize=(6, 3), title='Vietnam Major Cities Weather')
# save this dataset to disk in CSV format
df1.to_csv('simple_weather.csv')<jupyter_output><empty_output><jupyter_text>Reading Excel data into a DataFrame<jupyter_code># read an Excel file into a DataFrame (7dtd-weapons.xlsx, sheet name = "All")
# this should be available at: https://1drv.ms/x/s!AgtH78k0_cuvgkgBijDifi1YkIck
df2 = pd.read_excel('7dtd-weapons.xlsx', sheet_name='All')
# output the dataframe, but just a few (5) rows
df2.head()
# output the shape of the dataframe
df2.shape
# output dataframe info
df2.info()
# potential issue: "type" is string - probably supposed to be categorical
# convert column "Category" to a categorical series (an enumeration)
# output info again to see the difference
df2.Type = df2.Type.astype('category')
# potential issue: columns have spaces in them
# rename columns "Magazine Size" to "MagSize"
# output info again to see the difference
df2.rename(columns={'Magazine Size': 'MagSize'}, inplace=True)
# change the index to column 'Firearm', inplace
# output the head again to see the difference
#df2.reset_index(inplace=True)  # reset_index restores the original numeric index
df2.set_index(keys='Firearm', inplace=True)
# plot the head of the dataframe, with kind "barh"
df2.head().plot(kind='barh', figsize=(12, 4), title='Firearms Overview')
# reset the index again
df2.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>Reading CSV data (from a URL) into a DataFrame<jupyter_code>url = 'https://raw.github.com/pydata/pandas/master/pandas/tests/data/tips.csv'
# read a CSV file into DataFrame df3
df3 = pd.read_csv(url)
# output dataframe info
df3.info()
# output a few rows (10)
df3.head()
# take columns total_bill and tip and plot boxplot
#df3.plot(kind='box', figsize=(5, 10), subplots=False, by='sex')
df3.boxplot(column=['total_bill', 'tip'], by='sex', figsize=(8, 8))<jupyter_output><empty_output><jupyter_text>Reading CSV data with a time series<jupyter_code>oil_url = 'https://datahub.io/core/gold-prices/r/monthly.csv'
df4 = pd.read_csv(oil_url)
df4.shape
df4.head()
df4.Date = df4.Date.astype('datetime64')
#df4.reset_index(inplace=True)  # reset the index back to the original numeric index
df4.set_index('Date', inplace=True)
# output dataframe info, to verify the index is a Datetime index
df4.info()
# plot the dataset
df4.plot(figsize=(12,4), title='Gold Prices')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11474584801207852, 'fraction_numerical': 0.022395571212883745, 'mean_word_length': 5.985940246045694, 'pattern_counts': {'":': 3, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '485764', 'license': 'Apache-2.0', 'n_tokens_mistral': 1404, 'n_tokens_neox': 1324, 'n_words': 427, 'path': '01 Loading and Exploring data.ipynb', 'repo_name': 'rbscholtus/pandas_notebooks', 'size': '3974'}