<jupyter_start><jupyter_text>Pandas SeriesA Pandas Series is like a column in a table.It is a one-dimensional array holding data of any type.<jupyter_code>#Creating Pandas Series
a = [1, 7, 2]

myvar = pd.Series(a)

print(myvar)<jupyter_output>0    1
1    7
2    2
dtype: int64<jupyter_text>indexWith the index argument, you can name your own labels.<jupyter_code>#Creating a Pandas Index
a = [1, 7, 2]

myvar = pd.Series(a, index = ["x", "y", "z"])

print(myvar)
print(myvar["y"])
calories = {"day1": 420, "day2": 380, "day3": 390}

myvar = pd.Series(calories)

print(myvar)<jupyter_output>day1    420
day2    380
day3    390
dtype: int64<jupyter_text>DataFramesA Pandas DataFrame is a 2 dimensional data structure, like a 2 dimensional array, or a table with rows and columns.<jupyter_code>data = {
  "calories": [420, 380, 390],
  "duration": [50, 40, 45]
}

#load data into a DataFrame object:
df = pd.DataFrame(data)

print(df) 
#refer to the row index:
print(df.loc[0])
#use a list of indexes:
print(df.loc[[0, 1]])
# Add a list of names to give each row a name
data = {
  "calories": [420, 380, 390],
  "duration": [50, 40, 45]
}

df = pd.DataFrame(data, index = ["day1", "day2", "day3"])

print(df) 
#Use the named index in the loc attribute to return the specified row(s).
#refer to the named index:
print(df.loc["day2"])<jupyter_output>calories    380
duration     40
Name: day2, dtype: int64<jupyter_text>CSV FILES<jupyter_code>#Load a comma separated file (CSV file) into a DataFrame
df = pd.read_csv('data.csv')

print(df)
# By default, when you print a DataFrame, you will only get the first 5 rows, and the last 5 rows
df = pd.read_csv('data.csv')

print(df) 
# use to_string() to print the entire DataFrame.
df = pd.read_csv('data.csv')

print(df.to_string())<jupyter_output>Duration  Pulse  Maxpulse  Calories
0          60    110       130     409.1
1          60    117       145     479.0
2          60    103       135     340.0
3          45    109       175     282.4
4          45    117       148     406.0
5          60    102       127     300.0
6          60    110       136     374.0
7          45    104       134     253.3
8          30    109       133     195.1
9          60     98       124     269.0
10         60    103       147     329.3
11         60    100       120     250.7
12         60    106       128     345.3
13         60    104       132     379.3
14         60     98       123     275.0
15         60     98       120     215.2
16         60    100       120     300.0
17         45     90       112       NaN
18         60    103       123     323.0
19         45     97       125     243.0
20         60    108       131     364.2
21         45    100       119     282.0
22         60    130       101     300.0
23         45   [...]<jupyter_text>JSON Load the JSON file into a DataFrame<jupyter_code>df = pd.read_json('data.json')

print(df.to_string()) 
# Dictionary as JSON
import pandas as pd

data = {
  "Duration":{
    "0":60,
    "1":60,
    "2":60,
    "3":45,
    "4":45,
    "5":60
  },
  "Pulse":{
    "0":110,
    "1":117,
    "2":103,
    "3":109,
    "4":117,
    "5":102
  },
  "Maxpulse":{
    "0":130,
    "1":145,
    "2":135,
    "3":175,
    "4":148,
    "5":127
  },
  "Calories":{
    "0":409,
    "1":479,
    "2":340,
    "3":282,
    "4":406,
    "5":300
  }
}

df = pd.DataFrame(data)

print(df)<jupyter_output>Duration  Pulse  Maxpulse  Calories
0        60    110       130       409
1        60    117       145       479
2        60    103       135       340
3        45    109       175       282
4        45    117       148       406
5        60    102       127       300<jupyter_text>Viewing the Data<jupyter_code># The head() method returns the headers and a specified number of rows, starting from the top.
df = pd.read_csv('data.csv')
print(df.head(10))
# The tail() method returns the headers and a specified number of rows, starting from the bottom.
print(df.tail())<jupyter_output>Duration  Pulse  Maxpulse  Calories
164        60    105       140     290.8
165        60    110       145     300.0
166        60    115       145     310.2
167        75    120       150     320.4
168        75    125       150     330.4<jupyter_text>Info About the DataThe DataFrames object has a method called info(), that gives you more information about the data set<jupyter_code>print(df.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 169 entries, 0 to 168
Data columns (total 4 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Duration  169 non-null    int64  
 1   Pulse     169 non-null    int64  
 2   Maxpulse  169 non-null    int64  
 3   Calories  164 non-null    float64
dtypes: float64(1), int64(3)
memory usage: 5.4 KB
None<jupyter_text>Pandas - Cleaning Empty Cells<jupyter_code># the dropna() method returns a new DataFrame, and will not change the original.

df = pd.read_csv('data.csv')

new_df = df.dropna()

print(new_df.to_string())
# If you want to change the original DataFrame, use the inplace = True argument

df = pd.read_csv('data.csv')

df.dropna(inplace = True)

print(df.to_string())
# Replace Empty Values
# Replace NULL values with the number 130

df = pd.read_csv('data.csv')

df.fillna(130, inplace = True)<jupyter_output><empty_output><jupyter_text>Replace Using Mean, Median, or Mode<jupyter_code># Mean = the average value (the sum of all values divided by number of values)
df = pd.read_csv('data.csv')

x = df["Calories"].mean()

df["Calories"].fillna(x, inplace = True)
# Median = the value in the middle, after you have sorted all values ascending.
df = pd.read_csv('data.csv')

x = df["Calories"].median()

df["Calories"].fillna(x, inplace = True)
# Mode = the value that appears most frequently

df = pd.read_csv('data.csv')

x = df["Calories"].mode()[0]

df["Calories"].fillna(x, inplace = True)<jupyter_output><empty_output><jupyter_text>Cleaning Data of Wrong Format<jupyter_code># to_datetime()

df = pd.read_csv('dirtydata.csv')

df['Date'] = pd.to_datetime(df['Date'])

print(df.to_string())
# The result from the converting in the example above gave us a NaT value, which can be handled as a NULL value, and we can remove the row by using the dropna() method

df.dropna(subset=['Date'], inplace = True)
print(df.to_string())<jupyter_output>Duration       Date  Pulse  Maxpulse  Calories
0         60 2020-12-01    110       130     409.1
1         60 2020-12-02    117       145     479.0
2         60 2020-12-03    103       135     340.0
3         45 2020-12-04    109       175     282.4
4         45 2020-12-05    117       148     406.0
5         60 2020-12-06    102       127     300.0
6         60 2020-12-07    110       136     374.0
7        450 2020-12-08    104       134     253.3
8         30 2020-12-09    109       133     195.1
9         60 2020-12-10     98       124     269.0
10        60 2020-12-11    103       147     329.3
11        60 2020-12-12    100       120     250.7
12        60 2020-12-12    100       120     250.7
13        60 2020-12-13    106       128     345.3
14        60 2020-12-14    104       132     379.3
15        60 2020-12-15     98       123     275.0
16        60 2020-12-16     98       120     215.2
17        60 2020-12-17    100       120     300.0
18        45 2020-12-18     90 [...]<jupyter_text>Fixing Wrong Data<jupyter_code># Replacing Values
df.loc[7, 'Duration'] = 45
print(df.to_string())
"""Loop through all values in the "Duration" column. If the value is higher than 120, set it to 120:"""

for x in df.index:
  if df.loc[x, "Duration"] > 120:
    df.loc[x, "Duration"] = 120
""" Delete rows where "Duration" is higher than 120 """
for x in df.index:
  if df.loc[x, "Duration"] > 120:
    df.drop(x, inplace = True)<jupyter_output><empty_output><jupyter_text>Discovering Duplicates<jupyter_code>""" The duplicated() method returns a Boolean values for each row """
print(df.duplicated())
""" To remove duplicates, use the drop_duplicates() method. """
df.drop_duplicates(inplace = True)<jupyter_output><empty_output><jupyter_text>Pandas - Data Correlations<jupyter_code>"""Show the relationship between the columns"""
df.corr()<jupyter_output><empty_output><jupyter_text>Matplotlib<jupyter_code>df = pd.read_csv('data.csv')

df.plot()

plt.show()
# Scatter Plot
df = pd.read_csv('data.csv')

df.plot(kind = 'scatter', x = 'Duration', y = 'Calories')

plt.show()
df = pd.read_csv('data.csv')

df.plot(kind = 'scatter', x = 'Duration', y = 'Maxpulse')

plt.show()
# Histogram Plot
df["Duration"].plot(kind = 'hist')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1146570732273413, 'fraction_numerical': 0.1277706858535453, 'mean_word_length': 2.2422874341610233, 'pattern_counts': {'":': 35, '<': 50, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '221449', 'license': 'MIT', 'n_tokens_mistral': 3709, 'n_tokens_neox': 3007, 'n_words': 948, 'path': 'pandas_tutorial.ipynb', 'repo_name': 'Makhosandile/pandas_tutorial', 'size': '8617'}