<jupyter_start><jupyter_text>**Why pandas** - Flexibility of python. - Easier to work with big amount of data(where excels slows down and fails).   **Loading data****Data file used for the below exercise is - 'pokemon_data.csv'**You can find it in the /data_files folder at root.<jupyter_code>import pandas as pd
# read_csv
# index_col -> to change make a col as an index
# parse_dates -> parse the index

pd.read_csv('filename', index_col='Dates', parse_dates=True)
# upload file functionality through google colab libs

# from google.colab import files
# import io

# uploaded = files.upload()

# read csv files and storing file data in a data frame
# poke_df = pd.read_csv(io.BytesIO(uploaded['pokemon_data.csv']))

# read xls, xlsx files
# poke_df_xl = pd.read_excel(io.BytesIO(uploaded['pokemon_data.xlsx']))

# read txt files - need to pass delimiter by which values are separated in the read_csv func 
# poke_df_txt = pd.read_csv(io.BytesIO(uploaded['pokemon_data.txt']), delimiter='\t')<jupyter_output><empty_output><jupyter_text>**Checking the data**<jupyter_code>poke_df.head() # get starting 5 rows.
poke_df.head(7) # get 7 rows from start
poke_df.tail() # get 5 rows from end
poke_df.tail(10) # get 10 rows from end
# headers
headers = poke_df.columns

# reading specific columns

cols = poke_df['Name']  # reading single col
cols = poke_df[['Name', 'Attack', 'Defense']]   # multiple cols at once
cols = poke_df[['Name', 'Attack']][:]  # top 5 rows

cols
#  reading rows

 #through head or tail
poke_df.head(5)

# through iloc integer location
poke_df.iloc[5:10]

# read a specific loaction (R,C)
poke_df.iloc[0,1]

# loc - used to find data based on some key basically not through index
# poke_df.iloc[0,'Type 1']

# way to iterate
for index, row in poke_df.iterrows():
  print(index, row['Name'])
# describe data - gives out all the details about the data
poke_df.describe()<jupyter_output><empty_output><jupyter_text>**Sorting**<jupyter_code># ascending sort by col named 'Name' and Type 1
poke_df.sort_values('Name')
poke_df.sort_values('Type 1')

# to sort by descending order, we need to pass ascending False
poke_df.sort_values('Name', ascending=False)

# sorting on multiple columns and managing order

poke_df.sort_values(['Name', 'Type 1'], ascending=[1, 0]) # sorts data by ascending names and descending Type 1<jupyter_output><empty_output><jupyter_text>**Making Changes to the Data**<jupyter_code>cols = list(poke_df.columns.values)[4:10]

# adding new col
poke_df['Total'] = poke_df[cols].sum(axis=1)

poke_df['random col'] = 0

# deleting a column
poke_df = poke_df.drop(columns=['random col'])

# reordering columns
all_cols = list(poke_df.columns)
poke_df = poke_df[all_cols[0:4] + [all_cols[-1]] + all_cols[4:10]]<jupyter_output><empty_output><jupyter_text>**Saving data into csv, excel or txt file**<jupyter_code># saving and exporting the data to the csv file
poke_df.to_csv('modified_data.csv') # saves the index as well in the file

# if we dont wanna save the index
poke_df.to_csv('file_name.csv', index=False)

# saving data in excel
poke_df.to_excel('file_name.xlsx', index=False)

# saving data in txt
poke_df.to_csv('file_name.txt', index=False, sep='\t')<jupyter_output><empty_output><jupyter_text>**Filtering Data**<jupyter_code>#  through loc
filtered_poke_df = poke_df.loc[(poke_df['Type 1'] == 'Grass') & (poke_df['Type 2'] == 'Poison') & (poke_df['HP'] > 70)]

# resetting index on the filtered data

# below will store the old index in a col name index
filtered_poke_df_ind = filtered_poke_df.reset_index()

# to completely reset the index without storing it
filtered_poke_df = filtered_poke_df.reset_index(drop=True)

# resetting, droping index inplace
filtered_poke_df.reset_index(drop=True, inplace=True)
filtered_poke_df
# filtering based on textual patterns (Regex Filtering)

# filtering out all the pokemons which have Mega in their names
# ~ is used for negation

poke_df.loc[~poke_df['Name'].str.contains('Mega')]

# filtering all the rows which have type fire or grass
import re

# in flags re.I stands for ignoring case

poke_df.loc[(poke_df['Type 1'].str.contains('fire|grass', flags=re.I, regex=True))]
# filtering all the names those start with pi

pi_poke_df = poke_df.loc[poke_df['Name'].str.contains('^pi[a-z]*', flags=re.I, regex=True)]
pi_poke_df.sort_values('Name', ascending=False)<jupyter_output><empty_output><jupyter_text>**Conditional Changes**<jupyter_code># changing Type 1 Values which are Fire to Flamer

poke_df.loc[poke_df['Type 1'] == 'Fire', 'Type 1'] = 'Flamer'
# Making all pokemons Legendary whose Type 1 is Rock

poke_df.loc[poke_df['Type 1'] == 'Rock', 'Legendary'] = True
poke_df<jupyter_output><empty_output><jupyter_text>**Aggregate Statistics Using Group By**<jupyter_code># groups by Type 1 and calculates mean value of each col for every type
poke_df.groupby(['Type 1']).mean().sort_values('Attack', ascending=False)
# count

# setting count col to 1 for each row
poke_df['count'] = 1

count_data = poke_df.groupby(['Type 1', 'Type 2']).count()['count']<jupyter_output><empty_output><jupyter_text>**Working with large amount of data**<jupyter_code># read data from file size in chunks

# chunk size -> no of rows
for df in pd.read_csv('filename.csv', chunksize=10):
  print('DF CHUNK', df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1256120527306968, 'fraction_numerical': 0.009792843691148776, 'mean_word_length': 5.924380704041721, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '295246', 'license': 'MIT', 'n_tokens_mistral': 1966, 'n_tokens_neox': 1887, 'n_words': 536, 'path': 'pandas/pandas_basics.ipynb', 'repo_name': 'harshbhandari7/Data-Science', 'size': '5310'}