<jupyter_start><jupyter_text>PandasPandas is a opensource python library prividing high performance data manuplation and analysis tools using it's powerful data structures.<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>SeriesSeries is a single dimensional data structure with homogenous data type values.Also Series comes with index or are one single column where as list is just a list<jupyter_code>series = pd.Series([10,20,40,50], index = ["a" , "b","c", "d"])
lists = [1,2,3]
print(lists)
series<jupyter_output>[1, 2, 3]<jupyter_text>DataframeTwo dimensional object with heterogenous data type value.<jupyter_code>listz = [['Ravi' , 10],['Ram' , 20],["Raj", 30]]
print(listz)

df = pd.DataFrame(listz, columns=['Name', "Age"], index=["a", "b", 'c'])
df<jupyter_output>[['Ravi', 10], ['Ram', 20], ['Raj', 30]]<jupyter_text>Accessing DataFrame<jupyter_code>dfa = df['Name']
dfb = df[['Name']]
print(type(dfa))
print(type(dfb))
dfa<jupyter_output><class 'pandas.core.series.Series'>
<class 'pandas.core.frame.DataFrame'><jupyter_text>dfa is a series because we are directly accessing column values, whereas dfb is a DataFrame because we are accessing a total column Dictionary to DataFrame<jupyter_code>dictz = {"Name": ["Rahul", 'Rio'], "Age" : [20,40]}
print(dictz)

dfd = pd.DataFrame(dictz)
dfd<jupyter_output>{'Name': ['Rahul', 'Rio'], 'Age': [20, 40]}<jupyter_text>Array to DataFrame<jupyter_code>arr = np.array([[1,2,3] , [5,6,7]])
print(arr)

dfa = pd.DataFrame(arr)
print(dfa)

## Now dataframe into array
arrd = np.array(dfa)
arrd<jupyter_output>[[1 2 3]
 [5 6 7]]
   0  1  2
0  1  2  3
1  5  6  7<jupyter_text>Range DataSyntax: pd.date_range(date, period , freq)date format: YYYYMMDDperiod: it is the no of days or months or yearsfrequency(freq): it is the format i.e Month(M), Year(Y), Day(D)<jupyter_code>from datetime import datetime

ranges = pd.date_range(start=datetime.now(), periods=10, freq='Y')
type(ranges)
ranges<jupyter_output><empty_output><jupyter_text>Head Tail Func- pd.head()  Returns the first 5 values and to change these 5 values we can pass n in pd.head(3) it will return first 3 values.- pd.tail()  Returns the last 5 values and to change these 5 values we can pass n in pd.tail(3) it will return last 3 values.<jupyter_code># Create Df
df = np.random.randn(10,4)
print(df)
df = pd.DataFrame(df , index = ranges, columns = ["Random 1", "Random 2", "Random 3", "Random 4"])
print(df)
print(df.shape)
print(df.columns)
print(df.info())
print(df.head(3))
print("#####")
print(df.tail(3))<jupyter_output>[[ 0.50527098  1.52173556 -1.20641212 -1.1624954 ]
 [-0.6008181  -0.21567247  0.26203883 -0.0129101 ]
 [ 1.38121482 -0.22590057 -1.16423593 -0.32765459]
 [-0.73293844  0.91636682 -1.74082324 -0.94196156]
 [ 1.10281754 -0.57831048 -1.7754391   0.06139702]
 [-1.00228848  1.57609777 -1.05636544  0.64415274]
 [-1.80564219 -1.36540907  0.43524884 -0.3908289 ]
 [ 2.88237691 -0.95764453 -0.42411494 -1.18595926]
 [ 1.54012224  0.74221973  0.37203816 -1.91378617]
 [-0.89201017 -0.71280133 -0.74980965 -1.483079  ]]
                            Random 1  Random 2  Random 3  Random 4
2021-12-31 12:38:34.419815  0.505271  1.521736 -1.206412 -1.162495
2022-12-31 12:38:34.419815 -0.600818 -0.215672  0.262039 -0.012910
2023-12-31 12:38:34.419815  1.381215 -0.225901 -1.164236 -0.327655
2024-12-31 12:38:34.419815 -0.732938  0.916367 -1.740823 -0.941962
2025-12-31 12:38:34.419815  1.102818 -0.578310 -1.775439  0.061397
2026-12-31 12:38:34.419815 -1.002288  1.576098 -1.056365  0.644153
2027-12-31 12:38:34.[...]<jupyter_text>Null values<jupyter_code>print(df.isnull()) # null values are there or not
print(df.isnull().sum()) # sum of null values in each column
df.dropna() # by default it'll delete the entire row there nan value is present but to delete the column we can use df.dropna(axis=1) by default axis is 0<jupyter_output>Random 1  Random 2  Random 3  Random 4
2021-12-31 12:38:34.419815     False     False     False     False
2022-12-31 12:38:34.419815     False     False     False     False
2023-12-31 12:38:34.419815     False     False     False     False
2024-12-31 12:38:34.419815     False     False     False     False
2025-12-31 12:38:34.419815     False     False     False     False
2026-12-31 12:38:34.419815     False     False     False     False
2027-12-31 12:38:34.419815     False     False     False     False
2028-12-31 12:38:34.419815     False     False     False     False
2029-12-31 12:38:34.419815     False     False     False     False
2030-12-31 12:38:34.419815     False     False     False     False
Random 1    0
Random 2    0
Random 3    0
Random 4    0
dtype: int64<jupyter_text>Values<jupyter_code>print(df.value_counts()) # it'll return the no. of occurence of the data

df.dtypes # >> to know each datatype

# Same for column
print(df['Random 1'].value_counts())
print(df.corr()) # to find the relation between adjusent column
df.describe() ## Gives the summary of the columnwise data<jupyter_output><empty_output><jupyter_text>Slicing<jupyter_code>df[0:2]<jupyter_output><empty_output><jupyter_text>loc vs iloc- loc is used to access data by putting the location for the data. i.e `df.loc["2021-12-31 01:49:18.893933", "Random 1"][0]`  - In the first block he have inserted the row value and in second block we have inserted the column value.- iloc is used to access data using the index. i.e `df.iloc[[0,2,5],0:2]`<jupyter_code>df.loc["2021-12-31", "Random 1"]
df.iloc[[0,2,5],-1]
print(df)
df.iloc[[2,4,6], 0:2]<jupyter_output>Random 1  Random 2  Random 3  Random 4
2021-12-31 12:38:34.419815  0.505271  1.521736 -1.206412 -1.162495
2022-12-31 12:38:34.419815 -0.600818 -0.215672  0.262039 -0.012910
2023-12-31 12:38:34.419815  1.381215 -0.225901 -1.164236 -0.327655
2024-12-31 12:38:34.419815 -0.732938  0.916367 -1.740823 -0.941962
2025-12-31 12:38:34.419815  1.102818 -0.578310 -1.775439  0.061397
2026-12-31 12:38:34.419815 -1.002288  1.576098 -1.056365  0.644153
2027-12-31 12:38:34.419815 -1.805642 -1.365409  0.435249 -0.390829
2028-12-31 12:38:34.419815  2.882377 -0.957645 -0.424115 -1.185959
2029-12-31 12:38:34.419815  1.540122  0.742220  0.372038 -1.913786
2030-12-31 12:38:34.419815 -0.892010 -0.712801 -0.749810 -1.483079<jupyter_text>Concatenation & Descriptive Statistics- `df.drop()`:- mainly used to delete columns<jupyter_code>drd = df.drop(columns=["Random 1", "Random 2"])<jupyter_output><empty_output><jupyter_text>- `df.concat()`:- It is used to merge to dataframes row wise<jupyter_code>a = {"Name": ["Rio", "Nirobi"], "Age" :[18,20]}
a = pd.DataFrame(a)

b = {"Name": ["Tokyo", "Berlin", "Tokyo"], "Age" :[23,30,23]}
b = pd.DataFrame(b, index= [2,3,4])

ab = pd.concat([a,b])
ab<jupyter_output><empty_output><jupyter_text>- Knowing the existance of value.<jupyter_code>print(ab["Name"] == "Tokyo", '\n\n') ## tells the existance

print(ab[ab["Name"] == "Tokyo"]) ## Shows the existance<jupyter_output>0    False
1    False
2     True
3    False
4     True
Name: Name, dtype: bool 


    Name  Age
2  Tokyo   23
4  Tokyo   23<jupyter_text>- Deleting the dublicate `df.drop_duplicates(column_name)`  - it'll delete the whole row.<jupyter_code>ab = ab.drop_duplicates(['Name'])<jupyter_output><empty_output><jupyter_text>- Sorting values `df.sort_values(column_name)` it can sort numerically as well alphabatically.<jupyter_code>ab_sort_aplha = ab.sort_values('Name')<jupyter_output><empty_output><jupyter_text>- renaming columns `df.rename(columns = {"Name": "Emp_name"})`<jupyter_code>ab_sort_aplha.rename(columns={"Name":"Emp_Name" , "Age": "Emp_Age"})<jupyter_output><empty_output><jupyter_text>- Other functions  - `df.sum()` to find the sum of values  - `df.mean()` to find the mean of values  - `df.median()` to find the median of the values  -`df.mode()` to find the mode of the values  - `df.std()` to find the standard deviation of the values  - `df.min()` to find the minimum value from the all values  - `df.max()` to find the maximum value from the all values - `pd.merge` Merges df column wise, here `on="Name"` means we are looking for Name in both the df.<jupyter_code>di1 = {"Name":["Rahul" , "Sham" , "Ram"], "Age": [12,23,45]}
df1 = pd.DataFrame(di1)

di2 = {"Course": ["Machine Learning", "Data Science" , 'MLops'], 'Name': ["Rahul" , "Sham" , "Ram"], "Year":[2020,2017,2015]}
df2 = pd.DataFrame(di2)

df = pd.merge(df1, df2 , on='Name', how="inner" )
df
## Doing changes in df
df.rename(columns={"Name": "Student_Name"})
df.iloc[0,0] = 'Virat'
df<jupyter_output><empty_output><jupyter_text>Data Cleaning<jupyter_code>dfs = pd.Series(['Lakshit', '18', 'www.lakshit.io'])
dfs.str.capitalize()

dfs.str.islower()
dfs.str.lower()

dfs.str.isupper()
dfs.str.upper()

dfs.str.isnumeric()

dfs.str.swapcase()

dfs.str.len()

dfs.str.cat(sep="_") # >> Lakshit_18_www.lakshit.io

dfs.str.replace("www." , "yoyo.")

dfs.str.repeat(2)

dfs.str.count("w") # occurence of a letter<jupyter_output><empty_output><jupyter_text>Applying funtions to tables- `df.pipe(func_name)` it'll apply to the whole table.- `df.apply(func_name)` it'll apply to each column.<jupyter_code>num = np.random.rand(10,4)
dfNum = pd.DataFrame(num)
print(dfNum,'\n\n')

def add(a,b):
  return a + b

dfNum.pipe(type) # >> pandas.core.frame.DataFrame

# PIPE
print(dfNum.pipe(add,10), '\n\n')

# Apply
dfNum.apply(np.mean)
import os
os.getcwd() # to get the current working directiory
# os.chdir("dir_name") # to change the current working directiory

df.dtypes<jupyter_output><empty_output><jupyter_text>Importing csv file<jupyter_code>df = pd.read_csv("sample_data/california_housing_test.csv" ,header=0, skiprows=[1,2,3,4,5],\
                 names = ['Long', 'Lat', "house_med_age", 'Rooms', 'bedrooms','population_','household', "med_inc","med_val"])<jupyter_output><empty_output><jupyter_text>- here header = 0 is by default you can change header toany other row by putting the row number.- here skiprows will skip the rows which are specified - here names is used to give custom header name- put `header = None` to remove header<jupyter_code>df
df = pd.read_csv("sample_data/california_housing_test.csv", header= None, prefix="Column: ", na_values=["na value"], verbose=True)<jupyter_output>Tokenization took: 3.73 ms
Type conversion took: 9.48 ms
Parser memory cleanup took: 0.01 ms<jupyter_text>- here `prefix = "column: "` is used to add any prefix to the column names i.e Column: 1- `na_values=['na_value']` is used to fill replace the na value with na_value  - We can also use `sep=";"` if the seperator is not ',' we have to specify the seperator- `dtype={"Column: 0": 'int'}` if you want to change the data type of a column then specify it here.- `verbose=True` is used to get the time it takes to load the file<jupyter_code>print(df)
df.dtypes<jupyter_output>Column: 0  Column: 1  ...      Column: 7           Column: 8
0       longitude   latitude  ...  median_income  median_house_value
1     -122.050000  37.370000  ...       6.608500       344700.000000
2     -118.300000  34.260000  ...       3.599000       176500.000000
3     -117.810000  33.780000  ...       5.793400       270500.000000
4     -118.360000  33.820000  ...       6.135900       330000.000000
...           ...        ...  ...            ...                 ...
2996  -119.860000  34.420000  ...       1.179000       225000.000000
2997  -118.140000  34.060000  ...       3.390600       237200.000000
2998  -119.700000  36.300000  ...       2.289500        62000.000000
2999  -117.120000  34.100000  ...       3.270800       162500.000000
3000  -119.630000  34.420000  ...       8.560800       500001.000000

[3001 rows x 9 columns]<jupyter_text>Reading CSV Data from URL<jupyter_code>dfs = pd.read_csv("http://winterolympicsmedals.com/medals.csv", nrows=10, usecols=[0,1,5] )
dfs.shape
dfs.to_csv("new_.medals.csv", index=None)<jupyter_output><empty_output><jupyter_text>- `skipfooter=11` will remove last 11 rows from table- `nrows=10` will only show the first 10 rows- `usecols=[0,1,5]` will only return the columns which are present at the given index- `dfs.to_csv("new_medals.csv", index=None)` to save the file into csv format- `pd.read_excel(filename)` to read the excel file (check documentation for more)  Data Visualiztion using Pandas<jupyter_code>data = np.random.rand(100,4)
data = pd.DataFrame(data, columns=['f', 's', 't', 'fr'])
data['f'].plot.box(vert=False)<jupyter_output><empty_output><jupyter_text>- green line means 50% data is near 0.4- two ends are min and max values in dataframe- `vert = False` means we don't want to show data vertically<jupyter_code>data['f'].plot()
data['f'].plot.bar()<jupyter_output><empty_output><jupyter_text>- `data['f'].plot.barh()` to show data vertically<jupyter_code>data['f'].plot.hist()<jupyter_output><empty_output><jupyter_text>- `data['f'].plot.hist(bins=10)` here bins refers to the no. of blocks<jupyter_code>data['f'].plot.area()
data.plot.scatter(x = 'f' ,y = 's')
data['f'].plot.pie()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14850723299476762, 'fraction_numerical': 0.14989227454601417, 'mean_word_length': 4.076953125, 'pattern_counts': {'":': 12, '<': 108, '<?xml version=': 0, '>': 114, 'https://': 0, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '235603', 'license': 'MIT', 'n_tokens_mistral': 6199, 'n_tokens_neox': 5072, 'n_words': 1267, 'path': 'Pandas/pandas.ipynb', 'repo_name': 'lakshit2808/Machine-Learning-Notes', 'size': '12996'}