<jupyter_start><jupyter_text>Pandas<jupyter_code>import pandas as pd
pd.__version__
help(pd)<jupyter_output>Help on package pandas:

NAME
    pandas

DESCRIPTION
    pandas - a powerful data analysis and manipulation library for Python
    =====================================================================
    
    **pandas** is a Python package providing fast, flexible, and expressive data
    structures designed to make working with "relational" or "labeled" data both
    easy and intuitive. It aims to be the fundamental high-level building block for
    doing practical, **real world** data analysis in Python. Additionally, it has
    the broader goal of becoming **the most powerful and flexible open source data
    analysis / manipulation tool available in any language**. It is already well on
    its way toward this goal.
    
    Main Features
    -------------
    Here are just a few of the things that pandas does well:
    
      - Easy handling of missing data in floating point as well as non-floating
        point data.
      - Size mutability: columns can be inserted and deleted [...]<jupyter_text>Create dataframe<jupyter_code>df = pd.DataFrame({"Name": ["Braund, Mr. <NAME>",
                   "Allen, Mr. <NAME>",
                   "<NAME>"],
                   "Age": [22, 35, 58],
                   "Sex": ["male", "male", "female"]})
print(df)<jupyter_output>Name  Age     Sex
0   Braund, Mr. <NAME>   22    male
1  Allen, Mr. <NAME>   35    male
2  <NAME>   58  female<jupyter_text>Accessing columns<jupyter_code>print(df["Age"])
ages = pd.Series([22, 35, 58], name="Age")
print(ages)<jupyter_output>0    22
1    35
2    58
Name: Age, dtype: int64<jupyter_text>Describe<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>Max<jupyter_code>df["Age"].max()<jupyter_output><empty_output><jupyter_text>How to read CSV file<jupyter_code>titanic = pd.read_csv("titanic.csv")
titanic.head(5)<jupyter_output><empty_output><jupyter_text>Dtypes<jupyter_code>titanic.dtypes<jupyter_output><empty_output><jupyter_text>info<jupyter_code>titanic.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB<jupyter_text>filter specific rows from a DataFrame<jupyter_code>above_35 = titanic[titanic["Age"] > 35]
above_35
titanic["Age"] > 35<jupyter_output><empty_output><jupyter_text>isin<jupyter_code>class_23 = titanic[titanic["Pclass"].isin([2, 3])]
# OR
#class_23 = titanic[(titanic["Pclass"] == 2) | (titanic["Pclass"] == 3)]
class_23.head()<jupyter_output><empty_output><jupyter_text>notna<jupyter_code>age_no_na = titanic[titanic["Age"].notna() & titanic["Cabin"].notna()]
age_no_na<jupyter_output><empty_output><jupyter_text>loc<jupyter_code>adult_names = titanic.loc[titanic["Age"] > 35, "Name"]
adult_names<jupyter_output><empty_output><jupyter_text>iloc<jupyter_code>titanic.iloc[9:25, 2:5]
air_quality = pd.read_csv("air_quality_no2.csv", index_col=0, parse_dates=True) 
#The usage of the index_col and parse_dates parameters of the read_csv function to define the 
#first (0th) column as index of the resulting DataFrame and convert the dates in the column to Timestamp 
#objects, respectively.
air_quality
air_quality.head(50).plot()
import matplotlib.pyplot as plt
fig, axes = plt.subplots(nrows=4, ncols=2)
air_quality["station_paris"].plot.area(ax=axes[0,0],figsize=(36, 44))
air_quality["station_paris"].plot.bar(ax=axes[1,1])
air_quality["station_paris"].plot.barh(ax=axes[2,0])
air_quality["station_paris"].plot.box(ax=axes[3,1])<jupyter_output><empty_output><jupyter_text>How to create new columns derived from existing columns?<jupyter_code>air_quality["london_mg_per_cubic"] = air_quality["station_london"] * 1.882
air_quality.head(5)<jupyter_output><empty_output><jupyter_text>Rename the columns<jupyter_code>air_quality_renamed = air_quality.rename(
        columns={"station_antwerp": "BETR801",
                 "station_paris": "FR04014",
                 "station_london": "London Westminster"})
air_quality_renamed.head()
air_quality_renamed = air_quality_renamed.rename(columns=str.lower)
air_quality_renamed.head()<jupyter_output><empty_output><jupyter_text>How to calculate summary statistics?  Mean<jupyter_code>titanic["Age"].mean()
titanic[["Age", "Fare"]].median()
titanic[["Age", "Fare"]].describe()
titanic.agg({'Age': ['min', 'max', 'median', 'skew'],
                 'Fare': ['min', 'max', 'median', 'mean']})
titanic[["Sex", "Age"]].groupby("Sex").mean()
titanic.groupby("Sex").mean()
titanic.groupby("Sex")["Age"].mean()<jupyter_output><empty_output><jupyter_text>Count number of records by category<jupyter_code>titanic["Pclass"].value_counts()
titanic.groupby("Pclass")["Pclass"].count()<jupyter_output><empty_output><jupyter_text>Sort data according to particular column<jupyter_code>titanic.sort_values(by="Age").head()
titanic.sort_values(by=['Pclass', 'Age'], ascending=False).head()
# filter for no2 data only
air_quality_long = pd.read_csv("air_quality_long.csv")
air_quality_long[air_quality_long["parameter"] == "no2"].head()
air_quality.head()<jupyter_output><empty_output><jupyter_text>Combine dataset<jupyter_code>air_quality_no2 = pd.read_csv("air_quality_no2_long.csv",
                                  parse_dates=True)
air_quality_no2 = air_quality_no2[["date.utc", "location",
                                       "parameter", "value"]]
print(air_quality_no2.shape)
air_quality_no2.head()
air_quality_pm25 = pd.read_csv("air_quality_pm25_long.csv",
                                   parse_dates=True)
air_quality_pm25 = air_quality_pm25[["date.utc", "location",
                                         "parameter", "value"]]
print(air_quality_pm25.shape)
air_quality_pm25.head()
air_quality = pd.concat([air_quality_pm25, air_quality_no2], axis=0)
print(air_quality.shape)
air_quality.head()
air_quality = air_quality.sort_values("date.utc")
air_quality.head()
# Concate the particular key data

air_quality_ = pd.concat([air_quality_pm25, air_quality_no2],
                          keys=["PM25", "NO2"])
print(air_quality_.shape)
air_quality_.head()<jupyter_output>(3178, 4)<jupyter_text>Joins (Like in DBMS)<jupyter_code>stations_coord = pd.read_csv("air_quality_stations.csv")
air_quality = pd.merge(air_quality, stations_coord,
                           how='left', on='location')
air_quality.head()<jupyter_output><empty_output><jupyter_text>Pandas Attribute  Array<jupyter_code>pd.array(['a', 'b'], dtype=str)
pd.array(['2015', '2016'], dtype='datetime64[ns]')
pd.array(["1H", "2H"], dtype='timedelta64[ns]')
pd.array([pd.Period('2000', freq="D"), pd.Period("2000", freq="D")])
pd.array(['a', 'b', 'a'],
         dtype=pd.CategoricalDtype(['a', 'b', 'c'], ordered=True))<jupyter_output><empty_output><jupyter_text>Series<jupyter_code>import numpy as np
s = pd.Series([1, 3, 5, np.nan, 6, 8])
s<jupyter_output><empty_output><jupyter_text>Date_Range<jupyter_code>dates = pd.date_range('20130101', periods=6)

print(dates)
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list('ABCD'))

df<jupyter_output>DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',
               '2013-01-05', '2013-01-06'],
              dtype='datetime64[ns]', freq='D')<jupyter_text>Create Dataframe using series, timestamp,array and categorical<jupyter_code>df2 = pd.DataFrame({'A': 1.,
   ...:                     'B': pd.Timestamp('20130102'),
   ...:                     'C': pd.Series(1, index=list(range(4)), dtype='float32'),
   ...:                     'D': np.array([3] * 4, dtype='int32'),
   ...:                     'E': pd.Categorical(["test", "train", "test", "train"]),
   ...:                     'F': 'foo'})
print(df2)<jupyter_output>A          B    C  D      E    F
0  1.0 2013-01-02  1.0  3   test  foo
1  1.0 2013-01-02  1.0  3  train  foo
2  1.0 2013-01-02  1.0  3   test  foo
3  1.0 2013-01-02  1.0  3  train  foo<jupyter_text>Datatypes of col in dataframe<jupyter_code>df2.dtypes<jupyter_output><empty_output><jupyter_text>Tail<jupyter_code>df.tail(3)<jupyter_output><empty_output><jupyter_text>Index<jupyter_code>df.index<jupyter_output><empty_output><jupyter_text>Columns<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>Convert dataframe in numpy arrays<jupyter_code>df.to_numpy()
df2.to_numpy()<jupyter_output><empty_output><jupyter_text>Transpose the dataframe<jupyter_code>df.T<jupyter_output><empty_output><jupyter_text>Sorting<jupyter_code>df.sort_index(axis=1, ascending=False)
df.sort_values(by='B')<jupyter_output><empty_output><jupyter_text>Selection<jupyter_code>df['A']
df[0:3]
df['20130102':'20130104']<jupyter_output><empty_output><jupyter_text>Loc<jupyter_code>df.loc[dates[0]]
df.loc[:, ['A', 'B']]
df.loc['20130102':'20130104', ['A', 'B']]
df.loc['20130102', ['A', 'B']]
df.loc[dates[0], 'A']<jupyter_output><empty_output><jupyter_text>At:- For faster access of particular value<jupyter_code>df.at[dates[0], 'A']<jupyter_output><empty_output><jupyter_text>Setting Values by labels<jupyter_code>df.at[dates[0], 'A'] = 0
df<jupyter_output><empty_output><jupyter_text>iloc<jupyter_code>df.iloc[3]
df.iloc[3:5, 0:2]
df.iloc[[1, 2, 4], [0, 2]]
df.iloc[:, 1:3]
df.iloc[1, 1]<jupyter_output><empty_output><jupyter_text>iat<jupyter_code>df.iat[1, 1]<jupyter_output><empty_output><jupyter_text>Setting values by position<jupyter_code>df.iat[0, 1] = 0
df[df['A'] > 0]
df[df > 0]<jupyter_output><empty_output><jupyter_text>Create New Column<jupyter_code>df2 = df.copy()
df2['E'] = ['one', 'one', 'two', 'three', 'four', 'three']
df2<jupyter_output><empty_output><jupyter_text>isin<jupyter_code>df2[df2['E'].isin(['two', 'four'])]
s1 = pd.Series([1, 2, 3, 4, 5, 6], index=pd.date_range('20130102', periods=6))
s1
df2['F'] = s1
df['F'] = s1
df2<jupyter_output><empty_output><jupyter_text>Setting a value by assigning with a NumPy array:<jupyter_code>df.loc[:, 'D'] = np.array([5] * len(df))
df
df2['G'] = [1,np.nan,3,np.nan,5,np.nan]
df2<jupyter_output><empty_output><jupyter_text>Dropna<jupyter_code>df2.dropna(how='any')
df2.dropna(subset = ["G"], inplace=True)
df2<jupyter_output><empty_output><jupyter_text>fillna<jupyter_code>df2.fillna(value=5)<jupyter_output><empty_output><jupyter_text>isna<jupyter_code>pd.isna(df)<jupyter_output><empty_output><jupyter_text>Stats  Mean<jupyter_code>df.mean()<jupyter_output><empty_output><jupyter_text>Mean row wise<jupyter_code>df.mean(1)<jupyter_output><empty_output><jupyter_text>Subtraction<jupyter_code>s = pd.Series([1, 3, 5, np.nan, 6, 8], index=dates).shift(2)
s
df.sub(s, axis='index')
df
df.apply(np.cumsum)<jupyter_output><empty_output><jupyter_text>Grouping<jupyter_code>df = pd.DataFrame({'A': ['foo', 'bar', 'foo', 'bar',
   ....:                          'foo', 'bar', 'foo', 'foo'],
   ....:                    'B': ['one', 'one', 'two', 'three',
   ....:                          'two', 'two', 'one', 'three'],
   ....:                    'C': np.random.randn(8),
   ....:                    'D': np.random.randn(8)})
df
df.groupby('A').sum()
df.groupby(['A', 'B']).sum()
tuples = list(zip(*[['bar', 'bar', 'baz', 'baz',
   ....:                      'foo', 'foo', 'qux', 'qux'],
   ....:                     ['one', 'two', 'one', 'two',
   ....:                      'one', 'two', 'one', 'two']]))
tuples
index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])

stacked = df.stack()
stacked
stacked.unstack()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1700886990801577, 'fraction_numerical': 0.040325229960578184, 'mean_word_length': 4.5249546279491835, 'pattern_counts': {'":': 6, '<': 193, '<?xml version=': 0, '>': 198, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '48', 'dir': 'jupyter-structured-clean-dedup', 'id': '311633', 'license': 'Apache-2.0', 'n_tokens_mistral': 4945, 'n_tokens_neox': 4590, 'n_words': 768, 'path': 'Pandas/Pandas Module.ipynb', 'repo_name': 'himanshu-1205/Statistics', 'size': '12236'}