<jupyter_start><jupyter_text>Pandas Tips & Tricks & More Hello Kaggler! Objective of this kernal is to demonstrate most commonly used Pandas Tips & Tricks and More .  ContentsNote : Please use below links to navigate the note book1. [Check Package Version](CheckPackageVersion)1. [Ignore Warnings](IgnoreWarnings)1. Pandas Basics    1. [Data Read and Peak](ReadandPeak)    1. [Shape, Columns](ShapeColumns)    1. [pandas series to pandas dataframe](series2df)1. [Query Data Type](QueryDataType)1. [Columns With Missing Values as a List](ColumnsWithMissingValues)1. [Columns of object Data Type (Categorical Columns) as a List](CatColumns)1. [Columns that Contain Numeric Values as a List](NumColumns)1. [Categorical Columns with Cardinality less than N](CatColsCar)1. [Count of Unique Values in a Column](UniqueCount)1. [Select Columns Based on Data Types](DTypeColSelect)1. [Check whether data is ready for training](checkDataBeforeTraining)1. [Subplotting in Notebooks](SublottinginNotebooks)1. Tools to Deal with Missing Values    1. [Get Missing Values Info](GetMissingValuesInfo)    1. [Fill missing values](FillMissingValues)    1. [Drop columns where data is missing more than x%](dropDataMissingColumns)    1. [Columns With Missing Values as a List](ColumnsWithMissingValues)1. Feature Engineering    1. [Drop rows where target is missing](dropTargetMissingRows)    1. [OneHot Encode the Dataframe](OneHotEncode)    1. [Convert categorical columns in numerical dtype to object type](Convertnumericalcategoricalobject)    1. [Select Columns Based on Data Types](DTypeColSelect)    1. [Get feature columns df and target column from training data](getTrainX_TrainY)1. Modelling    1. [Logistic Regression](LogisticRegression)<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import os
from pprint import pprint
print(os.listdir("../input"))<jupyter_output>['titanic']<jupyter_text>Check Package Version[^](CheckPackageVersion)<jupyter_code>print(pd.__version__)
print(np.__version__)<jupyter_output>0.25.0
1.17.0<jupyter_text>Ignore Warnings[^](IgnoreWarnings)<jupyter_code>import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Data Read and Peak [^](ReadandPeak)<jupyter_code># Read & peek top
data = pd.read_csv("../input/titanic/train.csv")
data.head()
# Reading with Index column & peek tail
data2 = pd.read_csv("../input/titanic/train.csv",index_col='PassengerId')
data2.tail()<jupyter_output><empty_output><jupyter_text>Shape, Columns[^](ShapeColumns)<jupyter_code># Shape, Row Count, Column Count & Column Names
print('Shape of dataframe \t:', data.shape)
print('# of Rows \t\t:', data.shape[0])
print('# of Columns \t\t:', data.shape[1])
print('Columns in dataframe \t:', data.columns)<jupyter_output>Shape of dataframe 	: (891, 12)
# of Rows 		: 891
# of Columns 		: 12
Columns in dataframe 	: Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
       'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
      dtype='object')<jupyter_text>Query Data Type[^](QueryDataType)<jupyter_code>values = {}
arr = []
print('values is a ' ,type(values))
type(arr)<jupyter_output>values is a  <class 'dict'><jupyter_text>Columns With Missing Values as a List[^](ColumnsWithMissingValues)<jupyter_code>def getColumnsWithMissingValuesList(df):
    return [col for col in df.columns if df[col].isnull().any()] 

getColumnsWithMissingValuesList(data)<jupyter_output><empty_output><jupyter_text>Columns of object Data Type (Categorical Columns) as a List[^](CatColumns)<jupyter_code>def getObjectColumnsList(df):
    return [cname for cname in df.columns if df[cname].dtype == "object"]

cat_cols = getObjectColumnsList(data)
cat_cols<jupyter_output><empty_output><jupyter_text>Columns that Contain Numeric Values as a List[^](NumColumns)<jupyter_code>def getNumericColumnsList(df):
    return [cname for cname in df.columns if df[cname].dtype in ['int64', 'float64']]

num_cols = getNumericColumnsList(data)
num_cols<jupyter_output><empty_output><jupyter_text>Categorical Columns with Cardinality less than N[^](CatColsCar)<jupyter_code>def getLowCardinalityColumnsList(df,cardinality):
    return [cname for cname in df.columns if df[cname].nunique() < cardinality and df[cname].dtype == "object"]

LowCardinalityColumns = getLowCardinalityColumnsList(data,10)
LowCardinalityColumns<jupyter_output><empty_output><jupyter_text>Count of Unique Values in a Column[^](UniqueCount)<jupyter_code>data['Embarked'].nunique()<jupyter_output><empty_output><jupyter_text>OneHot Encode the Dataframe[^](OneHotEncode)<jupyter_code>def PerformOneHotEncoding(df,columnsToEncode):
    return pd.get_dummies(df,columns = columnsToEncode)

oneHotEncoded_df = PerformOneHotEncoding(data,getLowCardinalityColumnsList(data,10))
oneHotEncoded_df.head()<jupyter_output><empty_output><jupyter_text>Select Columns Based on Data Types[^](DTypeColSelect)<jupyter_code># select only int64 & float64 columns
numeric_data = data.select_dtypes(include=['int64','float64'])

# select only object columns
categorical_data = data.select_dtypes(include='object')
numeric_data.head()
categorical_data.head()<jupyter_output><empty_output><jupyter_text>Get Missing Values Info[^](GetMissingValuesInfo)<jupyter_code>def missingValuesInfo(df):
    total = df.isnull().sum().sort_values(ascending = False)
    percent = round(df.isnull().sum().sort_values(ascending = False)/len(df)*100, 2)
    temp = pd.concat([total, percent], axis = 1,keys= ['Total', 'Percentage'])
    return temp.loc[(temp['Total'] > 0)]

missingValuesInfo(data)<jupyter_output><empty_output><jupyter_text>Fill missing values[^](FillMissingValues)<jupyter_code># for Object columns fill using 'UNKOWN'
# for Numeric columns fill using median
def fillMissingValues(df):
    num_cols = [cname for cname in df.columns if df[cname].dtype in ['int64', 'float64']]
    cat_cols = [cname for cname in df.columns if df[cname].dtype == "object"]
    values = {}
    for a in cat_cols:
        values[a] = 'UNKOWN'

    for a in num_cols:
        values[a] = df[a].median()
        
    df.fillna(value=values,inplace=True)
    
    
HandleMissingValues(data)
data.head()
#check for NaN values
data.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>Drop columns where data is missing more than x%[^](dropDataMissingColumns)<jupyter_code># pass the DataFrame and percentage
def dropDataMissingColumns(df,percentage):
    print("Dropping columns where more than {}% values are Missing..".format(percentage))
    nan_percentage = df.isnull().sum().sort_values(ascending=False) / df.shape[0]
    missing_val = nan_percentage[nan_percentage > 0]
    to_drop = missing_val[missing_val > percentage/100].index.values
    df.drop(to_drop, axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Drop rows where target is missing[^](dropTargetMissingRows)<jupyter_code>def dropTargetMissingRows(df,target):
    print("Dropping Rows where Target is Missing..")
    df.dropna(axis=0, subset=[target], inplace=True)<jupyter_output><empty_output><jupyter_text>Logistic Regression[^](LogisticRegression)<jupyter_code>def logistic(X,y):
    X_train,X_test,y_train,y_test=train_test_split(X,y,random_state=42,test_size=0.2)
    lr=LogisticRegression()
    lr.fit(X_train,y_train)
    y_pre=lr.predict(X_test)
    print('Accuracy : ',accuracy_score(y_test,y_pre))<jupyter_output><empty_output><jupyter_text>pandas series to pandas dataframe[^](series2df)<jupyter_code>series = data['Fare']
d = {series.name : series}
df = pd.DataFrame(d) 
df.head()<jupyter_output><empty_output><jupyter_text>Convert categorical columns in numerical dtype to object type[^](Convertnumericalcategoricalobject)Sometimes categorical columns comes in numerical data types. This is the case for all most all ordinal columns. If not converted to 'category' descriptive statistic summary does not makes sense.<jupyter_code>PassengerClass = data['Pclass'].astype('category')
PassengerClass.describe()<jupyter_output><empty_output><jupyter_text>Check whether data is ready for training[^](checkDataBeforeTraining)<jupyter_code># checks whether df contatins null values or object columns
def checkDataBeforeTraining(df):
    if(df.isnull().sum().sum() != 0):
        print("Error : Null Values Exist in Data")
        return False;
    
    if(len([cname for cname in df.columns if df[cname].dtype == "object"])>0):
        print("Error : Object Columns Exist in Data")
        return False;
    
    print("Data is Ready for Training")
    return True;<jupyter_output><empty_output><jupyter_text>Get feature columns df and target column from training data[^](getTrainX_TrainY)<jupyter_code>def getTrainX_TrainY(train_df,target):
    trainY = train_df.loc[:,target]
    trainX = train_df.drop(target, axis=1)
    return trainX,trainY<jupyter_output><empty_output><jupyter_text>Subplotting in Notebooks[^](SublottinginNotebooks)<jupyter_code>#impoting required libraries for demo
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=500, n_features=4, n_informative=2,random_state=0, shuffle=False)


# Subploting lets 2*2 figure with sizes (14*14)
f,ax=plt.subplots(2,2,figsize=(14,14))

#first plot
sns.scatterplot(x=X[:,0], y=y, ax=ax[0,0])
ax[0,0].set_xlabel('Feature 1 Values')
ax[0,0].set_ylabel('Y Values')
ax[0,0].set_title('Sactter Plot : Feature 1 vs Y')

#second plot
sns.scatterplot(x=X[:,1], y=y,ax=ax[0,1])
ax[0,1].set_xlabel('Feature 2 Values')
ax[0,1].set_ylabel('Y Values')
ax[0,1].set_title('Sactter Plot : Feature 2 vs Y')

#Third plot
sns.scatterplot(x=X[:,2], y=y,ax=ax[1,0])
ax[1,0].set_xlabel('Feature 3 Values')
ax[1,0].set_ylabel('Y Values')
ax[1,0].set_title('Sactter Plot : Feature 3 vs Y')

#Fourth plot
sns.scatterplot(x=X[:,3], y=y,ax=ax[1,1])
ax[1,1].set_xlabel('Feature 4 Values')
ax[1,1].set_ylabel('Y Values')
ax[1,1].set_title('Sactter Plot : Feature 4 vs Y')

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12812344449975113, 'fraction_numerical': 0.015331010452961672, 'mean_word_length': 6.757528957528957, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 94, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '22560', 'license': 'MIT', 'n_tokens_mistral': 3556, 'n_tokens_neox': 3389, 'n_words': 770, 'path': 'titanic/pandas-tips-tricks-and-more.ipynb', 'repo_name': 'MLVPRASAD/KaggleProjects', 'size': '10045'}