<jupyter_start><jupyter_text>Basic OperationsFirst few lines of code<jupyter_code>print(601+49)
print(19*17)
print("Python code")

x=7
print('x:',x)<jupyter_output>650
323
Python code
('x:', 7)<jupyter_text>Types of objects in python<jupyter_code># Example 1
sales=30000
print(sales)
print("type of object is ", type(sales)) 

#Example 2
Avg_expenses =5000.25
print(Avg_expenses)
print("type of object is ", type(Avg_expenses))<jupyter_output>30000
('type of object is ', <type 'int'>)
5000.25
('type of object is ', <type 'float'>)<jupyter_text>Lists in python  Defining lists<jupyter_code>mylist1=["Sheldon","Tommy", "Benny"]
print(type(mylist1))<jupyter_output><type 'list'><jupyter_text>Accessing values from a list<jupyter_code>print(mylist1[0])
print(mylist1[1])<jupyter_output>Sheldon
Tommy<jupyter_text>Length of stringsNumber of elements in the list can be accessed by using len() function<jupyter_code>len(mylist1)<jupyter_output><empty_output><jupyter_text>List appending or combining two lists into one<jupyter_code>mylist2=["Ken","Bill"]
new_list=mylist1 + mylist2
print(new_list)
new_list.append("Mary")
print(new_list)<jupyter_output>['Sheldon', 'Tommy', 'Benny', 'Ken', 'Bill']
['Sheldon', 'Tommy', 'Benny', 'Ken', 'Bill', 'Mary']<jupyter_text>Updating an element in the list<jupyter_code>print("actual list",mylist1)
mylist1[0]="John"
print("list after updating" ,mylist1)<jupyter_output>('actual list', ['Sheldon', 'Tommy', 'Benny'])
('list after updating', ['John', 'Tommy', 'Benny'])<jupyter_text>Deleting an element from the list<jupyter_code>print("actual list",mylist2)
del mylist2[0]
print("list after deleting" ,mylist2)<jupyter_output>('actual list', ['Ken', 'Bill'])
('list after deleting', ['Bill'])<jupyter_text>Difference between array and a list<jupyter_code>val1=[1,7,6]
val2=[6,2,2]
val3=val1+val2
print(val3)

import numpy as np
val4=np.array(val1)+np.array(val2)
print(val4)
details=["John", 1500, "LA"]
print(details)<jupyter_output>['John', 1500, 'LA']<jupyter_text>Lists can contain different datatypes.<jupyter_code>details_all=["John", 1500, "LA",mylist1 ]
print(details_all)<jupyter_output>['John', 1500, 'LA', ['John', 'Tommy', 'Benny']]<jupyter_text>We can add list inside a list.  Packages in python<jupyter_code>print(log(10))
print(sqrt(256))<jupyter_output><empty_output><jupyter_text>Python is throwing errors. It is not due to the wrong syntax or erroneous function names. Python is a multi-purpose language; it doesn’t have log as one of its core functions. Python doesn’t have a square root function either.Python contains a package called math. Just import math package, and you can now safely use log and sqrt.<jupyter_code>import math
print(math.log(10))
print(math.sqrt(256))<jupyter_output>2.30258509299
16.0<jupyter_text>Numpy<jupyter_code>import numpy as np

income = np.array([6725, 9365, 8030, 8750])
print(type(income))
print(income) 
print(income[0])<jupyter_output><type 'numpy.ndarray'>
[6725 9365 8030 8750]
6725<jupyter_text>Creating a new array from income array<jupyter_code>expenses=income*0.65
print(expenses)
savings=income-expenses
print(savings)<jupyter_output>[2353.75 3277.75 2810.5  3062.5 ]<jupyter_text>Array VS list<jupyter_code>income_list=[6725, 9365, 8030, 8750]
income_array = np.array(income_list)
print(income_list*2)
print(income_array*2)<jupyter_output>[6725, 9365, 8030, 8750, 6725, 9365, 8030, 8750]
[13450 18730 16060 17500]<jupyter_text>Pandas<jupyter_code>import pandas as pd
# from google.colab import drive
# drive.mount('/content/drive')
bank= pd.read_csv('Datasets/Bank Tele Marketing/bank_market.csv')
print(bank)<jupyter_output>Cust_num  age            job   marital  education default  balance  \
0             1   58     management   married   tertiary      no     2143   
1             2   44     technician    single  secondary      no       29   
2             3   33   entrepreneur   married  secondary      no        2   
3             4   47    blue-collar   married    unknown      no     1506   
4             5   33        unknown    single    unknown      no        1   
5             6   35     management   married   tertiary      no      231   
6             7   28     management    single   tertiary      no      447   
7             8   42   entrepreneur  divorced   tertiary     yes        2   
8             9   58        retired   married    primary      no      121   
9            10   43     technician    single  secondary      no      593   
10           11   41         admin.  divorced  secondary      no      270   
11           12   29         admin.    single  secondary      no      390   [...]<jupyter_text>Matplotlib<jupyter_code>bank= pd.read_csv('Datasets/Bank Tele Marketing/bank_market.csv')
print(bank.columns)
import matplotlib.pyplot as plt
plt.scatter(bank.age,bank.balance)
plt.show()<jupyter_output><empty_output><jupyter_text>Conditions and Loops in python  If condition Example 1<jupyter_code>level=60
if level<50:
    print("Stage1")
print("Done with If")<jupyter_output>Done with If<jupyter_text>Example 2<jupyter_code>level=60
if level<50:
    print("Stage1")
    print("Done with If")<jupyter_output><empty_output><jupyter_text>Example 3<jupyter_code>level=40
if level<50:
    print("Stage1")
print("Done with If")<jupyter_output>Stage1
Done with If<jupyter_text>Example 4<jupyter_code>level=60
if level<50:
    print("Stage1")
else:
    print("Stage2")
print("Done with If")<jupyter_output>Stage2
Done with If<jupyter_text>For loop Example 1<jupyter_code>names=["Tommy", "Benny", "Ken"]
for i in names:
    print("The name is", i)<jupyter_output>('The name is', 'Tommy')
('The name is', 'Benny')
('The name is', 'Ken')<jupyter_text>Example 2<jupyter_code>nums=range(1,10)
cumsum=0
for i in nums:
    cumsum=cumsum+i
    print("Cumulative sum till", i ,"is", cumsum)<jupyter_output>('Cumulative sum till', 1, 'is', 1)
('Cumulative sum till', 2, 'is', 3)
('Cumulative sum till', 3, 'is', 6)
('Cumulative sum till', 4, 'is', 10)
('Cumulative sum till', 5, 'is', 15)
('Cumulative sum till', 6, 'is', 21)
('Cumulative sum till', 7, 'is', 28)
('Cumulative sum till', 8, 'is', 36)
('Cumulative sum till', 9, 'is', 45)<jupyter_text>Data Importing and basic details<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Importing data from csv files<jupyter_code>sales= pd.read_csv('Datasets/Sales/Sales.csv')
print(sales)<jupyter_output>Cust_id             CustName Product_code Country_code Sales_Type  \
0    SIE39906        <NAME>          HA1          SIE     Direct   
1    EST39196          <NAME>          GI2          EST    Website   
2    AFG39258          <NAME>          ER3          AFG    Website   
3    CAN39302       <NAME>          BR1          CAN    Website   
4    POR39323       <NAME>          LU4          POR    Website   
5    NOR39103       <NAME>          DA1          NOR      Other   
6    ITA39581          <NAME>          DI3          ITA    Website   
7    TUR39194       <NAME>          DE1          TUR    Website   
8    GRE39678        <NAME>          SO2          GRE    Website   
9    MIC39753         <NAME>          AN3          MIC     Direct   
10   PAL39849        <NAME>          JE3          PAL      Other   
11   FIN39735         <NAME>          MA1          FIN    Website   
12   NEW39723        Shad[...]<jupyter_text>Importing data from Microsoft Excel files   Basic Details of dataset.<jupyter_code>print(sales.shape)
print(sales.columns)
print(sales.head(10))
print(sales.tail(10)) 
print(sales.sample(n=10))
print(sales.dtypes) 
print(sales.describe())
print(sales["Invoice_Amount"].describe())
print(sales["Sales_Type"].value_counts())
print(sum(sales["Country_code"].isnull()))
print(sum(sales["CustName"].isnull()))
print(sum(sales["Invoice_Amount"].isnull()))<jupyter_output>21
0
0<jupyter_text>Basic statistics and Data exploration  Basic descriptive statitics<jupyter_code>import pandas as pd
income_data= pd.read_csv(r"Datasets/Census Income Data/Income_data.csv")
print(income_data.shape)
print(income_data.columns)<jupyter_output>(32561, 15)
Index([u'age', u'workclass', u'fnlwgt', u'education', u'education-num',
       u'marital-status', u'occupation', u'relationship', u'race', u'sex',
       u'capital-gain', u'capital-loss', u'hours-per-week', u'native-country',
       u'Income_band'],
      dtype='object')<jupyter_text>Mean<jupyter_code>print(income_data["capital-gain"].mean())<jupyter_output>1077.64884371<jupyter_text>Median<jupyter_code>print(income_data["capital-gain"].median())<jupyter_output>0.0<jupyter_text>Data exploration  Exploring numeric continuous variablesTo explore continuous variables, we use percentiles and percentile distributions. First, let’s try to explore what percentiles are.   Percentiles The variable we are going to use here is capital-gain from income data. In previous discussions, we had already identified something fishy about this variable. Mean is 1,077.6 and median is 0. Let’s now try to calculate it’s percentile values.<jupyter_code>print(income_data["capital-gain"].quantile(0.2))
print(income_data["capital-gain"].quantile([0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]))<jupyter_output>0.0        0.0
0.1        0.0
0.2        0.0
0.3        0.0
0.4        0.0
0.5        0.0
0.6        0.0
0.7        0.0
0.8        0.0
0.9        0.0
1.0    99999.0
Name: capital-gain, dtype: float64<jupyter_text>The output is not really what we expected. At the first glance it is confusing and not easy to comprehend. Let’s study through the output. There are two columns in the output. First column is the percentile, and the other its value. Which is value?  It is value of the variable, which is capital-gain in this case.<jupyter_code>print(income_data["capital-gain"].quantile([0.9,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99,1]))<jupyter_output>0.90        0.0
0.91        0.0
0.92     1458.2
0.93     2885.0
0.94     3818.0
0.95     5013.0
0.96     7298.0
0.97     7688.0
0.98    14084.0
0.99    15024.0
1.00    99999.0
Name: capital-gain, dtype: float64<jupyter_text>As you can see 91 percentiles is zero, from 92 percentile onwards we see some positive values. Which means, 91% of the data is simply zeros, only remaining 9% of the data has some positive values.  For exploring continuous variables we need to use percentiles. They will not only show you the complete distribution of a variable, but they will also help you in identifying outliers in the process. You can even guess what percentage of the data outliers are. If you have a continuous variable, it can take almost any value. It helps to use percentiles if you wish to get a feel of its distribution. **Example of Percentile**Let’s first get the standard percentiles as usual and identify any deep dive areas. For this exercise more than 60 hours is on the higher side and less than 20 lowers. we are not really interested in less than 24 and greater than 55. Let’s start with the deep dive of top 10 percentile. We need to drill down first 10 and last 10 percentiles. As our limits are 20 and 60,<jupyter_code>print(income_data["hours-per-week"].quantile([0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1]))
print(income_data["hours-per-week"].quantile([0.9,0.91,0.92,0.93,0.94,0.95,0.96,0.97,0.98,0.99,1]))
print(income_data["hours-per-week"].quantile([0,0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1]))<jupyter_output>0.0     1.0
0.1    24.0
0.2    35.0
0.3    40.0
0.4    40.0
0.5    40.0
0.6    40.0
0.7    40.0
0.8    48.0
0.9    55.0
1.0    99.0
Name: hours-per-week, dtype: float64
0.90    55.0
0.91    55.0
0.92    58.0
0.93    60.0
0.94    60.0
0.95    60.0
0.96    60.0
0.97    65.0
0.98    70.0
0.99    80.0
1.00    99.0
Name: hours-per-week, dtype: float64
0.00     1.0
0.01     8.0
0.02    10.0
0.03    14.8
0.04    15.0
0.05    18.0
0.06    20.0
0.07    20.0
0.08    20.0
0.09    21.0
0.10    24.0
Name: hours-per-week, dtype: float64<jupyter_text>Now let’s proceed to detection of outliers on the higher side. The output clearly shows 92% of the data is less than 60 and remaining 8% is more than or equal to 60%, so, 8% can be termed as high side outliers. If we are looking for strictly more than 60 hours, then it is 4% of the data.In the output we can clearly observe 8% of the data is less than or equal to 20 hours. The lower side outlier percentage is 8%. If you are strictly looking for the data less than 20 and ignore values equal to 20 hours, this percentile stands at 5%.  **Finally, we have 4% high side outliers and low side 5% outliers. This is how you detect outliers and explore continuous variables.**  Exploring discrete and categorical variablesCategorial variables have certain classes. These classes are nothing but the categories. To explore a categorical variable, we need to create a frequency table, which is much like a group by table or a summary table. A frequency table contains all the available categories (in a given set of data) and frequency counts or number of occurrences of each category type. You will get an idea on all unique classes and their weightage in the given data.<jupyter_code>income_data["education"].value_counts()<jupyter_output><empty_output><jupyter_text>**Along with percentage:**we are trying to first create frequency counts, then percentages are calculated using normalize=True option. We then concatenate frequency and percentages into a table. axis=1 , which indicates we are concatenating columns. The parameter keys mentions the actual column names in the resultant table.<jupyter_code>freq=income_data["education"].value_counts()
percent=income_data["education"].value_counts(normalize=True)
freq_table=pd.concat([freq,percent], axis=1, keys=["counts","percent"])
print(freq_table)
#Frequnecy example -2 
income_data["workclass"].value_counts()<jupyter_output><empty_output><jupyter_text>**Cross tab:**Only 24% are earning more than 50K in the entire available data. Now let’s create a cross tab by comparing both education and income band. After that we will try get to get the income band distribution in each class of education<jupyter_code>#Cross tab
print(income_data["Income_band"].value_counts())
print(income_data["Income_band"].value_counts(normalize=True))
cross_tab=pd.crosstab(income_data["education"],income_data['Income_band'])
cross_tab_p=cross_tab.astype(float).div(cross_tab.sum(axis=1), axis=0)
final_table=pd.concat([cross_tab,cross_tab_p], axis=1)
print(final_table)<jupyter_output><=50K    24720
 >50K      7841
Name: Income_band, dtype: int64
 <=50K    0.75919
 >50K     0.24081
Name: Income_band, dtype: float64
Income_band     <=50K   >50K     <=50K      >50K
education                                       
 10th             871     62  0.933548  0.066452
 11th            1115     60  0.948936  0.051064
 12th             400     33  0.923788  0.076212
 1st-4th          162      6  0.964286  0.035714
 5th-6th          317     16  0.951952  0.048048
 7th-8th          606     40  0.938080  0.061920
 9th              487     27  0.947471  0.052529
 Assoc-acdm       802    265  0.751640  0.248360
 Assoc-voc       1021    361  0.738784  0.261216
 Bachelors       3134   2221  0.585247  0.414753
 Doctorate        107    306  0.259080  0.740920
 HS-grad         8826   1675  0.840491  0.159509
 Masters          764    959  0.443413  0.556587
 Preschool         51      0  1.000000  0.000000
 Prof-school      153    423  0.265625  0.734375
 Some-college    5904   1387  0.8[...]
{'fraction_non_alphanumeric': 0.10882677216421298, 'fraction_numerical': 0.09047279330695894, 'mean_word_length': 3.494316525794229, 'pattern_counts': {'":': 0, '<': 137, '<?xml version=': 0, '>': 133, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '35', 'dir': 'jupyter-structured-clean-dedup', 'id': '233233', 'license': 'MIT', 'n_tokens_mistral': 6162, 'n_tokens_neox': 5357, 'n_words': 1543, 'path': 'CSCE 421/examples/In-class1 (Python Intro).ipynb', 'repo_name': 'ArthurChenCoding/Machine-Learning-projects', 'size': '15499'}