<jupyter_start><jupyter_text>Data Analysis With Pandas  Day Objectives- Series- DataFrame- Indexing- Features- Filtering- File imporitng- Save/Data File Exporting<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>**Series:** it is a group of 1d similar datatype of elementsList, Tuples, numpy-arraystring, complex, float, int -> String/Object...<jupyter_code>s1 = pd.Series([1, 2, 2, 5, 5])
s2 = pd.Series((1,2,3,4,5,6))
s3 = pd.Series(np.arange(1, 100))
type(s1), type(s2), type(s3)
s1
s2
s3
s3.max()
s3.min()
s3.sum()
s3.cumsum()
s1 = pd.Series([1, 2, 2, 5, 5], index = [50,51,52,53,54])
s1
s2
s1 = pd.Series([1, 2, 2, 5, 5], index = ['ele1','ele2','ele3','ele4','ele5'])
s1
s3.index
s1.index
datae = pd.date_range('01-01-2020', periods = 180)
type(datae)
datae
pd.date_range('01-01-2020', periods = 180, freq = '1H')
help(pd.date_range)
pd.date_range(start = '01-01-2020', end = '30-06-2020', freq = '5H')
pd.date_range(start = '01-01-2020', end = '30-06-2020', freq = '5S')
date = pd.Series(pd.date_range(start = '01-01-2020', end = '30-06-2020', freq = '5S'))
date<jupyter_output><empty_output><jupyter_text>DataFrameit is a group of 2d similar datatype of elementsList, Tuples, numpy-array, dictionarystring, complex, float, int -> String/Object...<jupyter_code>df = pd.DataFrame([[1 , 2], [3, 4]])
df1 = pd.DataFrame((1, 3), (2, 3))
df2 = pd.DataFrame(np.array([[1 , 2], [3, 4]]))
df3 = pd.DataFrame({'col1':[1, 3], 'col2':[5, 7]})
df
df2
df3
type(df3)
df3['col1']
help(pd.DataFrame)
df = pd.DataFrame([[1 , 2], [3, 4]], columns = ['col1', 'col2'], index = ['row1', 'row2'])

df
df.columns
df.columns = ['column1', 'column2']
df
df.index
df.index = ['index1', 'index2']
df
stu_data = {'rollNo':['20SDCDA' + str(i) for i in range(1, 101)],
           'Pandas':np.random.randint(1, 100, 100),
           'Numpy':np.random.randint(1, 100, 100), 
           'C':np.random.randint(1, 100, 100),
           'Java':np.random.randint(1, 100, 100)}

data = pd.DataFrame(stu_data)

data
data.head()
data.head(20)
data.tail()
data.tail(6)<jupyter_output><empty_output><jupyter_text>Indexing<jupyter_code>data['Pandas']
type(data['Pandas'])
data[['Pandas', 'Java']]<jupyter_output><empty_output><jupyter_text>iloc and loc<jupyter_code>data.iloc[50:61]
df
df.iloc[0]
df.loc['index1']
data.iloc[50:61][['rollNo','Numpy']]
data.iloc[37][['rollNo','Numpy']]
stdata = data.set_index('rollNo')
data
stdata.loc['20SDCDA55']
stdata.sum(axis = 'columns')
stdata
stdata['Total'] = stdata.sum(axis = 'columns')

stdata.head()
stdata['Percentage'] = (stdata['Total'] * 100)/400

stdata.head()
stdata['Grade'] = np.zeros(100, dtype = 'str')
stdata.head()<jupyter_output><empty_output><jupyter_text>Filtering<jupyter_code>stdata['Grade'][stdata['Percentage'] >= 40] = 'Pass'
stdata['Grade'][stdata['Percentage'] < 40] = 'Fail'
stdata.head()<jupyter_output><ipython-input-70-50cfeb1cc8c5>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  stdata['Grade'][stdata['Percentage'] >= 40] = 'Pass'
<ipython-input-70-50cfeb1cc8c5>:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  stdata['Grade'][stdata['Percentage'] < 40] = 'Fail'<jupyter_text>Data Importing and Exporting [Datasets Directory for Analysis](https://github.com/AP-State-Skill-Development-Corporation/Datasets/tree/master/Data%20Analysis) [Datasets Wiki for Reference](https://github.com/AP-State-Skill-Development-Corporation/Datasets/wiki)<jupyter_code>pd.read_clipboard()<jupyter_output><empty_output><jupyter_text>[DHS Daily ReportSocial Services](https://data.cityofnewyork.us/Social-Services/DHS-Daily-Report/k46n-sa2m)This dataset includes the daily number of families and individuals residing in the Department of Homeless Services (DHS) shelter system and the daily number of families applying to the DHS shelter system.<jupyter_code>df1 = pd.read_csv('https://raw.githubusercontent.com/AP-State-Skill-Development-Corporation/Datasets/master/Data%20Analysis/zip_to_puma.csv')
pd.read_csv('zip_to_puma.csv')
pd.read_csv('chipotle.tsv.txt', sep ='\t')
df = pd.read_json('dhs_daily_report.json')
df.to_csv('dhs_daily_report.csv')
syllabus = pd.read_html('https://github.com/AP-State-Skill-Development-Corporation/Data-Analysis-Using-Python-MB9/blob/master/Syllabus.md')[0]
syllabus.to_csv('syllabus.csv')
df.head()
df1.head()
df1['postalcity'].value_counts()
df1.shape
df1['zipcode'].value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1812108559498956, 'fraction_numerical': 0.05720250521920668, 'mean_word_length': 5.994160583941606, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '237641', 'license': 'CC0-1.0', 'n_tokens_mistral': 2081, 'n_tokens_neox': 1913, 'n_words': 354, 'path': 'Day04_30Nov2020/Day04_30Nov2020.ipynb', 'repo_name': 'AP-State-Skill-Development-Corporation/Data-Analysis-Using-Python-MB9', 'size': '4790'}