<jupyter_start><jupyter_text>01. How do I read a tabular data file into pandas?<jupyter_code>import pandas as pd
orders = pd.read_table('chipotle.tsv') #tab seperated
# pd.read_table('http://bit.ly/chiporders')
orders.head()
pd.read_table('u.user').head()
# WRONG: the table is not saparated by the tab
# no headers, need to be added
user_cols = ['user_id','age','gender','occupation','zip_code']
users = pd.read_table('u.user', sep='|', header=None,names = user_cols) #'|'separated
users.head()
# what if I only wanna read two columns from the file?
orders = pd.read_table('chipotle.tsv')
print(orders.columns)
orders = pd.read_table('chipotle.tsv',usecols=['quantity','order_id'])
print(orders.columns)
# what if I only wanna read two rows from the file?
orders = pd.read_table('chipotle.tsv', nrows=3)
orders<jupyter_output><empty_output><jupyter_text>02.How do I select a pandas Series from a DataFrame?<jupyter_code># DataFrame: Data table with rows and columns
# Series: each columns of DataFrame call series
import pandas as pd
ufo = pd.read_csv('ufo.csv')
type(ufo)
ufo.head()
type(ufo['City'])
# another way to select the column
# you cant use ufo.Colors Reported or ufo.Colors_Reported
ufo.City.head()
# add and Merge two columns
ufo['location'] = ufo.City + ',' + ufo.State
ufo.head()
# How do DF and Series work with regard to selecting individual entries and iteration
for c in ufo.City.head():
    print(c)
# interation
for index,row in ufo.head().iterrows():
    print(index,row.City,row.State)<jupyter_output>0 Ithaca NY
1 Willingboro NJ
2 Holyoke CO
3 Abilene KS
4 New York Worlds Fair NY<jupyter_text>03.Why do some pandas commands end with parentheses (and others don't)?<jupyter_code>import pandas as pd
# movies = pd.read_csv('http://bit.ly/imdbratings')
movies = pd.read_csv('imdb_1000.csv')
movies.head()
movies.shape # NO parentheses(No args)
movies.dtypes # NO parentheses(No args)<jupyter_output><empty_output><jupyter_text>04.How do I rename columns in a pandas DataFrame?<jupyter_code>import pandas as pd
ufo = pd.read_csv('ufo.csv')
ufo.head()
ufo.columns
# method 1
ufo.rename(
    columns={
        'Colors Reported': 'Colors_Reported',
        'Shape Reported': 'Shape_Reported'
    },
    inplace=True)
ufo.columns
# method 2 : replace all the name
ufo_cols = ['city','colors reported', 'shape reported', 'state', 'time']
ufo.columns = ufo_cols
ufo.head()

# method 3: rename the colunms when read the file
# names -> the new columns names
# header = 0 -> tell the computer where is the header of the table
ufo = pd.read_csv('ufo.csv', names=ufo_cols, header=0) 
ufo.head()
# replace all the space with '_'
ufo.columns = ufo.columns.str.replace(' ','_')
ufo.columns<jupyter_output><empty_output><jupyter_text>04.How do I remove columns from a pandas DataFrame?<jupyter_code>import pandas as pd
ufo = pd.read_csv('ufo.csv')
ufo.head()
# axis = 1, from up to down, columns
# axis = 0, from left to right, rows
ufo.drop('Colors Reported', axis = 1, inplace = True)
ufo.head()
ufo.drop(['City','State'],axis = 1, inplace = True)
ufo.head()
ufo.drop([2,4],axis = 0, inplace = True)
ufo.head()<jupyter_output><empty_output><jupyter_text>05.How do I sort a pandas DataFrame or a Series?<jupyter_code>import pandas as pd
movies = pd.read_csv('imdb_1000.csv')
movies.head()
movies.title.sort_values().head(20)
movies.title.sort_values(ascending=False).head(20)
# the sort method wont change the original data
# return a sorted series
movies.title.head(20)
movies.sort_values('title').head()  #return all dataFrame
movies.sort_values('duration').head()
movies.sort_values('duration', ascending=False).head()
# sort multiple columns
movies.sort_values(['content_rating','duration']).head(20)
# the programming sort the data according to the 'content_rating' first, then 'duration'<jupyter_output><empty_output><jupyter_text>06.How do I filter rows of a pandas DataFrame by column value?<jupyter_code>import pandas as pd
movies = pd.read_csv('imdb_1000.csv')
movies.head()
movies.shape
movies.columns
# filter the movies that longer than 200 mins
# Method 1
booleans = []
for length in movies.duration:
    if length >= 200:
        booleans.append(True)
    else:
        booleans.append(False)
is_long_200 = pd.Series(booleans)
is_long_200.head()
# Method 2
is_long_200 = movies.duration >= 200 # comparison between each of the elements,return True & False
is_long_200.head()
# NOT movies['is_long_200']
movies[is_long_200]
# set a boolean parameters to the [] of a DataFrame, the 'True' parts will output
# Method 3
# movies.duration >= 200 returns a boolean series with the same data length
movies[movies.duration >= 200] 
movies[movies.duration >= 200]['genre']
movies.loc[movies.duration >= 200, 'genre']<jupyter_output><empty_output><jupyter_text>07.How do I apply multiple filter criteria to a pandas DataFrame?<jupyter_code>import pandas as pd
movies = pd.read_csv('imdb_1000.csv')
movies.head()
movies.info()
movies[movies.duration >= 200]
# WORNG: movies[movies.duration >= 200 and movies.genre == 'Drama']
movies[(movies.duration >= 200) & (movies.genre == 'Drama')]
# movies either Crime or Drama or Action
movies[movies.genre.isin(['Crime', 'Drama', 'Action'])].head(20)<jupyter_output><empty_output><jupyter_text>08.Best way to drop every non-numeric column from a DF:<jupyter_code>import pandas as pd
drinks = pd.read_csv('drinks.csv')
drinks.dtypes
drinks.info()
import numpy as np
drinks.select_dtypes(include=[np.number]).dtypes<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1300472899236086, 'fraction_numerical': 0.0205529283375773, 'mean_word_length': 5.617328519855596, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '148835', 'license': 'Apache-2.0', 'n_tokens_mistral': 2025, 'n_tokens_neox': 1905, 'n_words': 536, 'path': 'Yudi TANG/Packages/Pandas/Pandas Q & A.ipynb', 'repo_name': 'JKChang2015/Machine_Learning', 'size': '5498'}