<jupyter_start><jupyter_text>1) Direct Filtering with square bracket `[]`- for data sereies, can use directly index [] => `[1]`- for data frame, need to use column name `["title"]`<jupyter_code>data["id"]
data["id"][1]
# we can use index on data series
# return data frame
data[["artist", "title"]]
data[["artist", "title"]][0]
# cause the error because we can't use index on the data frame
# instead we can use square bracket method on result data frame
# final result is data series
data[["artist", "title"]]["artist"]
# now the final result is data series, we can use indexing again
# but all these seem very complicated
data[["artist", "title"]]["artist"][1]
# selecting range of data by slicing
data[1:2]
data[1:5]
data[1:5]["title"]
data[data["year"] > 1800]
data[data["year"] > 1800]["year"]
data[data["year"] > 1800]["year"][4]<jupyter_output><empty_output><jupyter_text>-------  2) Data Indexing with `.loc`- Data Frame method `.loc` => look at row label, column labels- `df.loc[ROWS, COLS]`<jupyter_code>data = pd.read_csv("Data/artwork_sample.csv")
data.head(2)
data.loc[0, :]
data.loc[0:2, :]
data.loc[0:2, "title"]
data.loc[0:2, ["artist", "title"]]
data.loc[[1,5], ["artist", "title"]]
data.loc[[1,5], "artist":"medium"]
# filtering
data.loc[data.artist == "<NAME>", :]
data.loc[data.artist == "<NAME>", ["artist", "title"]]<jupyter_output><empty_output><jupyter_text>----------  3) Using `.iloc` to access Specific Rows or Columns- Data Frame method, look at `integer position` of Rows and Columns<jupyter_code>data = pd.read_csv("Data/artwork_sample.csv")
data.head(2)
data.iloc[0, :]<jupyter_output><empty_output><jupyter_text>Aware of the difference between `iloc` and `loc`<jupyter_code># take note that iloc look at integer position.
# so it only return data from 0 to 2 (start, end - exclusive)
# it acts same as normal python start, stop position way
data.iloc[0:3, :]
# if we use loc, this return row label number 0 to 3 (inclusive)
# because loc is looking at row label 
data.loc[0:3, :]<jupyter_output><empty_output><jupyter_text>replace row index with specific column<jupyter_code>data.set_index("id", inplace = True)
data.head(2)
# now we have changed row index label, we can clearly see the difference between loc and iloc
data.loc[1036:1038, :]
data.iloc[1:4, :]
# if we use the same integer postion for loc, it returns nothing
# because loc is looking atindex label and there is nothing with index label 1 to 4
data.loc[1:4, :]<jupyter_output><empty_output><jupyter_text>---------- now using for columns index<jupyter_code># as "id" is identified as index, it won't be counted in indexing
# that's why "accession_number" is 0, artist = 1, artistRole = 2
data.iloc[0:3, 0:3]
# if we reset the index label and retry the code above again, we will have different result
# because now "id" column is no longer  row index and it became part of column again.
data.reset_index(inplace = True)
data.iloc[0:3, 0:3]<jupyter_output><empty_output><jupyter_text>------<jupyter_code>data.iloc[[5,8], [0,5]]<jupyter_output><empty_output><jupyter_text>---------  4) Filtering data with `str.contains()`<jupyter_code>data = pd.read_csv("Data/artwork_sample.csv")
data.head(2)
data.medium
type(data.medium)
# for series, we can use .str method
data.medium.str
data.medium.str.contains("Graphite")
data[data.medium.str.contains("Graphite")]
data.loc[data.medium.str.contains("Graphite"), ["artist", "medium"]]
# when using case insensitvie
data.loc[data.medium.str.contains("Graphite", case=False), ["artist", "medium"]]
data.loc[data.medium.str.contains("(?i)Graphite", regex = True), ["artist", "medium"]]<jupyter_output><empty_output><jupyter_text>multiple conditions<jupyter_code>data.loc[data.medium.str.contains("(?i)Graphite", regex = True) | data.medium.str.contains("(?i)Line", regex = True), ["artist", "medium"]]
data.loc[data.medium.str.contains("Graphite|Line", case = False, regex = True), ["artist", "medium"]]<jupyter_output><empty_output><jupyter_text>---------  for column which are not string type, we need to type cast and use str methods<jupyter_code>data.dtypes
data.year.str
data.year.astype(str).str.contains("1826")
data.loc[data.year.astype(str).str.contains("1826")]<jupyter_output><empty_output><jupyter_text>--------<jupyter_code>data = pd.read_csv("Data/artwork_data.csv", low_memory = False)
data.head(2)
data.loc[data.dimensions.str.contains("support")]
# this will cause error because there are rows with missing value
# and NaN can't be converted to str type<jupyter_output><empty_output><jupyter_text>skipping NaN values<jupyter_code>data.loc[data.dimensions.str.contains("support", na = False)]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16445396145610278, 'fraction_numerical': 0.019057815845824413, 'mean_word_length': 5.971641791044776, 'pattern_counts': {'":': 1, '<': 51, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '530806', 'license': 'MIT', 'n_tokens_mistral': 1679, 'n_tokens_neox': 1630, 'n_words': 445, 'path': 'Pandas - Python Data Playbook - Cleaning Data/03.Indexing and Filtering datasets - Art Works Analysis.ipynb', 'repo_name': 'ptyadana/probability-and-statistics-for-business-and-data-science', 'size': '4686'}