<jupyter_start><jupyter_text>Due to cheap storage, it is common that data is collected greedily, and your first step with pandas is to cut down the data to a dataset that is appropriate for your use.<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Dropping useless data with indices<jupyter_code># Creating a MultiIndex (hierarchical index) object
levels = [('bar', 'one'),
 ('bar', 'two'),
 ('baz', 'one'),
 ('baz', 'two'),
 ('foo', 'one'),
 ('foo', 'two'),
 ('qux', 'one'),
 ('qux', 'two')]

index = pd.MultiIndex.from_tuples(levels, names=['first', 'second'])
index
pd.Series(np.random.randn(8), index=index)
# Basic indexing on axis with MultiIndex
df = pd.DataFrame(np.random.randn(3, 8), index=['A', 'B', 'C'], columns=[
    np.array(['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux']),
    np.array(['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'])
])
df
df["bar"]
df.T.loc[('bar', 'two')]
df.T.loc[('bar', 'two')]["A"]
df.T.loc['baz':'foo']
df.T
df.T.mean(level=0)
df.T.mean(level=1)<jupyter_output><empty_output><jupyter_text>Advanced data cleaning with query and where<jupyter_code># query
df = pd.DataFrame(np.random.rand(10, 3), columns=list('xyz'))
df
# we can query the df like this
df[(df.x< df.y) & (df.y < df.z)]
# or, equivalently
df.query('(x < y) & (y < z)')
# query based on the VALUE of the index
df.query('index > y > z')
# using in and not in
df = pd.DataFrame({'a': list('aabbccddeeff'), 'b': list('aaaabbbbcccc')})
df
df.query('a in b')
df.query('a not in b')
df.query('b == ["a", "b"]')
# where
# DataFrame.where(cond, other=nan, inplace=False, axis=None, 
#     level=None, errors='raise', try_cast=False, raise_on_error=None)

s = pd.Series(range(5))
s
# select a subset
s[s > 0]
# select a subset, but retain the shape of the Series
s.where(s > 0)
# similar, but different
s.mask(s > 0)<jupyter_output><empty_output><jupyter_text>Untangling chained indices with views and copies<jupyter_code># chained indices?
df = pd.DataFrame(np.random.randn(3, 8), index=['A', 'B', 'C'], columns=[
    np.array(['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux']),
    np.array(['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two'])
])
df
# chaining indices
df["bar"]["one"]
# one call
df.loc[:, ("bar", "one")]<jupyter_output><empty_output><jupyter_text>Views vs. Copies**Views** = a way of viewing the data of the array. the data of both objects is shared**Copy** = an actual copy of the data<jupyter_code># correct, safe way to handle the ambiguity
df = pd.DataFrame({'A': ['aaa', 'bbb', 'ccc'], 'B': [1, 2, 3]})
df.loc[0, 'A'] = 11
df
# don't do this
df = df.copy()
df["A"][2] = "ddd"
df
# don't do this either
df = pd.DataFrame({'A': ['aaa', 'bbb', 'ccc'], 'B': [1, 2, 3]})
df.loc[1]['A'] = "ff"<jupyter_output>C:\Users\Rudy\Documents\WinPython-64bit-3.6.2.0Qt5\python-3.6.2.amd64\lib\site-packages\ipykernel_launcher.py:3: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  This is separate from the ipykernel package so we can avoid doing imports until
{'fraction_non_alphanumeric': 0.2043613707165109, 'fraction_numerical': 0.011838006230529595, 'mean_word_length': 5.2592592592592595, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '346199', 'license': 'MIT', 'n_tokens_mistral': 1281, 'n_tokens_neox': 1235, 'n_words': 353, 'path': 'Section 2/Chapter 2.ipynb', 'repo_name': 'PacktPublishing/Troubleshooting-Pandas', 'size': '3210'}