<jupyter_start><jupyter_text>*+ This notebook is based on <NAME>'s awesome [pandas series](https://www.youtube.com/playlist?list=PL-osiE80TeTsWmV9i9c58mdDCSskIFdDS).*<jupyter_code>import pandas as pd
# dataset source: https://insights.stackoverflow.com/survey
df = pd.read_csv('developer_survey_2020/survey_results_public.csv', index_col='Respondent')
schema_df = pd.read_csv('developer_survey_2020/survey_results_schema.csv', index_col='Column')
pd.set_option('display.max_rows', 61)
pd.set_option('display.max_columns', 61)
df.shape
df.info()
df.head()  # / df.tail()
df.columns
schema_df<jupyter_output><empty_output><jupyter_text>Indexing<jupyter_code># df.set_index('Respondent', inplace=True)
# df.reset_index(inplace=True)

# schema_df.set_index('Column', inplace=True)
# schema_df.reset_index(inplace=True)
schema_df.loc['Hobbyist']
df['Hobbyist'].value_counts()
df.loc[:2, 'Hobbyist':'Age1stCode']  # df.loc[[1, 2], ['Hobbyist', 'Age', 'Age1stCode']]
schema_df.sort_index(ascending=True, inplace=False)<jupyter_output><empty_output><jupyter_text>Filtering<jupyter_code>schema_df.loc['ConvertedComp']
df.loc[df['ConvertedComp'] > 1000000, ['Country', 'LanguageWorkedWith', 'ConvertedComp']]
df.loc[df['Country'].isin(['United States', 'India', 'United Kingdom', 'Germany', 'Canada']), 'Country']
df.loc[df['LanguageWorkedWith'].str.contains('Python', na=False), 'LanguageWorkedWith']<jupyter_output><empty_output><jupyter_text>Updating rows and columns```pythondf = pd.DataFrame(    {        'First Name': ['Ol', 'Jane', 'Lorem'],        'Last Name': ['D', 'Doe', 'Ipsium'],        'Email': ['<EMAIL>', '<EMAIL>', '<EMAIL>']    }) ways to update column namesdf.columns = [x.lower() for x in df.columns]   make column names lowercasedf.columns = df.columns.str.replace(' ', '_')   replace spaces with underscoresdf.rename(columns={'first_name':'first', 'last_name':'last'}, inplace=True) way to update multiple rows at oncedf['email'] = df['email'].str.lower() 'apply' on series --> applies a function to every value in the seriesdf['email'] = df['email'].apply(lambda x: x.lower()) 'apply' on dataframe --> applies a function to series in the dataframedf.apply(lambda x: x.min())df.apply(pd.Series.min)df.apply(len, axis='columns') applymap --> used to apply a function to every element in the df (can't be used on series)df.applymap(len)df.applymap(str.lower) map --> used for substituting values in a series with another valuedf['email'].map({'<EMAIL>':'<EMAIL>'}) replace  --> replace the values which are mentioneddf = df['email'].replace({'<EMAIL>':'<EMAIL>'})```<jupyter_code>df['Hobbyist'] = df['Hobbyist'].map({'Yes': True, 'No': False})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.18225866567275437, 'fraction_numerical': 0.011181513231457324, 'mean_word_length': 7.361370716510903, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 31, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '378526', 'license': 'MIT', 'n_tokens_mistral': 980, 'n_tokens_neox': 930, 'n_words': 181, 'path': 'pandas/notebook.ipynb', 'repo_name': 'priyavrat-misra/py-libraries-practice', 'size': '2739'}