<jupyter_start><jupyter_text>Agenda1. Recap: list and loops // Questions about assignment2. List comprehension3. Dictionaries4. Pandas datatypes5. Read data with Pandas6. Explore data with Pandas7. Work with missing values List comprehension<jupyter_code>my_list = ['wordA', 'wordB']

#normal loop
new_list1 = []
for item in my_list:
    new_list1.append(item.upper())

#list comprehension
new_list2 = [item.upper() for item in my_list]

print(new_list1, new_list2)<jupyter_output>['WORDA', 'WORDB'] ['WORDA', 'WORDB']<jupyter_text>Python dictionaries- Dictionary is a python datatype that is used to store key-value pairs. It enables you to quickly retrieve, add, remove, modify values using a key. Dictionary is very similar to what we call associative array or hash in other languages.- {} and seprated by ,Dictionaries and lists share the following characteristics:- Both are mutable (can be changed)- Both are dynamic. They can grow and shrink as needed.- Both can be nested. A list can contain another list. A dictionary can contain another dictionary. A dictionary can also contain a list, and vice versa.- Dictionaries differ from lists primarily in how elements are accessed:List elements are accessed by their position in the list, via indexing.Dictionary elements are accessed via keys.<jupyter_code>mydict = {"name": "Demi",
         "birth_year": 1994, 
          "hobby": "programming"}
print(mydict['name'])
mydict[0]
mydict.keys()
mydict.values()
for key, value in mydict.items():
    print(key.upper())
for item in mydict.values():
    print(item)
#change a value
mydict['name'] = "DeeJay"
mydict.items()
# dictonaries can contain any data type
mydict = {"names": ["Demi", "DeeJay"],
         "birth_year": 1994, 
         "hobby": ["programming", "yoga", "drinking wine"]}<jupyter_output><empty_output><jupyter_text>Exercise- Create a dictionary about yourself, list at least 2 hobbies- Print only your second hobby- What is your birth_year?  Pandas- Pandas stands for “Python Data Analysis Library"- pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, it takes data (like a CSV or TSV file, or a SQL database) and creates a Python object with rows and columns called dataframe that looks very similar to table in a statistical software (think Excel or SPSS for example). - similar to R- pandas is a libary or module, therefore if we want to use it, we need to instal and import it. You can make use of the functions that are defined in the module by calling them with . (dot), like you did with list.split() or string.strip()<jupyter_code># Install a conda package in the current Jupyter kernel
import sys
!conda install --yes --prefix {sys.prefix} pandas<jupyter_output>Collecting package metadata (current_repodata.jsodone
Solving envidone

## Package Plan ##

  environment location: /usr/local/Caskroom/miniconda/base/envs/testj

  added / updated specs:
    - pandas


The following NEW packages will be INSTALLED:

  blas               pkgs/main/osx-64::blas-1.0-mkl
  intel-openmp       pkgs/main/osx-64::intel-openmp-2020.1-216
  libgfortran        pkgs/main/osx-64::libgfortran-3.0.1-h93005f0_2
  mkl                pkgs/main/osx-64::mkl-2019.4-233
  mkl-service        pkgs/main/osx-64::mkl-service-2.3.0-py37hfbe908c_0
  mkl_fft            pkgs/main/osx-64::mkl_fft-1.0.15-py37h5e564d8_0
  mkl_random         pkgs/main/osx-64::mkl_random-1.1.0-py37ha771720_0
  numpy              pkgs/main/osx-64::numpy-1.18.1-py37h7241aed_0
  numpy-base         pkgs/main/osx-64::numpy-base-1.18.1-py37h6575580_1
  pandas             pkgs/main/osx-64::pandas-1.0.3-py37h6c726b0_0
  pytz               pkgs/main/noarch::pytz-2020.1-py_0


Preparing transaction:done
Verifying [...]<jupyter_text>- after the installation we need to import the libary, you need to do import for every Jupyter notebook. - `as pd` is an alias, if you do not do 'as' you will have to type pandas everytime. Programmers are lazy, so we use shortcuts such as pd<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Pandas datatypesThere are two core objects in Pandas: the DataFrame and the Series. SeriesPandas Series is a one-dimensional labeled array, capable of holding data of any type (integer, string, float, python objects, etc.). The axis labels are collectively called index. Pandas Series is nothing, but a column in an Excel sheet. Like in Excel every row in the sheet has - an index- a value or datapoint (if you entered a value)**img from: https://codechalleng.es/bites/251/*** Did we already told you, you can do amazing stuff with markdown? https://about.gitlab.com/handbook/markdown-guide/<jupyter_code># assign the variable s to Series
s = pd.Series(data, index=index)
# lets define data
data = [2,4,6,5]
# lets try it again
s = pd.Series(data, index=index)
# we need to have the same amount of indexes as data points
my_index = [0,1,2,3]
# try to change my_index
pd.Series(data, index=my_index)<jupyter_output><empty_output><jupyter_text>ExerciseHow can you use python functions to define the index? Remember, you're lazy!- Hint:     - Length of the data and the index needs to be the same    - Have you used the range function before?  DataFrameA DataFrame is a table. It contains an array of individual entries, each of which has a certain value. Each entry corresponds to a row (or record) and a column.- not limited to integers also strings ** image from = https://www.geeksforgeeks.org/ and https://www.learndatasci.com/For example, consider the following simple DataFrame<jupyter_code>df_with_numbers = pd.DataFrame({'Yes': [53, 21], 'No': [13, 1]})
df_with_numbers
pd.DataFrame({'Bob': ['I liked it.', 'It was awful.'], 'Sue': ['Pretty good.', 'Boring.']})<jupyter_output><empty_output><jupyter_text>Read dataBeing able to create a DataFrame or Series manually is handy. But, most of the time, we won't actually create our own data manually. Instead, we'll be working with data that already exists.Data can be stored in any number of different forms and formats. By far the most basic is a CSV file. When you open a CSV file you get something that looks like this:Product A,Product B,Product C,30,21,9,35,34,1,41,11,11Download data from Kaggle or take a look at this data descriprion:https://www.kaggle.com/kimjihoo/ds4c-what-is-this-dataset-detailed-description<jupyter_code># read the data and store it in df variable
path = 'data/coronavirusdataset/Case.csv'
df = pd.read_csv(path)<jupyter_output><empty_output><jupyter_text>Viewing and Inspecting DataNow that you’ve loaded your data, it’s time to take a look at it. How does the dataframe look like? Running the name of the data frame would give you the entire table, but you can also use functions<jupyter_code># get the first n rows with df.head(n), or the last n rows with df.tail(n)
df.head()
len(df)
# check the number of rows and columns
df.shape
# important to check non-null values
df.info()
# check only the columns
df.columns
df.group
#df['group']
df['city'].describe()
df['province'].unique()
# view unique values and counts for a series (like a column or a few columns)
df['city'].value_counts()<jupyter_output><empty_output><jupyter_text>Exercise1. How many individual provinces does this dataset contain?2. Display the top three MENTIONED provinces  Slices<jupyter_code>df[1:4]
cases_in_gurogu = df[df.city == 'Guro-gu']
cases_in_gurogu
df.confirmed.sum()<jupyter_output><empty_output><jupyter_text>Exercise1. How many confirmed cases are there in Eunpyeong-gu ?  Missing dataEntries with missing values are given the value NaN, short for "Not a Number". For technical reasons these NaN values are always float64 dtype. Copying dataframeIn Pandas, indexing a DataFrame returns a reference to the initial DataFrame. By changing the subset we change the initial DataFrame. Thus, you'd want to use the copy if you want to make sure the initial DataFrame shouldn't be changed. Consider the following code:<jupyter_code># index, column
missing_data_df = df.copy()
missing_data_df
# create missing values
missing_data_df.at[0, 'confirmed'] = None
missing_data_df<jupyter_output><empty_output><jupyter_text>Pandas provides some methods specific to manipulating the missing data. To select NaN entries you can use pd.isnull() (or its companion pd.notnull()).<jupyter_code>df[pd.isnull(df.city)]
# df.isnull().values.any()
# df.info<jupyter_output><empty_output><jupyter_text>Replacing missing values is a common operation. Pandas provides a really handy method for this problem: fillna~(). fillna() provides a few different strategies for mitigating such data. For example, we can simply replace each NaN with an "Unknown":<jupyter_code># if any non value exsist, fill with unknown
df.city.fillna("Unknown")<jupyter_output><empty_output><jupyter_text>Exercise1. fill the missing values of the confirmed cases with the average of the confirmed cases Missing values are not always NaN, they can also be ["n/a", "na", "-", ""]. If needed we can also replace these values.<jupyter_code># df.latitude
df.latitude.unique() # check the - 
# replace values
df.latitude.replace('-', "unknown")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09275425578350066, 'fraction_numerical': 0.02062418158009603, 'mean_word_length': 4.615808823529412, 'pattern_counts': {'":': 7, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '667919', 'license': 'MIT', 'n_tokens_mistral': 2973, 'n_tokens_neox': 2803, 'n_words': 1092, 'path': 'lessons/Week3-lesson.ipynb', 'repo_name': 'pyladiesams/Bootcamp-Data-Analysis-beginner-apr-may2020', 'size': '9164'}