<jupyter_start><jupyter_text>Encoding Data  When we encode data, we remodel a numeric column in a dataframe to be on a standard scale (0 or 1).   For example if we had a column of BMI scores, we could encode that column so that all scores greater than or equal to 25 were recoded to the value 1 (bad) and all scores less than 25 were recoded to 0 (good).  For example:  ` def encode_bmi(df):`       > `if df['bmi'] >= 25:`  > >  `return 1`   >`else`:  > >`return 0`  `df["bmi"] = df.apply(encode_bmi, axis=1)`      Challenge 1 - prepare dataset for encoding ---1. Read Covid vaccination data from the `by_country` sheet in the Excel file at this link : https://github.com/lilaceri/Working-with-data-/blob/342abab10d93c4bf23b5c55a50f189f12a137c5f/Data%20Sets%20for%20code%20divisio/Covid%20Vaccination%20Data.xlsx?raw=true2. Find out which columns have missing values3. Remove all rows with missing data in the total_vaccination column  4. Remove all rows with missing data in the daily_vaccinations_per_million 4. find the median daily_vaccinations_per_million, storing this in a variable for use later     **Test output**:  1. dataframe is saved in a variable2. ```RangeIndex: 14994 entries, 0 to 14993Data columns (total 15 columns):    Column                               Non-Null Count  Dtype                                          0   country                              14994 non-null  object         1   iso_code                             14994 non-null  object         2   date                                 14994 non-null  datetime64[ns] 3   total_vaccinations                   9011 non-null   float64        4   people_vaccinated                    8370 non-null   float64        5   people_fully_vaccinated              6158 non-null   float64        6   daily_vaccinations_raw               7575 non-null   float64        7   daily_vaccinations                   14796 non-null  float64        8   total_vaccinations_per_hundred       9011 non-null   float64        9   people_vaccinated_per_hundred        8370 non-null   float64        10  people_fully_vaccinated_per_hundred  6158 non-null   float64        11  daily_vaccinations_per_million       14796 non-null  float64        12  vaccines                             14994 non-null  object         13  source_name                          14994 non-null  object         14  source_website                       14994 non-null  object        dtypes: datetime64[ns](1), float64(9), object(5)memory usage: 1.7+ MB```3. 9011 rows × 15 columns  4. 8815 rows * 15 columns 5. 6.65<jupyter_code>import pandas as pd
import numpy as np

url = "https://github.com/lilaceri/Working-with-data-/blob/342abab10d93c4bf23b5c55a50f189f12a137c5f/Data%20Sets%20for%20code%20divisio/Covid%20Vaccination%20Data.xlsx?raw=true"

# Task 1 
def create_dataframe(url, db_type='csv', sheetname=None):
  import pandas as pd
  if db_type == 'csv':
    df = pd.read_csv(url)
  elif db_type == 'excel':
    if sheetname == None:
      df = pd.read_excel(url)
    else:
      df = pd.read_excel(url, sheet_name=sheetname)
  else:
    df = pd.read_csv(url)
  return df

covid_df = create_dataframe(url, db_type='excel', sheetname='by_country')
# Task 2
display(covid_df.info())
display(covid_df.head())

# Task 3
covid_total_df = covid_df.dropna(subset=['total_vaccinations'])
print('')
display(covid_total_df.info())

# Task 4
covid_daily_df = covid_total_df.dropna(subset=['daily_vaccinations_per_million'])
print('')
display(covid_daily_df.info())

# Task 5
np_daily_vac_miln = np.array(covid_daily_df['daily_vaccinations_per_million'], np.float64)
daily_vac_miln_median = np.median(np_daily_vac_miln)

print(f'\nMedian (daily_vaccinations_per_million): {daily_vac_miln_median}')<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 14994 entries, 0 to 14993
Data columns (total 15 columns):
 #   Column                               Non-Null Count  Dtype         
---  ------                               --------------  -----         
 0   country                              14994 non-null  object        
 1   iso_code                             14994 non-null  object        
 2   date                                 14994 non-null  datetime64[ns]
 3   total_vaccinations                   9011 non-null   float64       
 4   people_vaccinated                    8370 non-null   float64       
 5   people_fully_vaccinated              6158 non-null   float64       
 6   daily_vaccinations_raw               7575 non-null   float64       
 7   daily_vaccinations                   14796 non-null  float64       
 8   total_vaccinations_per_hundred       9011 non-null   float64       
 9   people_vaccinated_per_hundred        8370 non-null   float64       
 10  people_ful[...]<jupyter_text>Challenge 2 - encode daily vaccinations ---Write a function to encode daily vaccinations per million, where values greater than or equal to median = 1 and values less than median = 0 **Test output**: using describe()```count    8816.000000mean        0.991493std         0.091847min         0.00000025%         1.00000050%         1.00000075%         1.000000max         1.000000Name: daily_vaccinations_per_million, dtype: float64```<jupyter_code># Task
def encode_daily(df, **kwds):
  key = kwds['key']
  median = kwds['median']
  if df[key] >= median:
    return 1
  else:
    return 0
  return

key = 'daily_vaccinations_per_million'
covid_daily_df['daily_vac_encoded'] = covid_daily_df.apply(encode_daily, axis=1, key=key, median=daily_vac_miln_median)
display(covid_daily_df['daily_vac_encoded'].describe())<jupyter_output>/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:12: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  if sys.path[0] == '':<jupyter_text>Challenge 3 - Encoding total vaccinations   ---The United Kingdom has been praised for its fast vaccine rollout. 1. Find the minimum total vaccinations for the United Kingdom 2. Write a function to encode total_vaccinations column so that all values less than the UK's min are 0 and all values greater than or equal to the UK's min are coded as 1 3. Display the unique countries for which total vaccinated is at the same rate or more than the UK**Test output**:1. 1402432.02. `df['total_vaccinations'].describe()` should output:```count    9011.00000mean        0.29808std         0.45744min         0.0000025%         0.0000050%         0.0000075%         1.00000max         1.00000Name: total_vaccinations, dtype: float64```3. ```array(['Argentina', 'Australia', 'Austria', 'Azerbaijan', 'Bangladesh',       'Belgium', 'Brazil', 'Cambodia', 'Canada', 'Chile', 'China',       'Colombia', 'Czechia', 'Denmark', 'Dominican Republic', 'England',       'Finland', 'France', 'Germany', 'Greece', 'Hong Kong', 'Hungary',       'India', 'Indonesia', 'Ireland', 'Israel', 'Italy', 'Japan',       'Kazakhstan', 'Malaysia', 'Mexico', 'Morocco', 'Nepal',       'Netherlands', 'Norway', 'Pakistan', 'Peru', 'Philippines',       'Poland', 'Portugal', 'Qatar', 'Romania', 'Russia', 'Saudi Arabia',       'Scotland', 'Serbia', 'Singapore', 'Slovakia', 'South Korea',       'Spain', 'Sweden', 'Switzerland', 'Thailand', 'Turkey',       'United Arab Emirates', 'United Kingdom', 'United States',       'Uruguay', 'Wales'], dtype=object)```<jupyter_code># Task 1
country_min = covid_total_df.groupby(['country'])['total_vaccinations'].min()
uk_min = country_min['United Kingdom']
display(uk_min)

# Task 2
def encode_total(df, **kwds):
  key = kwds['key']
  min = kwds['min']
  if df[key] >= min:
    return 1
  else:
    return 0
  return

key = 'total_vaccinations'
covid_total_df['total_vac_encoded'] = covid_total_df.apply(encode_total, axis=1, key=key, min=uk_min)
display(covid_total_df['total_vac_encoded'].describe())

# Task 3
covid_total_df_enc = covid_total_df[covid_total_df['total_vac_encoded'] == 1]['country'].unique()
print('')
display(covid_total_df_enc)<jupyter_output><empty_output><jupyter_text>Challenge 4 - create new series of total vaccinations for each manufacturer---To create a new column in your dataframe:`df['new_column'] = ...`For example:* to duplicate an existing column  * `df['new_column'] = df['old_column']`* to add two columns together   * `df['new_column'] = df['column1'] + df['column2']`* to make a percentages column   * `df['new_column'] = (df['column1']/df['column1].sum()) * 100`  1. read data from 'by_manufacturer' sheet from Covid data 2. find the sum of total vaccinations for each manufacturer3. create a new column that has the total vaccinations as a percentage of the overall sum of total vaccinations 4. find the median percentage 5. create a new column called 'encoded_percentages' which duplicates the percentages column6. encode the encoded_percentages column so that any values greater than or equal to the median percentage = 1 and any lesser than = 0 **Test output**:1.2. ```vaccineJohnson&Johnson        264839828Moderna               5548036383Oxford/AstraZeneca     539433203Pfizer/BioNTech       8690461304Sinovac                604660293Name: total_vaccinations, dtype: int64```3. ```	location	date	vaccine	total_vaccinations	percentages0	Chile	2020-12-24	Pfizer/BioNTech	420	0.0000031	Chile	2020-12-25	Pfizer/BioNTech	5198	0.0000332	Chile	2020-12-26	Pfizer/BioNTech	8338	0.0000533	Chile	2020-12-27	Pfizer/BioNTech	8649	0.0000554	Chile	2020-12-28	Pfizer/BioNTech	8649	0.000055...	...	...	...	...	...3291	United States	2021-05-01	Moderna	105947940	0.6770953292	United States	2021-05-01	Pfizer/BioNTech	129013657	0.8245043293	United States	2021-05-02	Johnson&Johnson	8374395	0.0535193294	United States	2021-05-02	Moderna	106780082	0.6824133295	United States	2021-05-02	Pfizer/BioNTech	130252779	0.8324233296 rows × 5 columns```4. 0.00111101943748969315. ```location	date	vaccine	total_vaccinations	percentage_of_total	encoded_percentages0	Chile	2020-12-24	Pfizer/BioNTech	420	0.000003	0.0000031	Chile	2020-12-25	Pfizer/BioNTech	5198	0.000033	0.0000332	Chile	2020-12-26	Pfizer/BioNTech	8338	0.000053	0.0000533	Chile	2020-12-27	Pfizer/BioNTech	8649	0.000055	0.0000554	Chile	2020-12-28	Pfizer/BioNTech	8649	0.000055	0.000055...	...	...	...	...	...	...3291	United States	2021-05-01	Moderna	105947940	0.677095	0.6770953292	United States	2021-05-01	Pfizer/BioNTech	129013657	0.824504	0.8245043293	United States	2021-05-02	Johnson&Johnson	8374395	0.053519	0.0535193294	United States	2021-05-02	Moderna	106780082	0.682413	0.6824133295	United States	2021-05-02	Pfizer/BioNTech	130252779	0.832423	0.8324233296 rows × 6 columns```6. ```	location	date	vaccine	total_vaccinations	percentages	encode	encoded0	Chile	2020-12-24	Pfizer/BioNTech	420	0.000003	0.000003	01	Chile	2020-12-25	Pfizer/BioNTech	5198	0.000033	0.000033	02	Chile	2020-12-26	Pfizer/BioNTech	8338	0.000053	0.000053	03	Chile	2020-12-27	Pfizer/BioNTech	8649	0.000055	0.000055	04	Chile	2020-12-28	Pfizer/BioNTech	8649	0.000055	0.000055	0...	...	...	...	...	...	...	...3291	United States	2021-05-01	Moderna	105947940	0.677095	0.677095	13292	United States	2021-05-01	Pfizer/BioNTech	129013657	0.824504	0.824504	13293	United States	2021-05-02	Johnson&Johnson	8374395	0.053519	0.053519	13294	United States	2021-05-02	Moderna	106780082	0.682413	0.682413	13295	United States	2021-05-02	Pfizer/BioNTech	130252779	0.832423	0.832423	13296 rows × 7 columns```<jupyter_code># Task 1
covid_manu_df = create_dataframe(url, db_type='excel', sheetname='by_manufacturer')
display(covid_manu_df.info())
display(covid_manu_df.head())
# Task 2
covid_manu_sum_df = covid_manu_df.groupby(['vaccine'])['total_vaccinations'].sum()
display(covid_manu_sum_df)
# Task 3
covid_manu_df['percentages'] = (covid_manu_df['total_vaccinations']/covid_manu_sum_df.sum()) * 100
display(covid_manu_df.head())
print('')
display(covid_manu_df.tail())
# Task 4
percentage_median = np.median(covid_manu_df['percentages'])
print(f'Percentages Median: {percentage_median}')
# Task 5
covid_manu_df['encode'] = covid_manu_df['percentages']

# Task 6
def encode_percentages(df, **kwds):
  key = kwds['key']
  median = kwds['median']
  if df[key] >= median:
    return 1
  else:
    return 0
  return

key = 'encode'
covid_manu_df['encoded'] = covid_manu_df.apply(encode_percentages, axis=1, key=key, median=percentage_median)
display(covid_manu_df['encoded'].describe())
display(covid_manu_df.head())
display(covid_manu_df.tail())<jupyter_output><empty_output><jupyter_text>Exercise 8 - encode daily vaccinations 1. find the median daily vaccinations per 1 million 2. write a function to encode daily vaccinations per 1 million, where values greater than or equal to median = 1 and values less than median = 0 Output: 1. 1915.52. ```0        06        022       044       059       0        ..14989    014990    014991    014992    014993    0Name: daily_vaccinations_per_million, Length: 9011, dtype: int64```<jupyter_code><jupyter_output><empty_output><jupyter_text>Exercise 9 - Encoding vaccinations per hundred  ---The United Kingdom has been praised for its fast vaccine rollout. 1. find the minimum total vaccinations for the United Kingdom 2. save this value in a variable rounded down to an integer3. write a function to encode total_vaccinations column so that all values less than the UK's min are 0 and all values greater than or equal to the UK's min are coded as 1 4. display the countries which total vaccinated is at the same rate or more than the UKOutput:1. 1402432.02. 14024323. `df['people_vaccinated_per_hundred']` should output:```0        06        022       044       059       0        ..14989    014990    014991    014992    014993    0Name: total_vaccinations, Length: 9011, dtype: int64```4. ```array(['Argentina', 'Australia', 'Austria', 'Azerbaijan', 'Bangladesh',       'Belgium', 'Brazil', 'Cambodia', 'Canada', 'Chile', 'China',       'Colombia', 'Czechia', 'Denmark', 'Dominican Republic', 'England',       'Finland', 'France', 'Germany', 'Greece', 'Hong Kong', 'Hungary',       'India', 'Indonesia', 'Ireland', 'Israel', 'Italy', 'Japan',       'Kazakhstan', 'Malaysia', 'Mexico', 'Morocco', 'Nepal',       'Netherlands', 'Norway', 'Pakistan', 'Peru', 'Philippines',       'Poland', 'Portugal', 'Qatar', 'Romania', 'Russia', 'Saudi Arabia',       'Scotland', 'Serbia', 'Singapore', 'Slovakia', 'South Korea',       'Spain', 'Sweden', 'Switzerland', 'Thailand', 'Turkey',       'United Arab Emirates', 'United Kingdom', 'United States',       'Uruguay', 'Wales'], dtype=object)```<jupyter_code><jupyter_output><empty_output><jupyter_text>Exercise 10 - create new series of total vaccinations percentages---To create a new column in your dataframe:`df['new_column'] = ...`For example:* to duplicate an existing column  * `df['new_column'] = df['old_column']`* to add two columns together   * `df['new_column'] = df['column1'] + df['column2']`* to make a percentages column   * `df['new_column'] = (df['column1']/df['column1].sum()) * 100`    1. read data from 'by_manufacturer' sheet from Covid data 2. find the sum of total vaccinations for each manufacturer3. create a new column that has the total vaccinations as a percentage of the overall sum of total vaccinations 4. find the median percentage 5. create a new column called 'encoded_percentages' which duplicates the percentages column6. encode the encoded_percentages column so that any values greater than or equal to the median percentage = 1 and any lesser than = 0 Output:1.2. ```vaccineJohnson&Johnson        264839828Moderna               5548036383Oxford/AstraZeneca     539433203Pfizer/BioNTech       8690461304Sinovac                604660293Name: total_vaccinations, dtype: int64```3. ```	location	date	vaccine	total_vaccinations	percentages0	Chile	2020-12-24	Pfizer/BioNTech	420	0.0000031	Chile	2020-12-25	Pfizer/BioNTech	5198	0.0000332	Chile	2020-12-26	Pfizer/BioNTech	8338	0.0000533	Chile	2020-12-27	Pfizer/BioNTech	8649	0.0000554	Chile	2020-12-28	Pfizer/BioNTech	8649	0.000055...	...	...	...	...	...3291	United States	2021-05-01	Moderna	105947940	0.6770953292	United States	2021-05-01	Pfizer/BioNTech	129013657	0.8245043293	United States	2021-05-02	Johnson&Johnson	8374395	0.0535193294	United States	2021-05-02	Moderna	106780082	0.6824133295	United States	2021-05-02	Pfizer/BioNTech	130252779	0.8324233296 rows × 5 columns```4. 0.00111101943748969315. 6. ```	location	date	vaccine	total_vaccinations	percentages	encode	encoded0	Chile	2020-12-24	Pfizer/BioNTech	420	0.000003	0.000003	01	Chile	2020-12-25	Pfizer/BioNTech	5198	0.000033	0.000033	02	Chile	2020-12-26	Pfizer/BioNTech	8338	0.000053	0.000053	03	Chile	2020-12-27	Pfizer/BioNTech	8649	0.000055	0.000055	04	Chile	2020-12-28	Pfizer/BioNTech	8649	0.000055	0.000055	0...	...	...	...	...	...	...	...3291	United States	2021-05-01	Moderna	105947940	0.677095	0.677095	13292	United States	2021-05-01	Pfizer/BioNTech	129013657	0.824504	0.824504	13293	United States	2021-05-02	Johnson&Johnson	8374395	0.053519	0.053519	13294	United States	2021-05-02	Moderna	106780082	0.682413	0.682413	13295	United States	2021-05-02	Pfizer/BioNTech	130252779	0.832423	0.832423	13296 rows × 7 columns```<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10617526428282595, 'fraction_numerical': 0.13095719484720697, 'mean_word_length': 3.5153886280646844, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 38, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '326409', 'license': 'MIT', 'n_tokens_mistral': 7534, 'n_tokens_neox': 5906, 'n_words': 1639, 'path': 'Worksheets/15_1_Encoding_and_Dummy_Coding.ipynb', 'repo_name': 'Imran-Github22/Programming-For-Data', 'size': '17311'}