<jupyter_start><jupyter_text>Week 1 & 2<NAME>[Green]:  (bce35b)[Purple]:  (ae8bd5)[Coral]:  (9c4957)[Grey]:  (8c8c8c) ****** Importing libraries<jupyter_code>import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import os
import requests
from collections import deque as dq
from itertools import permutations, dropwhile, zip_longest


## set current working directory
path = os.path.expanduser('~') + '\\OneDrive - Bellevue University\\Bellevue_University\\DSC 540 - Data Preparation'
os.chdir(path)


## matplotlib default settings
plt.style.use('dark_background')
mpl.rcParams.update({'lines.linewidth':3})
mpl.rcParams.update({'axes.labelsize':14})
mpl.rcParams.update({'axes.titlesize':16})
mpl.rcParams.update({'axes.titleweight':'bold'})
mpl.rcParams.update({'figure.autolayout':True})
mpl.rcParams.update({'axes.grid':True, 'grid.color':'#424242', 'grid.linestyle':'--'})

## see list of current settings
# plt.rcParams.keys()<jupyter_output><empty_output><jupyter_text>Instantiating random number generator<jupyter_code>rng_seed = 777
rng = np.random.default_rng(rng_seed)<jupyter_output><empty_output><jupyter_text>*** List Iteration<jupyter_code>my_lst = [1,2,5,9,3]

## creating e random numbers to add to list
rand_nums = rng.integers(low=1, high=20, size=3)

## loop through random numbers and append to list
for i in rand_nums:
    my_lst.append(i)
    
my_lst<jupyter_output><empty_output><jupyter_text>*** Line Chart<jupyter_code>## import data
df = pd.read_excel('world-population.xlsm')

plt.figure(figsize=(10,5))
plt.plot(df['Year'], df['Population']/100000000, c='lightgreen')
plt.xlabel('Year')
plt.ylabel('Population (100 million)')
plt.title('Population Trending by Year')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>*** Book Activities  Activity 1Generate list of random numbers and then another list from the first one, which only contains numbers that are divisible by three. Repeat the experiment 3 times. Then, we will calculate the average difference of length between the two lists.<jupyter_code>iterations = 3

## preparing array for test statistic values
samp_dis = np.zeros(iterations)

for i in range(iterations):
    ## create a list of 100 random numbers
    array = rng.integers(low=1, high=100, size=100)

    ## subset of random numbers that are divisible by 3
    subset = array[array%3==0]
    
    ## test statistic: number of random numbers generated that are not divisible by 3
    test_stat = array.size - subset.size
    samp_dis[i] = test_stat

## for 100 random integers from 1-100, out of 3 experiements, how many numbers on average are not divisible by 3
samp_dis.mean()<jupyter_output><empty_output><jupyter_text>Activity 2This section will ensure that you have understood the various basic data structures and their manipulation. We will do that by going through an activity that has been designed specifically for this purpose. In this activity, we will do the following: - Get multiline text and save it in a Python variable - Get rid of all new lines in it using string methods - Get all the unique words and their occurrences from the string  - Repeat the step to find all unique words and occurrences, without considering case sensitivity<jupyter_code>multiline_text = """It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife.

However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered as the rightful property of some one or other of their daughters.

“My dear Mr. Bennet,” said his lady to him one day, “have you heard that Netherfield Park is let at last?”

Mr. Bennet replied that he had not.

“But it is,” returned she; “for Mrs. Long has just been here, and she told me all about it.”

Mr. Bennet made no answer.

“Do not you want to know who has taken it?” cried his wife impatiently.

“You want to tell me, and I have no objection to hearing it.”

This was invitation enough.

“Why, my dear, you must know, <NAME> says that Netherfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise and four to see the place, and was so much delighted with it that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and some of his servants are to be in the house by the end of next week.”

“What is his name?”

“Bingley.”

“Is he married or single?”

“Oh! single, my dear, to be sure! A single man of large fortune; four or five thousand a year. What a fine thing for our girls!”

“How so? how can it affect them?”

“My dear <NAME>,” replied his wife, “how can you be so tiresome! You must know that I am thinking of his marrying one of them.”

“Is that his design in settling here?”

“Design! nonsense, how can you talk so! But it is very likely that he may fall in love with one of them, and therefore you must visit him as soon as he comes.”

“I see no occasion for that. You and the girls may go, or you may send them by themselves, which perhaps will be still better, for as you are as handsome as any of them, Mr. Bingley might like you the best of the party.”

“My dear, you flatter me. I certainly have had my share of beauty, but I do not pretend to be anything extraordinary now. When a woman has five grown-up daughters, she ought to give over thinking of her own beauty.”

“In such cases, a woman has not often much beauty to think of.”

“But, my dear, you must indeed go and see <NAME> when he comes into the neighbourhood.”

“It is more than I engage for, I assure you.”

“But consider your daughters. Only think what an establishment it would be for one of them. Sir William and <NAME> are determined to go, merely on that account, for in general, you know, they visit no newcomers. Indeed you must go, for it will be impossible for us to visit him, if you do not.”

“You are over scrupulous, surely. I dare say <NAME> will be very glad to see you; and I will send a few lines by you to assure him of my hearty consent to his marrying whichever he chooses of the girls; though I must throw in a good word for my little Lizzy.”

“I desire you will do no such thing. Lizzy is not a bit better than the others; and I am sure she is not half so handsome as Jane, nor half so good-humoured as Lydia. But you are always giving her the preference.”

“They have none of them much to recommend them,” replied he; “they are all silly and ignorant like other girls; but Lizzy has something more of quickness than her sisters.”

“Mr. Bennet, how can you abuse your own children in such a way? You take delight in vexing me. You have no compassion on my poor nerves.”

“You mistake me, my dear. I have a high respect for your nerves. They are my old friends. I have heard you mention them with consideration these twenty years at least.”

“Ah, you do not know what I suffer.”

“But I hope you will get over it, and live to see many young men of four thousand a year come into the neighbourhood.”

“It will be no use to us, if twenty such should come, since you will not visit them.”

“Depend upon it, my dear, that when there are twenty, I will visit them all.”

Mr. Bennet was so odd a mixture of quick parts, sarcastic humour, reserve, and caprice, that the experience of three-and-twenty years had been insufficient to make his wife understand his character. Her mind was less difficult to develop. She was a woman of mean understanding, little information, and uncertain temper. When she was discontented, she fancied herself nervous. The business of her life was to get her daughters married; its solace was visiting and news."""
type(multiline_text)
len(multiline_text)
multiline_text = multiline_text.replace("\n", "")
multiline_text
## remove all non string/number characters from the string

cleaned = ""
for char in multiline_text:
    if char == " ":
        cleaned += char
    elif char.isalnum():
        cleaned += char
    else:
        cleaned += " "

## create list from split string
clean_list = cleaned.split() # specifying no delimiter means to split by white space

len(clean_list) # list of words
## create dictionary for unique values and frequency
unique_dict = {}

## iterate clean list and add lower-case values to unique dict counting occurences 
for char in clean_list:
    char = char.lower()
    if char in unique_dict.keys():
        unique_dict[char] += 1
    else:
        unique_dict[char] = 1

unique_df = pd.DataFrame(data=unique_dict.values(), index=unique_dict.keys(), columns=['Frequency'])  ## create DataFrame from dict

unique_df.index.name = 'Word'  ## rename index

unique_df = unique_df.sort_values(by=['Frequency'], ascending=False)  ## sort descending by frequency

## top 20 words
print("Top 20 Words\n")
print(unique_df.head(20))<jupyter_output>Top 20 Words

      Frequency
Word           
you          31
of           29
to           22
a            21
the          18
i            17
and          17
that         15
is           14
it           14
my           13
in           12
them         11
for          11
be           11
his          11
he           11
mr           10
will          9
not           9<jupyter_text>Activity 3In this activity, we will be using permutations to generate all possible three-digit numbers that can be generated using 0, 1, and 2. Then, loop over this iterator, and also use isinstance and assert to make sure that the return types are tuples. Also, use a single line of code involving dropwhile and lambda expressions to convert all the tuples to lists while dropping any leading zeros (for example, (0, 1, 2) becomes [1, 2]). Finally, write a function that takes a list like before and returns the actual number contained in it.<jupyter_code>## generating permutation iterator with itertools.permutations()
p_iterator = permutations([0,1,2])

## loop through printing tuple and asserting if value is tuple
for permute in p_iterator:
    print(permute)
    assert isinstance(permute, tuple)

## loop through permutation iterator again but drop leading zero using lambda function
## generating permutation iterator with itertools.permutations()
p_iterator = permutations([0,1,2])

## loop through printing tuple and asserting if value is tuple
for permute in p_iterator:
    print(list(dropwhile(lambda x: x == 0, permute)))<jupyter_output>[1, 2]
[2, 1]
[1, 0, 2]
[1, 2, 0]
[2, 0, 1]
[2, 1, 0]<jupyter_text>Activity 4A CSV file is something you will encounter a lot in your life as a data practitioner. A CSV is a comma-separated file where data from a tabular format is generally stored and separated using commas, although other characters can also be used. In this activity, we will be tasked with building our own CSV reader and parser. Although it is a big task if we try to cover all use cases and edge cases, along with escape characters and all, for the sake of this small activity, we will keep our requirements small. We will assume that there is no escape character, meaning that if you use a comma at any place in your row, it means you are starting a new column. We will also assume that the only function we are interested in is to be able to read a CSV file line by line where each read will generate a new dict with the column names as keys and row names as values.<jupyter_code>def return_dict_from_csv_line(header, line):
    zipped_line = zip_longest(header, line, fillvalue=None)
    ret_dict = {kv[0]:kv[1] for kv in zipped_line}
    return ret_dict
    


with open('sales_record.csv', "r") as fd:
    fl = fd.readline()
    header = fl.replace("\n","").split(",")
    for i, line in enumerate(fd):
        line = line.replace("\n", "").split(",")
        d = return_dict_from_csv_line(header, line)
        print(d)
        if i > 10:
            break<jupyter_output>{'Region': 'Central America and the Caribbean', 'Country': 'Antigua and Barbuda ', 'Item Type': 'Baby Food', 'Sales Channel': 'Online', 'Order Priority': 'M', 'Order Date': '12/20/2013', 'Order ID': '957081544', 'Ship Date': '1/11/2014', 'Units Sold': '552', 'Unit Price': '255.28', 'Unit Cost': '159.42', 'Total Revenue': '140914.56', 'Total Cost': '87999.84', 'Total Profit': '52914.72'}
{'Region': 'Central America and the Caribbean', 'Country': 'Panama', 'Item Type': 'Snacks', 'Sales Channel': 'Offline', 'Order Priority': 'C', 'Order Date': '7/5/2010', 'Order ID': '301644504', 'Ship Date': '7/26/2010', 'Units Sold': '2167', 'Unit Price': '152.58', 'Unit Cost': '97.44', 'Total Revenue': '330640.86', 'Total Cost': '211152.48', 'Total Profit': '119488.38'}
{'Region': 'Europe', 'Country': 'Czech Republic', 'Item Type': 'Beverages', 'Sales Channel': 'Offline', 'Order Priority': 'C', 'Order Date': '9/12/2011', 'Order ID': '478051030', 'Ship Date': '9/29/2011', 'Units Sold': '4778', 'Unit Pri[...]
{'fraction_non_alphanumeric': 0.08880278961642774, 'fraction_numerical': 0.024176675707090273, 'mean_word_length': 4.04140625, 'pattern_counts': {'":': 1, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '243032', 'license': 'MIT', 'n_tokens_mistral': 3951, 'n_tokens_neox': 3652, 'n_words': 1821, 'path': 'DSC 540 - Data Preparation/Week 1 & 2.ipynb', 'repo_name': 'Hakuna-Patata/BU_MSDS_PTW', 'size': '12939'}