<jupyter_start><jupyter_text>20-08-13: Daily Practice ------ Practices  Daily* [x] [Job Search](2-Hour-Job-Search)  * [ ] 2HJS progress  * [x] Network  * [x] Social media* [ ] [Meta-data: read & write](Meta-data:-reading-&-writing)* [x] [Practice & learn](Practice-&-learn)  * [ ] Coding, algorithms & data structures  * [x] Brain teasers  * [ ] Interviewing, STAR storytelling  * [x] Data science: access, manipulation, analysis, visualization  * [x] Machine learning: Scikit-learn, TensorFlow, PyTorch  * [ ] Engineering: SQL, PySpark, APIs, TDD, OOP  WeeklyVim command(s) I'll get familiar with this week:* `:s/find/replace/flags`* `:%s` - entire file  * `:%s/find/replace/g`  * `:%s/find/replace/gc` (confirm before replacing)* `:s` - current line only  * `:s/find/replace/gc`* Specific range of lines  * `:36,42s/find/replace/g` ------ 2-Hour Job Search ------ Meta-data: read & write ------ Practice & learn --- Coding, algorithms & data structures ---> A box has 12 red cards and 12 black cards. Another box has 24 red cards and 24 black cards. You want to draw two cards at random from one of the two boxes, one card at a time. Which box has a higher probability of getting cards of the same color and why? The 24x24 box has a higher probability of taking out two cards of the same color in a row.In the 12x12 deck, when one is taken out, the resulting probability of taking one out of the same color is:$\frac{11}{11+12} = \frac{11}{23} = 0.478$And in the 24x24 deck:$\frac{23}{23+24} = \frac{23}{47} = 0.489$ ---> You are at a casino and have two dice to play with. You win \$10 every time you roll a 5. If you play till you win and then stop, what is the expected payout?Assume the cost of playing is \$2 per game.<jupyter_code># Find total combinations and number that equal five
total = 0
fives = 0

for d1 in range(1, 7):
    for d2 in range(1, 7):
        total += 1
        if d1 + d2 == 5:
            fives += 1
            
print(total)
print(fives)
print(f"Probability of rolling 5: {fives//4}/{total//4}")
# Calculate expected payout
cost = -2
payout = 10

exp_payout = ((fives//4) * payout) + ((total//4) * cost)
print(exp_payout)<jupyter_output>-8<jupyter_text>---> How can you tell if a given coin is biased? A simple hypothesis test can be run to determine if a coin is biased. A coin toss is a binomial distribution, and a fair coin should see both sides come up with equal frequency.The null hypothesis is that the coin is not biased, and therefore the probability of flipping heads is 0.5. The alternative hypothesis is that the coin is biased, and therefore has a probability of flipping heads **not** equal to 0.5.To test this, the given coin could be tossed 1,000 times. The data from this experiment (which side comes up each time) can be averaged to give the approximate probability of each side landing up.Calculate the z-score of the test, and compare the p-value against the alpha (two-sample t-test - alpha = 0.025). If p-value is less than alpha, then reject the null hypothesis and conclude that the coin is biased. Otherwise, accept the null that the coin is not biased. ---> Make an unfair coin fair. A coin toss has a binary result. An unfair coin can be made fair by flipping it twice and betting on both sides coming up.$P(heads) \times P(tails) = P(tails) \times P(heads)$ ---> You are about to get on a plane to London, you want to know whether you have to bring an umbrella or not. You randomly call three of your friends and ask each one of them if it's raining. The probability that your friend is telling the truth is 2/3 and the probability that they are playing a prank on you by lying is 1/3.> If all 3 of them tell you that it is raining, then what is the probability that it is actually raining in London? Assume that the probability of it raining in London on any given day is 25% or 0.25.* P(A) = proba of rain = 0.25* P(B) = proba that all 3 friends say it's raining* P(A|B) = proba that it's raining given they say it's raining* P(B|A) = proba that all 3 say it's raining given it is actually raining = (2/3)^3 = 8/27Step 1: Solve for P(B)  $P(A|B) = \frac{P(B|A) \times P(A)}{P(B)}$  $P(B) = (P(B|A) \times P(A)) + (P(B\neg A) \times P(\neg A))$  $P(B) = (\frac{2}{3}^3 \times 0.25) + (\frac{1}{3}^3 \times 0.75) = (0.25 \times \frac{8}{27}) + (0.75 \times \frac{1}{27})$  Step 2: Solve for P(A|B)$P(A|B) = \frac{0.25 \times \frac{8}{27}}{(0.25 \times \frac{8}{27}) + (0.75 \times \frac{1}{27})}$$P(A|B) = \frac{8}{8 + 3} = \frac{8}{11}$ ---You are given 40 cards with four different colors: 10 green, 10 red, 10 blue, and 10 yellow. The cards of each color are numbered from one to ten. Two cards are picked at random. Find the probability that the cards picked do not have the same number and same color.<jupyter_code># Set up a dataframe to visualize this
import pandas as pd
import numpy as np

cols = ["green", "red", "blue", "yellow"]
data = [list(range(1, 11)) for c in range(4)]
data_dict = {i:v for (i,v) in zip(cols, data)}

df = pd.DataFrame(data=data_dict, columns=cols)
df
# For example, let's pick green 10 first
diff = 40 - 13
proba_diff = diff / 40
proba_diff<jupyter_output><empty_output><jupyter_text>Nope! Apparently that's not how it should be solved. Since these events are not independent, we can use the rule:$P(A \cap B) = P(A) * P(B|A)$which is also equal to$P(\neg A \cap \neg B) = P(\neg A) * P(\neg B | \neg A)$For example:$P(\neg 4 \cap \neg yellow) = P(\neg 4) * P(\neg yellow | \neg 4)$$P(\neg 4 \cap \neg yellow) = \frac{36}{39} * \frac{27}{36}$$P(\neg 4 \cap \neg yellow) = 0.692$ --- Interview questions  DS Case StudyYou are provided with a dataset of houses and their respective characteristics in a location. Characteristics include number of rooms, bathrooms, floor, age, etc. Anything that could appear on a property site is part of your features. Your aim is to build a model to predict the property price.Please share how you would:Frame the problem (e.g., metric, problem statement)  * Supervised: the target is the property price  * Regression: property price is a continuous variable  * Eval metrics:    * Mean absolute error    * RMSE    * ...Prepare the data      COST      "$100"      "10.00"      "10.20"      "USD1000"      "100SGD"      "1000USD"      "100,000.00"Explore the dataCreate a train-validation split* 1MM records* 2015 - 2019  Assess a handful of models* *LR*, DT/RF, NNFor the machine learning technique you mentioned, please explain to me how it works like I'm a five year old.      ---------      ---------      ----X----      ---XXX---      -XX--XX--      XX----XX-      X------XXDecide on a model for productionNow that we have a working model, we want to take the next step of *purchasing* these homes, refurnishing them, and then selling them. This is the Zillow Offers product. Given the high risk of purchasing an overpriced home, how would you design a process / framework to ensure we do not purchase overpriced homes?<jupyter_code># You're part of a warehousing team running simulations on 
# robots performing the pick-and-pack task. Another team has 
# completed the simulations and provided a log of the robot routes. 
# Your task is to validate if the routes are valid. 

# Heres' how a route looks like:

# [
# [1, 0, 0, 0, 0],
# [1, 0, 0, 0, 0],
# [0, 1, 0, 0, 0],
# [0, 0, 1, 0, 0],
# ]

# [
# [0, 0, 0, 0, 1],
# [0, 0, 0, 1, 0],
# [0, 0, 1, 0, 0],
# [0, 0, 0, 1, 0],
# [0, 0, 0, 0, 1],
# ]

# In the above, the warehouse has 4 floors (rows) and 5 shelves per floor (columns). The robot:
# - Inits at shelf 1 at floor 4
# - Moves to shelf 1 at floor 3
# - Moves to shelf 2 at floor 2
# - Moves to shelf 3 at floor 1

# Some constraints
# - A robot can only, and must, appear on each floor once
# - A robot can only move vertically (e.g., 1st shelf to 1st shelf on next floor), 
# or diagonally (e.g., 3rd shelf to 2nd or 4th shelf on next floor)

# Please develop the following function to validate routes, which will return a boolean.

# Start of code

from typing import List


def validate(route: List[List]) -> bool:
    # Catch case of empty route
    if len(route) < 1:
        return False
    
    # keep track of previous shelf #
    prev_shelf = None
    
    # Iterate through outer list
    for floor in route:
        # 1. One and only one `1` in each row
        # If sum != 1, invalid
        if sum(floor) != 1:
            return False
      
        cur_shelf = floor.index(1)
      
        # 2. Moving: i -/+ 1
        if prev_shelf is not None:
            # Compare current shelf to prev
            # if cur_shelf (index) differs by more than 1, invalid
            if abs(cur_shelf - prev_shelf) > 1:
                return False
        
        # Set previous shelf to current shelf
        prev_shelf = cur_shelf
    
    # If iteration completes, route is valid
    return True

    # print('Not implemented yet')
    # raise NotImplementedError
    
if __name__ == '__main__':
    # test_route = [
    #     [1, 0, 0],
    #     [1, 0, 0],
    #     [1, 0, 0]
    # ]
    
    # print(validate(test_route))
    
    # test_route_2 = [
    #   [0, 0, 0, 0, 1],
    #   [0, 0, 0, 1, 0],
    #   [0, 0, 1, 0, 0],
    #   [0, 0, 0, 1, 0],
    #   [0, 0, 0, 0, 1],
    # ]
    
    # assert validate(test_route_2) == True
    
    # test_route_3 = [
    #   [0, 0, 0, 0, 1],
    #   [0, 0, 0, 1, 0],
    #   [0, 0, 1, 0, 0],
    #   [1, 0, 0, 0, 0],
    #   [0, 0, 0, 0, 1],
    # ]
    
    # assert validate(test_route_3) == False
    
    # test_route_4 = [
    #   [0, 1, 0, 1],
    #   [0, 0, 1, 0],
    #   [0, 1, 0, 0],
    #   [1, 0, 0, 0],
    #   [0, 0, 0, 1],
    # ]
    
    # assert validate(test_route_4) == False
    
    # test_route = [
    #     [1, 0, 0, 0],
    #     [0, 1, 0, 0],
    #     [0, 0, 1, 0],
    #     [0, 0, 1, 0],
    #     [0, 0, 0, 1]
    # ]
    
    # result = validate(test_route)  # True
    # print('Valid route: {} \n'.format(result))
    
    # test_route = [
    #     [1],
    #     [1],
    #     [1],
    #     [1],
    #     [1]
    # ]
    
    # result = validate(test_route)  #True
    # print('Valid route: {} \n'.format(result))
    
    test_route = [
    ]
    
    result = validate(test_route)  # False
    print('Valid route: {} \n'.format(result))

    test_route = [
        [0, 0, 1],
        [0, 1, 0],
        [0, 0, 0],
    ]
    
    result = validate(test_route)  # False
    print('Valid route: {} \n'.format(result))
    
    test_route = [
        [1, 0, 1],
        [0, 1, 0],
        [0, 0, 0],
    ]

    result = validate(test_route)  # False
    print('Valid route: {} \n'.format(result))<jupyter_output><empty_output><jupyter_text>--- Data science --- Machine learning  Polynomial Regression: Office Prices The ProblemCharlie wants to purchase office-space. He does a detailed survey of the offices and corporate complexes in the area, and tries to quantify a lot of factors, such as the distance of the offices from residential and other commercial areas, schools and workplaces; the reputation of the construction companies and builders involved in constructing the apartments; the distance of the offices from highways, freeways and important roads; the facilities around the office space and so on.Each of these factors are quantified, normalized and mapped to values on a scale of 0 to 1. Charlie then makes a table. Each row in the table corresponds to Charlie's observations for a particular house. If Charlie has observed and noted `F` features, the row contains `F` values separated by a single space, followed by the office-space price in dollars/square-foot. If Charlie makes observations for `H` houses, his observation table has `(F+1)` columns and `H` rows, and a total of `(F+1) * H` entries.Charlie does several such surveys and provides you with the tabulated data. At the end of these tables are some rows which have just `F` columns (the price per square foot is missing). Your task is to predict these prices. `F` can be any integer number between 1 and 5, both inclusive.There is one important observation which Charlie has made.* The prices per square foot, are (approximately) a polynomial function of the features in the observation table.  * This polynomial always has an order less than 4<jupyter_code># === Imports === #
import pandas as pd

from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
# === Set up the data === #
datapath = "assets/data/office_space_price.txt"
# No column names given - trying to predict the third column
# Training set is all of it except last 4 rows
offices = pd.read_csv(datapath, sep=" ", header=None)
print(offices.shape)
offices.head()
# Last 4 lines are test set to submit
offices.tail()
# === Split into train and test === #
offices_train = offices.loc[:99].copy()
print("Train shape:", offices_train.shape)

offices_test = offices.loc[100:].copy()
print("Test shape:", offices_test.shape)
offices_test.head()
# === Arrange into features and target === #
X_train = offices_train.drop(columns=[2])
y_train = offices_train[2]

X_test = offices_test.drop(columns=[2])
y_test = offices_test[2]

(X_train.shape, y_train.shape), (X_test.shape, y_test.shape)<jupyter_output><empty_output><jupyter_text>ValidationI'm not going to split into train/val. The dataset is very small, therefore I'm going to use k-fold cross-validation.<jupyter_code># === Fit polynomial pipeline === #

# Instantiate pipeline components
poly = PolynomialFeatures(degree=3)
model = LinearRegression()

# Create pipeline
pipe = Pipeline([
    ("poly", poly),
    ("model", model),
])

# Fit pipeline to training data
pipe.fit(X_train, y_train)
# === Get validation scores === #
k = 3
scores = cross_val_score(pipe, X_train, y_train, cv=k, scoring="neg_mean_absolute_error")
print(f"MAE for {k} folds: %0.2f (+/- %0.2f)" % (-scores.mean(), -scores.std() * 2))
# === Predict on test set === #
pred = pipe.predict(X_test)
pred<jupyter_output><empty_output><jupyter_text>Adapting for HackerRank PlatformThe input is the raw text directly from stdin. So I'm adding this bit to clean and load it.<jupyter_code># === Read from raw file === #

with open("assets/data/office_space_raw.txt") as f:
    # Exclude first line
    raw_data = f.read().splitlines()[1:]
# === Split up train/test by slicing === #
train_list = raw_data[:100]
test_list = raw_data[101:]

# === Split up rows into nested lists === #
train_lists = [r.split() for r in train_list]
test_lists = [r.split() for r in test_list]

# === Load train/test into dataframes === #
train = pd.DataFrame(data=train_lists)
test = pd.DataFrame(data=test_lists)

train.shape, test.shape
# === Arrange into features and target === #
X_train = train.drop(columns=[2])
y_train = train[2]

X_test = test

X_train.shape, y_train.shape, X_test.shape
# === Fit polynomial pipeline === #

# Instantiate pipeline components
poly = PolynomialFeatures(degree=3)
model = LinearRegression()

# Create pipeline
pipe = Pipeline([
    ("poly", poly),
    ("model", model),
])

# Fit pipeline to training data
pipe.fit(X_train, y_train)

# === Get validation scores === #
k = 3
scores = cross_val_score(pipe, X_train, y_train, cv=k, scoring="neg_mean_absolute_error")
print(f"MAE for {k} folds: %0.2f (+/- %0.2f)" % (-scores.mean(), -scores.std() * 2))
# === Predict on test set === #
pred = pipe.predict(X_test)

# Print to stdout
for p in pred:
    print(round(p, 4))<jupyter_output>180.3768
1312.0651
440.1293
343.7154<jupyter_text>Success!Here's the full code from the HackerRank editor:<jupyter_code>import sys
import pandas as pd

from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

# === Read the data in from stdin === #
raw_data = sys.stdin.read().splitlines()[1:]

# === Split up train/test by slicing === #
train_list = raw_data[:100]
test_list = raw_data[101:]

# === Split up rows into nested lists === #
train_lists = [r.split() for r in train_list]
test_lists = [r.split() for r in test_list]

# === Load train/test into dataframes === #
train = pd.DataFrame(data=train_lists)
test = pd.DataFrame(data=test_lists)

# === Arrange into features and target === #
X_train = train.drop(columns=[2])
y_train = train[2]
X_test = test

# === Fit polynomial pipeline === #

# Instantiate pipeline components
poly = PolynomialFeatures(degree=3)
model = LinearRegression()

# Create pipeline
pipe = Pipeline([
    ("poly", poly),
    ("model", model),
])

# Fit pipeline to training data
pipe.fit(X_train, y_train)

# === Predict on test set === #
pred = pipe.predict(X_test)

# === Print to stdout === #
for p in pred:
    print(round(p, 4))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1250591856060606, 'fraction_numerical': 0.03125, 'mean_word_length': 3.294027954256671, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '202391', 'license': 'MIT', 'n_tokens_mistral': 5876, 'n_tokens_neox': 5385, 'n_words': 2182, 'path': 'ds/practice/daily_practice/20-08/20-08-13-226-thu.ipynb', 'repo_name': 'tobias-fyi/vela', 'size': '16896'}