<jupyter_start><jupyter_text>CalculationWealth factor is defined as: $ \large  cost\_of\_living = \sum_{i=1}^n (a_i * weight)$Meaning of results:* $ a_i $ - the price of a product with a corresponding type (in USD)* $ weight $ - the quantity of a product in contrary to yearly usage<jupyter_code>cost_of_living_weights = {
    '1 Pair of Jeans (Levis 501 Or Similar)': 0.35,
    '1 Summer Dress in a Chain Store (Zara, H&M, ...)': 0.35,
    '1 Pair of Nike Running Shoes (Mid-Range)': 0.35,
    '1 Pair of Men Leather Business Shoes': 0.35,
    'Milk (regular), (1 liter)': 25,
    'Loaf of Fresh White Bread (500g)': 31,
    'Rice (white), (1kg)': 14,
    'Eggs (regular) (12)': 20,
    'Local Cheese (1kg)': 12,
    'Chicken Fillets (1kg)': 15,
    'Beef Round (1kg) (or Equivalent Back Leg Red Meat)': 15,
    'Apples (1kg)': 31,
    'Banana (1kg)': 25,
    'Oranges (1kg)': 30,
    'Tomato (1kg)': 22,
    'Potato (1kg)': 24,
    'Onion (1kg)': 10,
    'Lettuce (1 head)': 18,
    'Water (1.5 liter bottle)': 30,
    'Bottle of Wine (Mid-Range)': 4,
    'Domestic Beer (0.5 liter bottle)': 6,
    'Imported Beer (0.33 liter bottle)': 6,
    'Cigarettes 20 Pack (Marlboro)': 15,
    'Apartment (1 bedroom) in City Centre': 0,
    'Apartment (1 bedroom) Outside of Centre': 0,
    'Apartment (3 bedrooms) in City Centre': 0,
    'Apartment (3 bedrooms) Outside of Centre': 0,
    'Meal, Inexpensive Restaurant': 16,
    'Meal for 2 People, Mid-range Restaurant, Three-course': 3.5,
    'McMeal at McDonalds (or Equivalent Combo Meal)': 6,
    'Domestic Beer (0.5 liter draught)': 5,
    'Imported Beer (0.33 liter bottle)': 5,
    'Cappuccino (regular)': 15,
    'Coke/Pepsi (0.33 liter bottle)': 6,
    'Water (0.33 liter bottle)': 6,
    'Fitness Club, Monthly Fee for 1 Adult': 2.3,
    'Tennis Court Rent (1 Hour on Weekend)': 3,
    'Cinema, International Release, 1 Seat': 6,
    'One-way Ticket (Local Transport)': 20,
    'Monthly Pass (Regular Price)': 1.5,
    'Taxi Start (Normal Tariff)': 5,
    'Taxi 1km (Normal Tariff)': 20,
    'Taxi 1hour Waiting (Normal Tariff)': 0.7,
    'Gasoline (1 liter)': 60,
    'Volkswagen Golf': 0.0035,
    'Toyota Corolla Sedan 1.6l 97kW Comfort (Or Equivalent New Car)': 0.0035,
    'Basic (Electricity, Heating, Cooling, Water, Garbage) for 85m2 Apartment': 1,
    '1 min. of Prepaid Mobile Tariff Local (No Discounts or Plans)': 320,
    'Internet (60 Mbps or More, Unlimited Data, Cable/ADSL)': 1
}<jupyter_output><empty_output><jupyter_text>Data preparation Gather raw data<jupyter_code>import os
from pandas import concat as pd_concat
from pandas import to_numeric
from pandas import read_csv


years = [str(year) for year in range(2010, 2021)]
results_by_year = dict(zip(years, [[] for i in range(len(years))]))



directory = "web_scraper/every_item_by_country_in_usd"
    
for file in os.listdir(directory):
    file_path = directory + '/' + file
    if file_path.endswith('.csv'):
        results_by_year[file[8:12]].append(read_csv(file_path))
        results_by_year[file[8:12]][-1].drop(['Unnamed: 0', 'Rank'], axis=1, inplace=True)

for key in results_by_year:
    results_by_year[key] = pd_concat(results_by_year[key], axis=1, join='outer')
    results_by_year[key] = results_by_year[key].loc[:,~results_by_year[key].columns.duplicated()]
    results_by_year[key][list(cost_of_living_weights.keys())] = results_by_year[key][list(cost_of_living_weights.keys())].apply(to_numeric, errors='coerce', axis=1)
    results_by_year[key]['Cost of living'] = results_by_year[key][list(cost_of_living_weights.keys())].mul(list(cost_of_living_weights.values())).sum(1)

for key in results_by_year:
    results_by_year[key] = results_by_year[key][['City', 'Cost of living']]
    results_by_year[key].rename(columns = {'City': 'Country', 'Cost of living': key}, inplace = True)
all_countries = set()
for key in results_by_year:
    all_countries = all_countries.union(set(results_by_year[key]['Country']))

all_countries = list(all_countries)
all_countries.sort()
import pandas as pd
import numpy as np

df = pd.DataFrame(all_countries, columns =['Country'])

for key in results_by_year:
    df = pd.merge(left=df, right=results_by_year[key], how='left', left_on=['Country'], right_on = ['Country'])

df<jupyter_output><empty_output><jupyter_text>Remove all countries that contains more than one missing dataWe have decided that we will use approximation only for those countries that has no more than one missing data.It is due to a possibility that while approximating more than one missing data we may distort the analysis.<jupyter_code>for index, row in df.iterrows():
    if row.isnull().sum() > 1 or np.isnan(row['2010']) or np.isnan(row['2020']):
        df.drop(index, inplace=True)

df<jupyter_output><empty_output><jupyter_text>Fill missing data<jupyter_code>df[~df[[str(x) for x in range(2010, 2021)]].notnull().all(1)]
df.at[56, '2011'] = (df.at[56, '2010']+df.at[56, '2012'])/2
df.at[82, '2012'] = (df.at[82, '2011']+df.at[82, '2013'])/2
df[(df['Country'].str.contains(fr'Kazakhstan|Panama', regex=True, case=False))]
df[~df[[str(x) for x in range(2010, 2021)]].notnull().all(1)]<jupyter_output><empty_output><jupyter_text>Save data to a file<jupyter_code># filename = f'./dataset/cost_of_living/cost_of_living_after_data_wrangling.csv'
# with open(filename, 'w') as file:
#     df.to_csv(file, index=False, line_terminator='\n')<jupyter_output><empty_output><jupyter_text>Analysis<jupyter_code>df.set_index('Country')
import matplotlib.pyplot as plt
df_countries = df.set_index('Country')
sort_by_mean = df_countries.median(axis=1).sort_values()
fig, ax = plt.subplots(figsize=(20,df_countries.shape[0]//2))
df_countries.reindex(sort_by_mean.index, axis=0).T.boxplot(vert=False,ax=ax)
plt.title("Cost of living for Countries",fontsize=30)
pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1514736120630569, 'fraction_numerical': 0.0430089102124743, 'mean_word_length': 4.860441767068273, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '356281', 'license': 'MIT', 'n_tokens_mistral': 2370, 'n_tokens_neox': 2139, 'n_words': 567, 'path': 'cost_of_living_calculation.ipynb', 'repo_name': 'Vergenter/familyWealth', 'size': '5836'}