<jupyter_start><jupyter_text>First, I drop the name,email, timestamp, columns because they are not necessary for the analysis.<jupyter_code>data = data.drop(labels = ["Name", "Email", "Timestamp", "Store_Name"], axis=1)
data.shape
# data.head()<jupyter_output><empty_output><jupyter_text>Data Formatting  Normalizing the prices First, I need to bring the prices to the same quantity - 1 kilogram. Some of the quantities have 10,000 number - indicating that this value is invalid. because the price for that qunatity is N/A, the prices will not be affected. Therefore, no need to correct thise quantities. N/A valueds will be dropped. Then, I need to convert the prices into one currency - American dollar -  so we can compare the prices of different countries.<jupyter_code>###bringing all the prices to the price per 1 kilo

for i in range(5, 65, 2): #for each of the produce price
    data[data.columns[i]] /= data[data.columns[i-1]]
    # data = data.drop(data.columns[i-1], axis = 1)
# data.head()


#dropping the quantities
columns_to_drop = [i for i in range(4, 64, 2)]
data_normalized = data.drop(data.columns[columns_to_drop], axis = 1)


##bringing the prices to the same currency - USD

data_normalized.Currency.unique()

# creating a list of conversion value to dollar for each of the google from response
exchange_rate = []
for cur in data_normalized.Currency:

  if cur == "USD":
    exchange_rate.append(1)

  elif cur == "SEK":
    exchange_rate.append(0.12)
  elif cur ==  'UAH':
    exchange_rate.append(0.036)

  elif cur == "KES":
    exchange_rate.append(0.0091)

  elif cur == "BRL":
    exchange_rate.append(0.17)

  elif cur == "BDT":
    exchange_rate.append(0.012)
data_normalized['Currency_Converter'] = exchange_rate

data_normalized.shape
# data_normalized.head()
#multiplying the prices by the conversion parameters so that all the prices are in dollars

for i in range(4, 34): #for each of the produce price
    data_normalized[data_normalized.columns[i]] *= data_normalized[data_normalized.columns[34]]<jupyter_output><empty_output><jupyter_text>Now the prices are unified across the weight and across the currencies, therefroe, they can be comparable.   Countries FormattingIn this section, I am standardizing and replacing the countries value to contain standardized names that later can be classified and put into dictionary.<jupyter_code>#seeing the current countries listed
data_normalized["Country"].unique()
#replacing with a more unified notation
data_normalized.Country = data_normalized.Country.replace(["United States", "US", 'USA (San Francisco)',
                                                           'United States of America', 'Dallas, TX, USA',
                                                           'San Francisco, USA', 'San Francisco', 'United States ', 'San Francisco, United States'], 
                                                           ["USA", "USA", "USA","USA","USA","USA","USA", "USA", 'USA']
 )
data_normalized.Country.unique()<jupyter_output><empty_output><jupyter_text>Store Type FormatingIn this section, I am making the types of the stores (budget, mid-range, luxury) consistent so that the three parameters will have three multipliers.<jupyter_code>#previewing the raw names for the types of stores
print('Initial store classification', data_normalized.Store_Perception.unique())


#replacing with a unified notation
data_normalized.Store_Perception = data_normalized.Store_Perception.replace('Small store in expensive neighborhood (high prices)','Luxury (expensive)' )

print('Formated store classification', data_normalized.Store_Perception.unique())<jupyter_output>Initial store classification ['Mid-range' 'Luxury (expensive)' 'Budget (cheap)'
 'Small store in expensive neighborhood (high prices)']
Formated store classification ['Mid-range' 'Luxury (expensive)' 'Budget (cheap)']<jupyter_text>Forming a dictionary Here, I first drop the columns that we no longer need and leaving only the prices, store type, and country. Then, I am using the pandas melt function that will help me to create a dictionry of a format, suitable for the stan model. I am leaving only the store type, the country, and the prices as the columns in the pandas object.<jupyter_code>data_normalized.head()


#dropping the irrelevant columns from the df
final_data =data_normalized.drop(labels = ["Currency", "Currency_Converter"], axis=1)


#getting the value_vars parameter for the melt function to melt them into one column
value_var = ['Apple1_p', 'Apple2_p', 'Apple3_p',
         'Banana1_p', 'Banana2_p', 'Banana3_p',
         'Tomatoes1_p', 'Tomatoes2_p', 'Tomatoes3_p',
         'Potato1_p', 'Potato2_p', 'Potato3_p',
         'Flour1_p', 'Flour2_p', 'Flour3_p',
         'Rice1_p', 'Rice2_p', 'Rice3_p',
         'Milk1_p', 'Milk2_p', 'Milk3_p',
         'Butter1_p', 'Butter2_p', 'Butter3_p',
         'Eggs1_p', 'Eggs2_p', 'Eggs3_p',
         'Chicken1_p', 'Chicken2_p', 'Chicken3_p']


#melting the df on the price
final_data_input = pd.melt(final_data, id_vars=['Country', 'Store_Perception'],
                     value_vars=value_var,
                     value_name = 'Price', var_name = 'Product')

print(final_data_input.head())
final_data_input.shape<jupyter_output>Country    Store_Perception   Product     Price
0     USA           Mid-range  Apple1_p  3.445996
1     USA  Luxury (expensive)  Apple1_p  5.484581
2     USA      Budget (cheap)  Apple1_p  4.383260
3     USA  Luxury (expensive)  Apple1_p  9.000000
4  Sweden      Budget (cheap)  Apple1_p  3.480000<jupyter_text>Now I need to change the values of the product to contain 10 keywords for each of the product type.<jupyter_code>#stripping the endings of the Product column so it only contains 10 types of vegetables
final_data_input["Product"] = final_data_input["Product"].apply(lambda x: re.sub(r'[0-9]{1}_p', '',x))

print(final_data_input.head())

#saving the names of the countries, products, and countries to plot the distributions later
products_names = final_data_input.Product.unique()
country_names = final_data_input.Country.unique()
store_names = final_data_input.Store_Perception.unique()
#replacing the strings of stores and countries with the indexes to insert into Stan model
final_data_input.Store_Perception = final_data_input.Store_Perception.replace(
    ['Mid-range', 'Luxury (expensive)', 'Budget (cheap)'],
    [1,2,3])


#replacing the countries
final_data_input.Country = final_data_input.Country.replace(
    ['USA', 'Sweden', 'Ukraine', 'Kenya', 'Brazil', 'Bangladesh'],
    [1,2,3,4,5,6])


#replacing the products
final_data_input.Product = final_data_input.Product.replace(
    ['Apple', 'Banana', 'Tomatoes', 'Potato', 'Flour', 'Rice', 'Milk',
       'Butter', 'Eggs', 'Chicken'],
    [1,2,3,4,5,6,7,8,9,10])

print(final_data_input.head())
final_data_input = final_data_input.dropna()
final_data_input.shape
model_data = {
    
    #Data values
    "prices_N" : final_data_input.shape[0], #prices observations, 30*(number of forms)
    'prices' : final_data_input.Price.values, #prices for each of teh observation


    'store_N': len(final_data_input.Store_Perception.unique()), # types of stores - 3
    "store_types" : final_data_input.Store_Perception.values, # vals 1-3 to indicate the store pricing

    'country_N': len(final_data_input.Country.unique()), # countries in analysis - 6
    "countries" : final_data_input.Country.values, #cals 1-6 to indicate different countries 


    'products_N': len(final_data_input.Product.unique()), #products observed  - 10  
    'products': final_data_input.Product.values # vals 1-10 for each product type

}
# type(list(model_data['countries']))
# print(model_data['countries'])
# print((model_data['prices_N']))
# final_data_input.Country.unique()<jupyter_output><empty_output><jupyter_text>Building the modelIn this seciton, I will build the stan models. Then, I will visualize the results of the stan model with the predicted prices, country, and store type multipliers distribution, as we have previously conducted with the election votes. This will help to jusge on whether the predictions seem plausible.<jupyter_code>stan_code = """

data {
  int<lower=1> prices_N;                  // number of observed prices
  real<lower =0> prices[prices_N];         //list of prices


  int<lower=1> store_N;                 // number of different store types
  int<lower =1> store_types[prices_N];            // list of store types


  int<lower=1> country_N;               // number of countries 
  int<lower = 1> countries[prices_N];  // list of countries 


  int<lower = 1> products_N;            // number of different products presented
  int<lower = 1> products[prices_N];  // list of different products

}


parameters {

  // 3 multipliers to input into likelihood function
  real<lower = 0> base_price[products_N]; // 10 different base prices
  real<lower=0> country_multiplier[country_N]; // 6 different multipliers
  real<lower = 0> store_multiplier[store_N]; // 3 multipliers



// unknown parameters

real<lower = 0> sigma;
real<lower=0> lambda;
}

model {

//Hyperparameters priors

sigma ~ gamma(1,1); //uninformed prior about the uncertainty of the N distribution
lambda ~ gamma(1, 0.2); //hyperparameter for the exponential likelihood function


// Priors

base_price ~ exponential(lambda);
country_multiplier ~ lognormal(0,1);
store_multiplier ~ lognormal(0,1);


//Likelihood

for (datum in 1:prices_N) {
    prices[datum] ~ normal(base_price[products[datum]]*country_multiplier[countries[datum]]*store_multiplier[store_types[datum]], sigma);


}


}
"""
stan_model = pystan.StanModel(model_code=stan_code)
results = stan_model.sampling(data=model_data)
print(results)

samples = results.extract()<jupyter_output><empty_output><jupyter_text>Distribution of predicted base pricesThe base price looks okay, with butter and chicken having the highest prices and highest uncertainty. By looking at the data, this trends looks consistent. This graph, however, does not reproduce the real prices because prices will be change by the country and store type multipliers, the distribution for which are shown below.<jupyter_code>plt.figure(figsize=(15,6))

for i in range(model_data['products_N']): #for each of the product
    plt.plot(stats.uniform.rvs(loc=i, scale=0.4, size = 4000),
             samples['base_price'][:,i], ',', alpha=2)
    
# plt.plot(range(1, len(final_data.Product.unique()) + 1),
#          samples['base_price'].mean(axis=0), marker='.',
#          linewidth=0,color="black", alpha=0.8, markersize=12)

plt.title('Samples of posterior distributions of base prices')
plt.xticks(range(1,11), (products_names))
plt.ylim(0, 30)
plt.show()<jupyter_output><empty_output><jupyter_text>Distribution of country multiplierThe multipliers are mostly above or very close to 1. For the more expensive and developed countries, the multiplier range is higher (USA and Sweden) that will make the prices more expensive. For Kenya, Brazil, and Bangladesh, the multipliers are lower. This distribution seems relevant and error-free<jupyter_code>plt.figure(figsize=(15,6))

for i in range(model_data['country_N']): #for each of the country
    plt.plot(stats.uniform.rvs(loc=i, scale=0.4, size = 4000),
             samples['country_multiplier'][:,i], ',', alpha=2)


plt.title('Samples of posterior distributions of base prices')
plt.xticks(range(1,7), (country_names))
plt.ylim(0, 4)
plt.show()<jupyter_output><empty_output><jupyter_text>Distribution of Store Type MultiplierThe budget stores have the lowest values. They are slightly below 1, reducing the price of the products. The mid range and the expensive stores have higher values, with mid range more concentrated in the lower part of the distribution. They are very similar. This might be because of the students' perception of the stores and where in one location, the store is considered mid-range, but in the other - as luxury, but they will have similar prices or their correlation<jupyter_code>plt.figure(figsize=(15,6))

for i in range(model_data['store_N']): #for each of the store type
    plt.plot(stats.uniform.rvs(loc=i, scale=0.4, size = 4000),
             samples['country_multiplier'][:,i], ',', alpha=2)


plt.title('Samples of posterior distributions of store types')
plt.xticks(range(1,4), (store_names))
plt.ylim(0, 10)
plt.show()<jupyter_output><empty_output><jupyter_text>Rental Prices and Country Multipliers' CorrelationIn this section, I will extract the Correlation Coefficient (R-squared) for the rental prices and the country multiplier. First, I cleaned the rental data to contain only numbers. I converted all the values that were in dollars to current currency and looked through the numbers for sanity check. Then, I saved the data in the correct format. Last, I calculated $R^2$ and outputted the correlation plot.<jupyter_code>### CLEANING THE RENTAL DATA

#country multipliers data
countries_multipliers = samples['country_multiplier'].mean(axis=0)



#creating df containing country and the rental price
rental_df_data = [final_data.Rental_Price, final_data.Country, data_normalized.Currency_Converter]
rental_df_headers = ['Rental_prices', 'Country', "Currency_Converter"]
rental_price = pd.concat(rental_df_data, axis =1, keys = rental_df_headers)


#manually cleaning the rental prices

rental_price.loc[8,'Rental_prices'] = 2595
rental_price.loc[9,'Rental_prices'] = 1695
rental_price.loc[10,'Rental_prices'] = 28000
rental_price.loc[11,'Rental_prices'] = 42000
rental_price.loc[44,'Rental_prices'] = 4000
rental_price.loc[45,'Rental_prices'] = 3000
rental_price.loc[21,'Rental_prices'] = 1934
rental_price.loc[38,'Rental_prices'] = 2595	
rental_price.loc[39,'Rental_prices'] = 2595	

#converting the prices to dollars

rental_price.Rental_prices= pd.to_numeric(rental_price.Rental_prices, downcast="float")
rental_price.Rental_prices *=rental_price.Currency_Converter




  

## RENTAL DATA IN THE RIGHT FORMAT - MEANS PER COUNTRY IN DOLLARS

USA = []
Kenya = [455]
Brazil = []
Bangladesh = [48, 36]
Ukraine = []
Sweden = [826.44, 826.44	]



#get the prices based on a country
for i in range (50):
  #sort out US rental prices
  if rental_price.loc[i,'Country'] == 'USA':
    USA.append(rental_price.loc[i,'Rental_prices'])	

  elif rental_price.loc[i,'Country'] == 'Brazil':
    Brazil.append(rental_price.loc[i,'Rental_prices'])	


  elif rental_price.loc[i,'Country'] == 'Ukraine':
    Ukraine.append(rental_price.loc[i,'Rental_prices'])	




#getting the rental prices means for each country
USA_mean = np.mean(USA)
Kenya_mean = 455
Brazil_mean = np.mean(Brazil)
Bangladesh_mean = np.mean(Bangladesh)
Sweden_mean = 826.44
Ukraine_mean = np.mean(Ukraine)


#putting the mean values into 1 list
means = [USA_mean, Sweden_mean, Ukraine_mean, Kenya_mean, Brazil_mean, Bangladesh_mean]




##CALCULATING THE CORRELATION

correlation_matrix = np.corrcoef(means, countries_multipliers)
correlation_xy = correlation_matrix[0,1]
r_squared = correlation_xy**2 

print(r_squared)


# regression plot using seaborn
fig = plt.figure(figsize=(15, 8))
sns.regplot(x=means, y=countries_multipliers, color='blue', marker='+')

# Legend, title and labels.
plt.title('Relationship between Rental Prices and Country Multipliers', size=24)
plt.xlabel('Mean Values for the Rental Prices in Different countries', size=18)
plt.ylabel('Country Multipliers', size=18);<jupyter_output>0.4911380003665486
{'fraction_non_alphanumeric': 0.10779608250097289, 'fraction_numerical': 0.02328447269425347, 'mean_word_length': 4.7255848496101, 'pattern_counts': {'":': 5, '<': 55, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '35646', 'license': 'MIT', 'n_tokens_mistral': 5110, 'n_tokens_neox': 4779, 'n_words': 1461, 'path': 'Location_Based_Assignment_Price_Modeling.ipynb', 'repo_name': 'Jeongwoo-KGI/sides-and-school', 'size': '15418'}