<jupyter_start><jupyter_text>SALib applicationThe following notebook can be used in order to replicate the SALib application developed for **The modifiable areal unit problem in geospatial least-cost electrification modelling** with the DBSCAN algorithm. Please refer **ADD LINK TO MENDELEY** for the final input files used in the analysis. The code generates sensitivity measures for given input parameters with regards to a specific output parameter. TheorySALib1 is an open-source python library containing applications of some of the most common sensitivity analysis methods. For our publication we have used the Delta Moment-Independent Measure (DMIM)2, 3.  InputA "summary of summaries" file is needed. For the paper a summary file was generated using the summary files zipped in the **Results_used_in_Salib** folder (the finished input file is also available on this repo at: ).The summary files includes:* 6,240 rows representing three countries  * Column C through P are model inputs * Column Q through AF are model outputs * Each country has three specific values that effect our model and are usually not subject to sensitivity analysis (current electrification rate, population in 2030 and national population density). They are the same across all national scenarios (2,080 entries per value). These values are taken from literature and considered "known", but when comparing all three countries they have a certain effect and are therefore included. **Note** these values are not in the file and added in the cells below.  Pre-processingIf you wish to conduct the sensitivity analysis on your own data you need to first generate the population layers (please refer to the DBSCAN.ipynb for information on how to do so). You also need to run OnSSET for the desired scenarios. Information on how to run OnSSET can be found on the official OnSSET websites and the various resources linked there (http://www.onsset.org/). The OnSSET codes used for this work specifically differs from the official OnSSET codes currently available throught the OnSSET repository, the codes for the paper are available in this repository with documentation of what has changed.  OutputThe output includes:* **delta** - DMIM delta value for each input * **S1** - First order Sobol index for each input* **delta_conf** - The confidence intervall for each input's DMIM delta value* **S1_conf** - The confidence intervall for each input's first order Sobol index____________________________________________________________________________________________________________________________1 <NAME>. & <NAME>. SALib: An open-source Python library for Sensitivity Analysis. Journal of Open Source Software 2, 97 (2017). 2 <NAME>. (2007). “A new uncertainty importance measure.” Reliability Engineering & System Safety, 92(6):771-784, doi:10.1016/j.ress.2006.04.015. 3 <NAME>., <NAME>, and <NAME> (2013). “Global sensitivity measures from given data.” European Journal of Operational Research, 226(3):536-550, doi:10.1016/j.ejor.2012.11.047.   Cell 1 - Importing packages**Do not edit this cell**<jupyter_code>import sys

from SALib.analyze import delta
from SALib.util import read_param_file
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Cell 2 - PreprocessingPlease note that this section has to be ran and is highly specific to country and input data. Here we add: * The electrification rate in the start year for the countries studied (by the iso-2 code included in the summary file)* The population in the end year (2030) by the iso-2 codeThe values entered here may have to be updated if the country is changed or new data becomes available.After the new columns have been generated the columns are rearranged in order to have all the inputs first and all the outputs last. The dataframe is also shuffled.**Note** the path of the summary of summaries have to be specified after you have downloaded everything to your own computer (this is the first line of the cell)<jupyter_code>df = pd.read_csv("Summary of summaries.csv", sep=';')

df['ElecRate'] = 0.42
df["Pop2030"] = 15672000

df['ElecRate'] = np.where(df['Country'] == 'na', 0.54, df['ElecRate'])
df['ElecRate'] = np.where(df['Country'] == 'mw', 0.18, df['ElecRate'])

df['Pop2030'] = np.where(df['Country'] == 'na', 3010873, df['Pop2030'] )
df['Pop2030'] = np.where(df['Country'] == 'mw', 24849000, df['Pop2030'])

cols = ['EDemand',
 'GridGenCost',
 'PVCost',
 'GridCapInvCost',
 'DiscountRate',
 'LVCost',
 'MVCost',
 'GridLosses',
 'Core',
 'Buffer',
 'Admin',
 'Method',
 'Res',
 'ElecRate',
 'NatPopDens',
 'Pop2030',
 'PopGrid',
 'PopMG',
 'PopSA',
 'AddedCapGrid',
 'AddedCapSA',
 'AddedCapMG',
 'GridInv',
 'SAInv',
 'MGInv',
 'TotInv',
 'Country']

df = df[cols]

df["RatioGridPop"] = 100*(df["PopGrid"]/df['Pop2030'])
df["RatioMGPop"] = 100*(df["PopMG"]/df['Pop2030'])
df["RatioSAPop"] = 100*(df["PopSA"]/df['Pop2030'])

df["RatioGridInv"] = 100*(df["GridInv"]/df['TotInv'])
df["RatioSAInv"] = 100*(df["SAInv"]/df['TotInv'])
df["RatioMGInv"] = 100*(df["MGInv"]/df['TotInv'])

df_shuffled = df.sample(frac=1)<jupyter_output><empty_output><jupyter_text>Cell 3 - Splitting the file into national versions<jupyter_code>dfB = df_shuffled.loc[df['Country'] == 'bj']
dfN = df_shuffled.loc[df['Country'] == 'na']
dfM = df_shuffled.loc[df['Country'] == 'mw']
dfB['id'] = np.arange(len(dfB))
dfN['id'] = np.arange(len(dfN))
dfM['id'] = np.arange(len(dfM))<jupyter_output><empty_output><jupyter_text>Cell 4 - Reading param filesSALib requires a parameter file. The parameter files are provided in the **Sample input** folder. The files have to include one row for each input paramter and three columns: 1) name of input, 2) minimum value and 3) maximum valueIf you use your own data these have to be generated from scratch. We are reading four parameter files 1) one for the merged data, 2) one for Benin, 3) one for Namibia and 4) one for Malawi.**Note** the paths have to be updated after you have downloaded everything to your own computer<jupyter_code>problem_merged = read_param_file('params_merged.txt')
problem_bj =read_param_file('params_bj.txt')
problem_na = read_param_file('params_na.txt')
problem_mw = read_param_file('params_mw.txt')<jupyter_output><empty_output><jupyter_text>Merged DMIM assessmentIf you are interested in a specific country bypass the three cells below.**Note** This code assumes that you have 16 independent factors. If you increase or decrease this number please ensure that the first line below is updated. Also the first line in the second cell below needs to be updated based on which output variable you want to assess the sensitivity with regards to.<jupyter_code>param_values=df_shuffled.iloc[:, : 16]
np.savetxt(r'model_input_merged.txt', param_values.values)
X_merged = np.loadtxt(r"model_input_merged.txt")
out = df_shuffled["RatioMGPop"]
np.savetxt(r'model_output_merged.txt', out.values)
Y_merged = np.loadtxt(r"model_output_merged.txt")
S_merged = delta.analyze(problem_merged, X_merged, Y_merged, num_resamples=10, conf_level=0.95, print_to_console=True)<jupyter_output><empty_output><jupyter_text>Benin DMIM assessment**Note** This code assumes that you have 14 independent factors. If you increase or decrease this number please ensure that the first line below is updated. Also the first line in the second cell below needs to be updated based on which output variable you want to assess the sensitivity with regards to.<jupyter_code>param_values=dfB.iloc[:, : 14]
np.savetxt(r'model_input_ben.txt', param_values.values)
X_bj = np.loadtxt(r"model_input_ben.txt")
out = dfB["RatioMGPop"]
np.savetxt(r'model_output_ben.txt', out.values)
Y_bj = np.loadtxt(r"model_output_ben.txt")
Si_ben = delta.analyze(problem_bj, X_bj, Y_bj, num_resamples=10, conf_level=0.95, print_to_console=True)<jupyter_output><empty_output><jupyter_text>Malawi DMIM assessment**Note** This code assumes that you have 14 independent factors. If you increase or decrease this number please ensure that the first line below is updated. Also the first line in the second cell below needs to be updated based on which output variable you want to assess the sensitivity with regards to.<jupyter_code>param_values=dfM.iloc[:, : 14]
np.savetxt(r'model_input_mal.txt', param_values.values)
X_mw = np.loadtxt(r"model_input_mal.txt")
out = dfM["LCOE"]
np.savetxt(r'model_output_mal.txt', out.values)
Y_mw = np.loadtxt(r"model_output_mal.txt")
Si_mal = delta.analyze(problem_mw, X_mw, Y_mw, num_resamples=10, conf_level=0.95, print_to_console=True)<jupyter_output><empty_output><jupyter_text>Namibia DMIM assessment**Note** This code assumes that you have 14 independent factors. If you increase or decrease this number please ensure that the first line below is updated. Also the first line in the second cell below needs to be updated based on which output variable you want to assess the sensitivity with regards to.<jupyter_code>param_values=dfN.iloc[:, : 14]
np.savetxt(r'model_input_nam.txt', param_values.values)
X_na = np.loadtxt(r"model_input_nam.txt")
out = dfN["LCOE"]
np.savetxt(r'model_output_nam.txt', out.values)
Y_na = np.loadtxt(r"model_output_nam.txt")
Si_nam = delta.analyze(problem_na, X_na, Y_na, num_resamples=10, conf_level=0.95, print_to_console=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09974176888315042, 'fraction_numerical': 0.024101570905960833, 'mean_word_length': 5.9677661169415295, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '88748', 'license': 'MIT', 'n_tokens_mistral': 2973, 'n_tokens_neox': 2731, 'n_words': 1062, 'path': 'SALib application.ipynb', 'repo_name': 'babakkhavari/MAUP', 'size': '9321'}