<jupyter_start><jupyter_text>This is a script to find the best performing configuration from a set of dimensions (partition, schema, storage) Query runtimes of a single dimension set of configuration<jupyter_code># SINGLE DIMENSION #
import pandas as pd
import scipy.stats as ss
import numpy as np
file_format = ['csv', 'avro', 'parquet', 'orc']
schemas = ['st','vt', 'pt', 'extvt', 'wpt']
partition = ['horizontal','predicate', 'subject']
dimension = 'storage'
size = '100'

file_reader(dimension,size)<jupyter_output><empty_output><jupyter_text>R Scores is a single dimensional ranking criteria which is used to help provide high-level view of the system performance across a set of tasks. We apply the ranking criteria independently for each dimension<jupyter_code>dimension = 'storage'
size = 100
R_score(dimension, size) #give formula in markdown<jupyter_output><empty_output><jupyter_text>Here we can select 2 parameters that we want to see based on the full table of rank scores that we see before<jupyter_code>param1 = ['horizontal','predicate']
param2 = ['st', 'vt']
rank_part(param1, param2) #implement feature for query runtimes<jupyter_output><empty_output><jupyter_text>This diagram shows the performance of a single dimension parameters that the user choose in terms of their rank scores<jupyter_code>getDiagramSD('st')<jupyter_output><empty_output><jupyter_text>However, ranking over one dimension ignoring the others ends up with selecting different configurations as we can see in the radar chart below<jupyter_code>dimension = 'storage'
size = 100
radarDiagram(dimension, size)<jupyter_output>['orc', 'st', 'predicate']
[0.9696969696969697, 0.6363636363636364, 0.22727272727272727]<jupyter_text>This is why we propose an optimization technique, aim to find the optimal non-dominated solutions or the configuration combinations by optimizing all dimensions at the same time which utilize the NSGA2 Algorithm. The pareto_Q apply the NSGA2 considering the rank sets obtained by sorting each query result individually. While the pareto_Agg operates on the single dimensional ranking criteria (schema, partition, storage) which aims to maximizing the performance of the three ranks all together<jupyter_code># PARETO #
getPareto_Q(250)
# getPareto_Agg(250)

# Pareto's Diagram #
# getDiagramPareto()<jupyter_output><empty_output><jupyter_text>Conformance is a measure of the adherence of the top-ranked configurations according to the actual query positioning of those configurations<jupyter_code># HEADLINE RANKING CRITERIA GOODNESS METRICS
# CONFORMANCE
dataset = [100,250]
dimension = ['storage', 'partition', 'schema', 'pareto_Q', 'pareto_Agg']
getConformance(dimension, dataset, 5, 13)<jupyter_output>storage 100
storage 250
partition 100
partition 250
schema 100
schema 250
pareto_Q 100
pareto_Q 250
pareto_Agg 100
pareto_Agg 250<jupyter_text>Coherence is the measure agreement between two ranking sets that use the same ranking criteria across different experiments<jupyter_code># COHERENCE
dimension = ['storage', 'partition', 'schema', 'pareto_Q', 'pareto_Agg']
getCoherence(dimension, 100, 250)<jupyter_output><empty_output><jupyter_text>CALCULATE COHERENCE<jupyter_code>getPareto_Agg(100)
var1 = pd.concat(rank_dataframe, axis = 0)
var1 = var1.loc[list(pareto_agg.values())]
var1 = var1['Result']

getPareto_Agg(250)
var2 = pd.concat(rank_dataframe, axis = 0)
var2 = var2.loc[list(pareto_agg.values())]
var2 = var2['Result']

new_df = pd.concat([var1, var2], axis = 1)
column_name = new_df.columns.values
column_name[0] = str(100) #var1
column_name[1] = str(250) #var2

# new_df.columns = column_name
# new_df = new_df.dropna()
# var1 = new_df[str(100)]
# var2 = new_df[str(250)]
new_df
file_format = ['csv', 'avro', 'parquet', 'orc']
schemas = ['st','vt', 'pt', 'extvt', 'wpt']
partition = ['horizontal','predicate', 'subject']
dimension = 'storage'
size = '250'

# for i in schemas:
#     for j in partition:
#         for k in file_format:
            # check folder
            # print(k,i,j)
            # print(f'./logs/{k}/{size}M_{i}_{j}.txt')
# dimension = 3

for i in range(dimension):
    for k in file_format:
        print(k)
            # df = pd.read_csv(f'./logs/{k}/{size}M_{i}_{j}.txt',sep = ',', header = None)
storage = ['csv', 'avro', 'parquet', 'orc']
schemas = ['st','vt', 'pt', 'extvt', 'wpt']
partition = ['horizontal','predicate', 'subject']
# dimension = ['storage', 'schemas', 'partition']
size = '250'

# for i in schemas:
#     for k in file_format:
#         for j in partition:
#             print(i,k,j)
# def nested(dimension):
#     total_loop = len(storage)*len(schemas)*len(partition)
#     # return total_loop
#     for i in range(total_loop):
#         print(partition)

# nested(3)
for i in range(len(dimension)):
    print(len(dimension[i]))
def nested(number_of_loops, range_list, execute_function, current_index=0, iter_list = []):
    if iter_list == []:
        iter_list = [0]*number_of_loops

    if current_index == number_of_loops-1:
        for iter_list[current_index] in range_list[current_index]:
            execute_function(iter_list)
    else:
        for iter_list[current_index] in range_list[current_index]:
            nested(number_of_loops, iter_list = iter_list, range_list = range_list,  current_index = current_index+1, execute_function = execute_function)

def read_log(index_list):
    # index_list[0]
    print(index_list[0], index_list[1], index_list[2], index_list[3])
        # print(f'./logs/{index_list[2]}/{size}M_{index_list[0]}_{index_list[1]}.txt')
nested(range_list = dimension, execute_function = read_log , number_of_loops=len(dimension))
newDimension = ['option1', 'option2']
dimension = [schemas, partition, storage, newDimension]
# size = '100'
iter_list = [0]*3
iter_list
if current_index == number_of_loops-1:
        for iter_list[current_index] in range_list[current_index]:
            execute_function(iter_list)
import itertools

a = list(itertools.product(*dimension))
print(len(a))

def file_reader2(dimensions, size: str):
    global df
    global dfs
    configurations = list(itertools.product(*dimensions))
    for c in configurations:
        
    if dimension == "storage":
        idx = []
        li=[]
        # READ LOG - STORAGE FORMAT
        for i in schemas:
            for j in partition:
                for k in file_format:
                    # check folder
                    # print(f'./logs/{k}/{size}M_{i}_{j}.txt')

                    df = pd.read_csv(f'./logs/{k}/{size}M_{i}_{j}.txt',sep = ',', header = None)
                    df = df.fillna(0)
                    avg = df.mean(axis = 0)
                    li.append(avg)
                    idx.append(k + f"_{i}_{j}")

        # create dataframe
        df = pd.DataFrame(li, index = [idx])
        df = df.set_axis(["Q"+str(i+1) for i in range(11)], axis = 1)
        df = df.fillna(5000)
        # split dataframe
        dict = {}
        count = 0
        loop = len(file_format)
        dfs = []
        for i in range(int(len(partition) * len(file_format) * len(schemas) / len(file_format))):
            dict['df_{}'.format(i)] = df[count:loop]
            count = loop
            loop = loop+len(file_format)
            dfs.append(dict[f'df_{i}'])
import plotly.express as px
import pandas as pd
df = pd.DataFrame(dict(
    r=[0.98, 0.55, 0.23],
    theta=['ORC','ST','Predicate']))
fig = px.line_polar(df, r='r', theta='theta', line_close=True)
fig.show()
s = R_score('storage', 250)['Result'].idxmax().split()
s
x = 'st'
y = 'predicate'
z = 'orc'
r = rank_dataframe_rscore.loc[rank_dataframe_rscore.index.str.contains(fr'(?=.*\b{x}\b)(?=.*\b{y}\b)(?=.*\b{z}\b)', regex=True)]['Result']
r[0]
d = ['er', 'eraq', 'ewr']
for i in d:
    print(i)
- dimensions:
    - schemas:
        wpt
        vt
    - partition:
        horizontal
    -storage:
        avro
[(i,j) for i in range(10) for j in range(10,20)]

dimensionsAll = getRanking().reset_index().values
dimensions = np.array(getRanking()[:], dtype = np.float64)
dimensionsAll<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.130930261365044, 'fraction_numerical': 0.027127461910070604, 'mean_word_length': 4.055729492798998, 'pattern_counts': {'":': 1, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '209498', 'license': 'MIT', 'n_tokens_mistral': 2851, 'n_tokens_neox': 2669, 'n_words': 709, 'path': '.ipynb_checkpoints/log_reading_dimensions-checkpoint.ipynb', 'repo_name': 'satria-ui/RDF_BenchRankingLib', 'size': '8073'}