<jupyter_start><jupyter_text>Data ProcessingWe used a data set that contains images, each image is a representation of a planning task, and a table with total run time for a collection of planners on these tasks. Firstly, we simplify the table to indicate which planners are considered preferable to use on each planning task.Secondly, we  Data-Frame Processing   Import the necessary packages.<jupyter_code>import os
import pandas as pd
from utils import *<jupyter_output><empty_output><jupyter_text>Load data frame.<jupyter_code>CURRENT_DIR = os.getcwd()
DATA_DIR = os.path.expanduser(CURRENT_DIR + '/IPC-image-data/lifted')
df = pd.read_csv(CURRENT_DIR + '/IPC-image-data/runtimes.csv')<jupyter_output><empty_output><jupyter_text>Let's take a look at the data:<jupyter_code>df.head(5)<jupyter_output><empty_output><jupyter_text>Each row contains statistics about a certain planning problem: its name and the time it took for each planner to solve it The list of planners is:<jupyter_code>list(df.columns[1:])<jupyter_output><empty_output><jupyter_text>We aim to establish a net that will classify each planning problem to its preferable planner.For training purposes we create a new abstracted data-frame.In the new data-frame we mark 1 where the planner is preferable (top 25%), else we mark 0.Creation of a new data-frame:<jupyter_code>temp_df = df.drop('filename', axis=1)<jupyter_output><empty_output><jupyter_text>For each planning problem we establish a threshold that will help us decide which planner is preferable.A planner is preferable if it did not reach timeout and it is faster than 75% of the other planners.<jupyter_code>threshold = temp_df.apply(lambda x: np.percentile(x,25), axis=1)
threshold[threshold==10000] = temp_df.apply(lambda x: np.percentile(x,20), axis=1)[threshold==10000]
threshold[threshold==10000] = temp_df.apply(lambda x: np.percentile(x,15), axis=1)[threshold==10000]
threshold[threshold==10000] = temp_df.apply(lambda x: np.percentile(x,10), axis=1)[threshold==10000]
threshold[threshold==10000] = temp_df.apply(lambda x: np.percentile(x,5), axis=1)[threshold==10000]
threshold[threshold==10000] = temp_df.apply(lambda x: np.percentile(x,1), axis=1)[threshold==10000]
# list(threshold)<jupyter_output><empty_output><jupyter_text>Add the threshold to the a new column in the data-frame.<jupyter_code>df['threshold'] = threshold
columns = list(df.columns)
columns.remove('filename')

for col in columns:
    cond = (df[col] < df['threshold']) & (df[col] != -1)
    df.loc[cond, col] = 1

df.head()
df.replace(10000, 0, inplace=True)
df.head()
for col in columns:
    cond = (df[col] != 1) & (df[col] != -1)
    df.loc[cond, col] = 0
df.head()
df.drop('threshold', axis=1, inplace=True)

df.head()<jupyter_output><empty_output><jupyter_text>Image ProcessingWe use standard python packages to load image data into a numpy array. Then we convert this array into a torch.Tensor.<jupyter_code>import os
import numpy as np
# import pandas as pd
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
# import matplotlib.pyplot as plt
# from skimage import io, transform, util
# from scipy.fftpack import fft2
# from sklearn.preprocessing import RobustScaler, MinMaxScaler
from pp_dataset import PlannerPortfolioDataset
from architectures import PlaNet
import argparse
import logging
import time
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)<jupyter_output>cuda<jupyter_text>Get path information:<jupyter_code>CURRENT_DIR = os.getcwd()
processed_df_path = os.path.join(CURRENT_DIR, 'df.csv')
image_folder_path = os.path.join(CURRENT_DIR, 'IPC-image-data/lifted/')<jupyter_output><empty_output><jupyter_text>Initiate the dataset:<jupyter_code>plan_dataset = PlannerPortfolioDataset(processed_df_path, image_folder_path, ftransform=1)<jupyter_output><empty_output><jupyter_text>Check out a gray-scale image:<jupyter_code>image = plan_dataset.get_image(150)
plt.figure()
plt.imshow(image, cmap="gray")
plt.show()<jupyter_output><empty_output><jupyter_text>Check out a Fourier-Transformed image:<jupyter_code>image = plan_dataset.get_ft_image(150)
plt.figure()
plt.imshow(20*np.log10(abs(image)), cmap='gray')
plt.show()
fig = plt.figure()
l = [0,120,200]

for i,idx in enumerate(l):
    plt.subplot(2,len(l),i+1)
    image = plan_dataset.get_ft_image(idx)
    plt.imshow(20*np.log10(abs(image)), cmap='gray')

for i,idx in enumerate(l):
    plt.subplot(2,len(l),i+len(l)+1)
    image = plan_dataset.get_image(idx)
    plt.imshow(image, cmap='gray')


plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11652136933871997, 'fraction_numerical': 0.02466510737826919, 'mean_word_length': 6.7368421052631575, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '186964', 'license': 'MIT', 'n_tokens_mistral': 1682, 'n_tokens_neox': 1577, 'n_words': 416, 'path': 'data_process.ipynb', 'repo_name': 'ronen85/deep_planner_portfolio', 'size': '4703'}