<jupyter_start><jupyter_text>ReclasificationWe ran a LightGBM model with the given labels and got ~45% accuracy. Thus, we decided to try different reclasification approaches according to what was discussed during the Q&As about the ambiguity of the original "9-labels" classification, such as `Kmeans` and `Gaussian Mixture`. The idea with this was see if the data itself could be separated in more general classes that the ones that were given. This route didn't take us anywhere as most of the samples were classified in the same group.At the end, we decided to go for a reclasification of labels looking for the most parcimonious ECMs, as follows:<jupyter_code>reclasification = {
    "L-R-RCPE": "0",
    "L-R-RCPE-RCPE": "0",
    "L-R-RCPE-RCPE-RCPE": "0",
    "RC-G-G": "1",
    "RC-RC-RCPE-RCPE": "2",
    "RCPE-RCPE": "2",
    "RCPE-RCPE-RCPE": "2",
    "RCPE-RCPE-RCPE-RCPE": "2",
    "Rs_Ws": "3"
}
inverse_reclasification = {
    "0": "L-R-RCPE",
    "1": "RC-G-G",
    "2": "RCPE-RCPE",
    "3": "Rs_Ws"
}
eis_data["circuit_recla"] = eis_data["Circuit"].map(reclasification)
def complex_parts(cmp: pd.Series) -> tuple[pd.Series, pd.Series]:
    real_part= cmp.apply(np.real)
    imag_part= cmp.apply(np.imag)
    return (real_part, imag_part)

eis_data["Z_real"], eis_data["Z_imag"] = complex_parts(eis_data.Z)
display(eis_data.head())<jupyter_output><empty_output><jupyter_text>Feature extractionFirst, we interpolated these two mappings: frequency-to-Z_real and frequency-to-Z_imaginary. Then, we numerically computed the first and second derivatives. The idea with this feature extraction was to make notable points such as maxima, minima, inflection points and straight lines, more evident.<jupyter_code>def nth_derivatives(x: np.array, y: np.array, n: int = 1) -> list[np.array]:
    this_der = y
    res = [this_der]
    for i in range(n):
        nth = i+1
        this_der = numerical_derivative(x, this_der)
        res.append(this_der)
    return res
def numerical_derivative(x: np.array, y: np.array) -> np.array:
    assert x.shape == y.shape
    dy = np.gradient(y)
    dx = np.gradient(x)
    der = dy/dx
    return der
from typing import Callable
def get_interpolating_function(x: np.ndarray, y: np.ndarray) -> Callable:
    #print(f"len(x)= {len(x)}, len(y={len(y)})")
    assert len(x) == len(y)
    # Obtain functions for interpolation
    interp = interpolate.interp1d(x, y, fill_value="extrapolate") #, assume_sorted = True)
    return interp

def eval_func(x: np.ndarray, func: Callable) -> np.ndarray:
    return func(x)
freq_interpolation = np.logspace(1.0, 5.0, num=100, base=10.0)
eis_data["freq_interpolation"] = None
eis_data["freq_interpolation"] = eis_data["freq_interpolation"].apply(lambda x: freq_interpolation)

cols = []
for i in ["Z_real", "Z_imag"]:
    inter_p_col = f'{i}_interp'
    eis_data[inter_p_col] = eis_data.apply(lambda x: get_interpolating_function(x.freq, x[i]), axis=1)
    values_col = f'{i}_values'
    eis_data[values_col] = eis_data.apply(lambda x: eval_func(x.freq_interpolation, x[inter_p_col]), axis=1)
    cols.extend([inter_p_col, values_col])
eis_data["Z_real_d1_values"] = eis_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_real_values, 2)[1], axis=1)    
eis_data["Z_real_d2_values"] = eis_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_real_values, 2)[2], axis=1)
eis_data["Z_imag_d1_values"] = eis_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_imag_values, 2)[1], axis=1)    
eis_data["Z_imag_d2_values"] = eis_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_imag_values, 2)[2], axis=1)

cols= [
        'Z_real_values'
        ,'Z_real_d1_values'
        ,'Z_real_d2_values'
        ,'Z_imag_values'
        ,'Z_imag_d1_values'
        ,'Z_imag_d2_values'
    ]<jupyter_output><empty_output><jupyter_text>Data PreprocessingHere we extract nd arrays into columns.<jupyter_code>print(cols)
exploded_eis_data = eis_data.explode(cols).reset_index()
exploded_eis_data["idx"] = exploded_eis_data.groupby("index").transform("cumcount")
unstacked_dfs = []
exploded_cols = []
for c in cols:
    name_col = c + "_idx"
    name_template = c + "_{x:03d}"
    print(name_template)
    exploded_eis_data[name_col] = exploded_eis_data["idx"].apply(lambda x: name_template.format(x=int(x)))
    exploded_cols.extend(set(exploded_eis_data[name_col]))
    exploded_eis_data[c] = exploded_eis_data[c].astype('float32')
    temp_df = exploded_eis_data.set_index(["index", name_col])[c].unstack()
    unstacked_dfs.append(temp_df)

res = eis_data.join(unstacked_dfs)<jupyter_output>['Z_real_values', 'Z_real_d1_values', 'Z_real_d2_values', 'Z_imag_values', 'Z_imag_d1_values', 'Z_imag_d2_values']
Z_real_values_{x:03d}
Z_real_d1_values_{x:03d}
Z_real_d2_values_{x:03d}
Z_imag_values_{x:03d}
Z_imag_d1_values_{x:03d}
Z_imag_d2_values_{x:03d}<jupyter_text>Normalize features<jupyter_code>X = sklearn.preprocessing.normalize(res[exploded_cols], axis=0)<jupyter_output><empty_output><jupyter_text>Split data into train and test setsWe do this to evaluate the model in unseen data<jupyter_code>from sklearn.model_selection import train_test_split
# case = case_001
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(
    res[exploded_cols],
    res["circuit_recla"],
    test_size = 0.3,
    # random_state = 0
)<jupyter_output><empty_output><jupyter_text>Preparation of the LightGBMHere we prepare the classifying model. As the model was suffering overfitting, we use max_depth and min_data_in_leaf to control it.Note to the reader: we adapted the usage of LightGBM from the article by <NAME> "LightGBM Classifier in Python". Please refer to: https://www.kaggle.com/code/prashant111/lightgbm-classifier-in-python/notebook<jupyter_code>clf = lgb.LGBMClassifier(max_depth=5, min_data_in_leaf=100)
clf.fit(X_train, y_train)


y_pred=clf.predict(X_test)
from sklearn.metrics import accuracy_score
accuracy=accuracy_score(y_pred, y_test)
print('LightGBM Model accuracy score: {0:0.4f}'.format(accuracy_score(y_test, y_pred)))

y_pred_train = clf.predict(X_train)
print('Training-set accuracy score: {0:0.4f}'. format(accuracy_score(y_train, y_pred_train)))
# print the scores on training and test set

print('Training set score: {:.4f}'.format(clf.score(X_train, y_train)))
print('Test set score: {:.4f}'.format(clf.score(X_test, y_test)))<jupyter_output>Training set score: 0.8798
Test set score: 0.7200<jupyter_text>Confusion MatrixAs can be seen by the confusion matrix, still the groups "0" (L-R-RCPE-n) and "2" (RCPE-n) are confused bewteen each other.<jupyter_code>from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
print('Confusion matrix\n\n', cm)
cm_matrix = pd.DataFrame(
    data=cm,
    columns=[f"Predicted_{i}" for i in range(len(cm))], 
    index=[f"Real_{i}" for i in range(len(cm))]
)

sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')<jupyter_output><empty_output><jupyter_text>UNLABELED DATANow we face the real challenge: `unlabeled_data`<jupyter_code>here = !pwd

challenge_data_path = path.join(path.dirname(here[0]), "unlabeled_data.csv")
challenge_data = eis_dataframe_from_csv(challenge_data_path)
challenge_data["len_freq"] = challenge_data["freq"].apply(lambda x: x.shape[0])
challenge_data["len_Z"] = challenge_data["Z"].apply(lambda x: x.shape[0])
challenge_data["Z_real"], challenge_data["Z_imag"] = complex_parts(challenge_data.Z)
#challenge_data["len_Z_real"] = challenge_data["Z_real"].apply(lambda x: x.shape[0])
#challenge_data["len_Z_imag"] = challenge_data["Z_imag"].apply(lambda x: x.shape[0])

challenge_data[["Z_real", "Z_imag"]]
#display(challenge_data.head())

#challenge_data["freq"][0].shape
freq_interpolation = np.logspace(1.0, 5.0, num=100, base=10.0)
challenge_data["freq_interpolation"] = None
challenge_data["freq_interpolation"] = challenge_data["freq_interpolation"].apply(lambda x: freq_interpolation)

cols = []
for i in ["Z_real", "Z_imag"]:
    
    inter_p_col = f'{i}_interp'
    challenge_data[inter_p_col] = challenge_data.apply(lambda x: get_interpolating_function(x.freq, x[i]), axis=1)
    values_col = f'{i}_values'
    challenge_data[values_col] = challenge_data.apply(lambda x: eval_func(x.freq_interpolation, x[inter_p_col]), axis=1)
    cols.extend([inter_p_col, values_col])
 
    
challenge_data["Z_real_d1_values"] = challenge_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_real_values, 2)[1], axis=1)    
challenge_data["Z_real_d2_values"] = challenge_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_real_values, 2)[2], axis=1)
challenge_data["Z_imag_d1_values"] = challenge_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_imag_values, 2)[1], axis=1)    
challenge_data["Z_imag_d2_values"] = challenge_data.apply(lambda x: nth_derivatives(x.freq_interpolation, x.Z_imag_values, 2)[2], axis=1)

cols= [
        'Z_real_values'
        ,'Z_real_d1_values'
        ,'Z_real_d2_values'
        ,'Z_imag_values'
        ,'Z_imag_d1_values'
        ,'Z_imag_d2_values'
    ]
#samples = challenge_data.sample(1000)
print(cols)
exploded_eis_data = challenge_data.explode(cols).reset_index()
print(f"shape exploded_data: {exploded_eis_data.shape}")
exploded_eis_data["idx"] = exploded_eis_data.groupby("index").transform("cumcount")
unstacked_dfs = []
exploded_cols = []
for c in cols:
    name_col = c + "_idx"
    name_template = c + "_{x:03d}"
    print(name_template)
    exploded_eis_data[name_col] = exploded_eis_data["idx"].apply(lambda x: name_template.format(x=int(x)))
    exploded_cols.extend(set(exploded_eis_data[name_col]))
    print(f"Total feature cols: {len(exploded_cols)}")
    exploded_eis_data[c] = exploded_eis_data[c].astype('float32')
    temp_df = exploded_eis_data.set_index(["index", name_col])[c].unstack()
    unstacked_dfs.append(temp_df)

res = challenge_data.join(unstacked_dfs)
submission_pred =clf.predict(res[exploded_cols]
challenge_data["reclasification_pred"] = submission_pred
challenge_data["Circuit"] = challenge_data["reclasification_pred"].map(inverse_reclasification)
submission_data = challenge_data[["reclasification_pred", "Circuit"]].copy()
submission_data["Parameters"] = None
submission_data.head()
submission_data[["Circuit", "Parameters"]].to_csv(path.join(path.dirname(here[0]), "submission.csv"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13552618886852513, 'fraction_numerical': 0.015047747660846917, 'mean_word_length': 6.038696537678208, 'pattern_counts': {'":': 13, '<': 32, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '209579', 'license': 'MIT', 'n_tokens_mistral': 4068, 'n_tokens_neox': 3915, 'n_words': 720, 'path': 'notebooks/LightGBM-001.ipynb', 'repo_name': 'elpablete/dev22-team-sjn', 'size': '10378'}