<jupyter_start><jupyter_text>LUCAS Processing        This notebook refers to the studies presented in [3] and to Chapter 4 of the Ph.D. thesis [4].    With this notebook, you can process the raw LUCAS 2012 soil dataset to work with the 1D CNNs [2].    We can not guarantee completeness or correctness of the code.    If you find bugs or if you have suggestions on how to improve the code, we encourage you to post your ideas as GitHub issue.     Imports<jupyter_code>import lucas_processing as lucas<jupyter_output><empty_output><jupyter_text>Config<jupyter_code># CHANGE path to data
path_to_data = r"../data/"

# CHANGE verbosity
verbose = 1
# DO NOT CHANGE if you don't know what you are doing
path_to_lucas_csv = path_to_data+"0_LUCAS_TOPSOIL_v1_spectral.csv"
path_to_datafile = path_to_data+"0_LUCAS_TOPSOIL_v1.csv"
path_to_chunk_concat = path_to_data+"1_lucas_fromchunks.csv"
path_to_combined = path_to_data+"2_lucas_fromchunks_combined.csv"
path_to_full = path_to_data+"3_lucas_full.csv"
path_to_final = path_to_data+"4_lucas_final.csv"
path_to_subsets = path_to_data+"5_lucas_final_"
path_to_chunks = path_to_data+"chunks/"
path_to_data = path_to_data+""<jupyter_output><empty_output><jupyter_text>ProcessingAll of the following steps are needed to process the LUCAS dataset. 1. Dimensionality reduction<jupyter_code>lucas.dimension_reduction(
    csv_path=path_to_lucas_csv,
    path_to_chunks=path_to_chunks,
    chunksize=1000,
    verbose=verbose)<jupyter_output><empty_output><jupyter_text>2. Divide dataset into chunks<jupyter_code>lucas.concat_chunks(
    path_to_chunks=path_to_chunks,
    output_file=path_to_chunk_concat,
    verbose=verbose)<jupyter_output><empty_output><jupyter_text>3. Combine hyperspectral samples<jupyter_code>lucas.combine_hyp_samples(
    hyp_csv_path=path_to_chunk_concat,
    output_path=path_to_combined,
    verbose=verbose)<jupyter_output><empty_output><jupyter_text>4. Combine CSV and XLS files<jupyter_code>lucas.combine_csv_xls(
    hyp_csv_path=path_to_combined,
    other_csv_path=path_to_datafile,
    output_path=path_to_full,
    verbose=verbose)<jupyter_output><empty_output><jupyter_text>5. Add categories and superclasses<jupyter_code>lucas.add_categories_and_superclasses(
    input_path=path_to_full,
    output_path=path_to_final,
    verbose=verbose)<jupyter_output><empty_output><jupyter_text>6. Split dataset<jupyter_code>lucas.split_lucas_dataset(
    full_dataset_path=path_to_final,
    output_path_prefix=path_to_subsets,
    random_state=42,
    train_frac=0.8,
    val_frac=0.2,
    verbose=verbose)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07772220084647942, 'fraction_numerical': 0.013466717968449404, 'mean_word_length': 6.878787878787879, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '98888', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1010, 'n_tokens_neox': 954, 'n_words': 172, 'path': 'py/Process_LUCAS_Dataset.ipynb', 'repo_name': 'felixriese/lucas-processing', 'size': '2599'}