<jupyter_start><jupyter_text>About these notebooksThese three notebooks can be run to replicate the dataset, XGBoost model and results from the tree-based structure prediction. The notebooks save and load data in local folders in the Metalfinder_restructured folder. The first notebook simulates the metal clusters and PDFs. The second notebook trains an XGBoost model. The third notebook tests the XGBoost model and here the model can be used to predict the structure of experimental PDFs. Simulate DataThis notebook simulates clusters and pair distribution function datasets**How to use:** Run the cells underneath from top to bottom The first cell imports packages and functions from the backend. The python packages Atomic Simulation Environment (ASE)*(1)* and DiffPy-CMI*(2)* are used to simulate data.The second cell simulates metal clusters of seven different structure types (MMNPs) and saves them in a custom folder in /xyz_files/. The third cell simulates Atomic Pair Distribution functions from the metal clusters and saves them in a custom folder in /PDF_datasets/. This dataset can be loaded in "2_Training.ipynb", where a Gradient Boosted Decision Tree model can be trained to predict the MMNPs from a PDF.  Import packages and functions from backend<jupyter_code>import sys, os, os.path, h5py, time, shutil, mendeleev
import pandas as pd
import numpy as np

from os import walk
from tqdm import tqdm
from shutil import copyfile
from scipy import spatial

from multiprocessing import Pool
from diffpy.srreal.pdfcalculator import DebyePDFCalculator, PDFCalculator
from diffpy.Structure import loadStructure, Structure, Lattice, Atom

from diffpy.Structure import Structure
from diffpy.Structure import Atom
from diffpy.Structure import loadStructure, Structure, Lattice

from ase.data import covalent_radii, atomic_numbers, chemical_symbols
from ase.cluster.cubic import FaceCenteredCubic, BodyCenteredCubic, SimpleCubic
from ase.lattice.hexagonal import Hexagonal, HexagonalClosedPacked, Graphite
from ase.cluster.decahedron import Decahedron
from ase.cluster.icosahedron import Icosahedron
from ase.cluster.octahedron import Octahedron

sys.path.append("Backend")

from simulate_data import get_simulate_data_backend, PDF_generator, sort_filenames, make_df_parallelized, PDF_from_index, reduce_mem_usage, save_PDFs, make_data, make_data_SC_FCC_BCC, make_data_HCP, make_data_Icosahedron, make_data_Decahedron, make_data_Octahedron, new_structure_checker, remove_same_structures, structure_maker, remove_dub, simulate_clusters   
PDF_obj, sorted_filenames_flat, xyz_path = get_simulate_data_backend() #Put the rest of the returned stuff in later<jupyter_output><empty_output><jupyter_text>Simulate metal clustersThis simulates metal clusters of different sizes and structure types, then checks for and deletes duplicates. The metal clusters are saved in a specified folder_name in the "xyz_files/" folder.<jupyter_code>folder_name = "natoms200"
simulate_clusters(folder_name, maximum_atoms = 200, minimum_atoms = 6)<jupyter_output><empty_output><jupyter_text>Simulate PDF dataset from metal clustersThis simulates atomic pair distribution functions from each metallic nanoparticle cluster. The size of the dataset can be controlled with train_multip and valtest_multip. train_multip determines how many pair distributions are simulated from each metal cluster in the training set. valtest_multip is similar, but with the validation and test set. In the article, train_multip = 100 and valtest_multip = 15 is used. threads specifies how many CPU cores are used.<jupyter_code>#Set seeds
import torch, random
import pytorch_lightning as pl
import numpy as np
seed = 37
torch.manual_seed(seed)
pl.seed_everything(seed)
np.random.seed(seed)
random.seed(seed)
folder_name = "natoms200"
data_dir = "xyz_files/" + folder_name + "/"

threads = 64
X_train, y_train, X_val, y_val, X_test, y_test = make_df_parallelized(sorted_filenames_flat, train_multip = 100, valtest_multip = 15, n_processes = threads, xyz_path = data_dir)<jupyter_output>Making PDF dataset from the clusters in xyz_files/natoms200/.
The training set will have 100 PDFs from each cluster and the validation and test set will have 15 PDFs from each cluster
Multithreading: 64 processes<jupyter_text>Save PDF datasetThis saves the datasets in the specified folder in "PDF_datasets/". These datasets can be loaded in 2_Training.ipynb where an XGBoost model can be trained on the datasets.<jupyter_code>folder_name = "big_seed37"
save_PDFs(folder_name, X_train, y_train, X_val, y_val, X_test, y_test)<jupyter_output>PDF dataset saved to PDF_datasets/big_seed37/
{'fraction_non_alphanumeric': 0.04882263987902355, 'fraction_numerical': 0.009289263339814214, 'mean_word_length': 6.419871794871795, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '533173', 'license': 'Apache-2.0', 'n_tokens_mistral': 1386, 'n_tokens_neox': 1300, 'n_words': 532, 'path': '1_Simulate_Data.ipynb', 'repo_name': 'AndySAnker/MetalFinder', 'size': '4629'}