<jupyter_start><jupyter_text>Unsupervised agent demonstration Here we demonstrate the use of a filesystem based `DirectoryAgent,`using the `NMFCompanion` agent that follows the `tell`--`report`--`ask` paradigm.<jupyter_code>from bnl_ml_examples.unsupervised.agent import NMFCompanion
from bnl_ml_examples.utils.filesystem import ObservationalDirectoryAgent
from pathlib import Path
import numpy as np<jupyter_output><empty_output><jupyter_text>First we set up a linspace that contains the ROI for the measurement,and a data transformation that takes the complete measurement (Q, I(Q))and trims it to the region on interest.<jupyter_code>x_linspace = np.linspace(0, 10, 545)


def data_transform(data):
    """Trim data to Region of interest"""
    x, y = data
    idx_min = (
        np.where(x < x_linspace[0])[0][-1] if len(np.where(x < x_linspace[0])[0]) else 0
    )
    idx_max = (
        np.where(x > x_linspace[-1])[0][0]
        if len(np.where(x > x_linspace[-1])[0])
        else len(y)
    )
    return x[idx_min:idx_max], y[idx_min:idx_max]<jupyter_output><empty_output><jupyter_text>Next we instatiate the NMF companion. This agent takes care ofthe data normalization, the model construction using the scikit-learn backend,and the instructions for what to do when told about data, or asked for a report.In this case, report constructs a plot that encompases the NMF decompositionrelated to the independent variable of the experiment (temperature).<jupyter_code>companion = NMFCompanion(4, x_linspace=x_linspace)<jupyter_output><empty_output><jupyter_text>Following this, we construct the `DirectoryAgent.`This takes the place of a Bluesky callback or service, when dealing withlocal file streams instead of document streams.It has some extra options to draw the experimental independent variableout of a path manipulation, as users will commonly store essential metadata in filenames. This agent will pay attention to a tmp directory in the example_data folder.<jupyter_code>directory = Path("../") / "example_data" / "tmp"
directory.mkdir(exist_ok=True)

da = ObservationalDirectoryAgent(
    companion,
    directory,
    path_spec="*.chi",
    data_transform=data_transform,
    independent_from_path=lambda path: float(path.name.split("_")[-2][1:-1]),
)
da.load_dir()<jupyter_output><empty_output><jupyter_text>Finally, we spin the agent. This is a continuous operation towatch and react to the target directory (`example_data/tmp`).After executing the following cell, go ahead and copy some or all of the data from`example_data/NaCl_CrCl3_pdf_ramp` into `example_data/tmp`.The spin will only run for 2 minutes, before stopping.<jupyter_code>da.spin(sleep_delay=5, timeout=120)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08954128440366972, 'fraction_numerical': 0.009908256880733945, 'mean_word_length': 5.537170263788969, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '357827', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 849, 'n_tokens_neox': 823, 'n_words': 270, 'path': 'example_scripts/run_unsupervised.ipynb', 'repo_name': 'stuartcampbell/pub-ML_examples', 'size': '2725'}