<jupyter_start><jupyter_text>Model Inference for New DataAfter training a model on historical data, most common next step is inferering the outcome for the new data. MultiViz Analytics Engine (MVG) supports this by providing a feature called `ApplyModel`. To invoke this feature a successful analysis is required. Then, the `request_id` of that anaysis is used to refer to the model and apply it on any range of data on the same source or any source that has the same columns (or channels). This example using data from the Iris dataset illustrates how this process can be achieved.  Setup the connection<jupyter_code>import os
from mvg import MVG, plotting
from sklearn.datasets import load_iris
import pandas as pd
ENDPOINT = "http://api.beta.multiviz.com"
# Replace by your own Token
TOKEN = os.environ["TEST_TOKEN"]
session = MVG(ENDPOINT, TOKEN)<jupyter_output><empty_output><jupyter_text>Load the dataThe data from Iris dataset needs to be wrangled to conform to the format used by MVG. The data is in the form of a 2D numpy array which needs to be converted to a dictionary where each key holds the name of a column (in this case features of the plants) and the corresponding values.<jupyter_code>dataset = load_iris()
column_names = dataset["feature_names"]
column_names
data = {}
for idx, name in enumerate(column_names):
    data[name] = list((dataset["data"][:,idx]))<jupyter_output><empty_output><jupyter_text>Note that this dataset is not a timeseries dataset. It is chosen because of its availability in the common python packages like sklearn. However, it is sufficient for demonstration purposes of the `ApplyModel` feature. MVG requires timestamps for each datapoint to be provided as well. We use dummy timestamps from 0 to 149.<jupyter_code>data["timestamp"] = list(range(len(dataset["data"])))<jupyter_output><empty_output><jupyter_text>Create source and measurementsA tabular source with these measurements needs to be created for the model to be built.<jupyter_code>session.create_tabular_source(sid="iris", columns=column_names, meta={}, exist_ok=True)
session.create_tabular_measurement(sid="iris", data=data, meta={}, exist_ok=True)<jupyter_output><empty_output><jupyter_text>The three classes in the data are indicated in `dataset["target"]`. Data is divided into three equal parts, each with 50 samples for each class.<jupyter_code>dataset["target"]<jupyter_output><empty_output><jupyter_text>Build the modelTo demonstrate `ApplyModel` as a feature, we train a model on the middle 100 samples and then apply the trained model on the entire data.<jupyter_code>request_data = session.request_analysis(
    sid="iris", 
    feature="ModeId",
    start_timestamp=25, 
    end_timestamp=124)
request_id = request_data["request_id"]<jupyter_output><empty_output><jupyter_text>Helper function `wait_for_analysis` allows the possibility of waiting for the termination a group of analyses jobs.<jupyter_code>session.wait_for_analyses(request_id_list=[request_id])
mode_output = session.get_analysis_results(request_id=request_id)<jupyter_output><empty_output><jupyter_text>Review the resultsWe can plot the result of the analysis that shows the three distinct modes.<jupyter_code>mode_output["results"].pop("mode_info", None)
mode_output["results"].pop("mode_probabilities", None)
plotting.modes_over_time(pd.DataFrame(mode_output["results"]), request_id)<jupyter_output><empty_output><jupyter_text>Comparing the resulting labels with the ground-truth in the original iris dataset shows that the borders are identified precisely. Note that `ModeId` feature sorts the ID of modes in a way that the ID for the first encountered mode is reported as 0, the second mode as 1, and so on. This happens to be the case in the iris dataset as well but for other datasets sorting of modes needs to be done before comparison.<jupyter_code>all(mode_output["results"]["labels"] == dataset["target"][25:125])<jupyter_output><empty_output><jupyter_text>Apply the model on the whole datasetNow, we apply the model on the entire dataset. To that end, we need to call `ApplyModel` feature and provide the `request_id` of the analysis that contains the model. Note that running `ApplyModel` is much faster than `ModeId`.<jupyter_code>request_data_am = session.request_analysis(
    sid="iris", 
    feature="ApplyModel",
    parameters={"model_ref": request_id}
)
request_id_am = request_data_am["request_id"]
session.wait_for_analyses(request_id_list=[request_id_am])
am_output = session.get_analysis_results(request_id=request_id_am)<jupyter_output><empty_output><jupyter_text>Review the results of the inferenceComparing the results, show that the first 25 and last 25 elements of the data that was excluded while building the model in earlier steps are now correctly classified.<jupyter_code>all(am_output["results"]["labels"]==dataset["target"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07954079540795408, 'fraction_numerical': 0.005330053300533005, 'mean_word_length': 6.818910256410256, 'pattern_counts': {'":': 1, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '596255', 'license': 'Apache-2.0', 'n_tokens_mistral': 1413, 'n_tokens_neox': 1370, 'n_words': 525, 'path': 'docs/source/content/examples/8-inference.ipynb', 'repo_name': 'vnadhan/mvg', 'size': '4878'}