<jupyter_start><jupyter_text>Introduction to MindSphere Analytics APIs - TrendPrediction This demo will use the mindconnect-nodejs CLI to call the mindsphere APIs. The information about the installation and usage of CLI:https://opensource.mindsphere.io/docs/mindconnect-nodejs/cli/index.htmlYou will need to get and configure the service credentials to use this notebook. Here is a description how to acquire and setup the service credentials:https://opensource.mindsphere.io/docs/mindconnect-nodejs/li/setting-up-the-cli.htmlThis is only a **demo** for the Trend Prediction API of the mindsphere and not a serious data analysis :)https://developer.mindsphere.io/apis/analytics-trendprediction/api-trendprediction-overview.html Setup the mindconnect-nodejs CLIplease change the line below to the passkey you were using when setting up the service credentials:<jupyter_code>passkey = "<PASSWORD>"<jupyter_output><empty_output><jupyter_text>Trend Prediction with CLI```mc trend-prediction``` CLI command:<jupyter_code>!mc trend-prediction --help<jupyter_output>Usage: mc trend-prediction|tp [options]

perform trend prediction (linear/polynomial) @

Options:
  -f, --file <timeseries>                                      timeseries file (default: "timeseries-sample.json")
  -m, --mode [train|predict|trainandpredict|list|read|delete]  mode see @ Additional Documentation
  -o, --output <output>                                        output variables
  -i, --input <input>                                          input variables (comma separated)
  -e, --modelid <modelid>                                      modelid of the stored model for prediction
  -r, --predict <predict>                                      regression parameters for prediction (comma separated)
  -c, --predictfile <predictfile>                              regression parameters for prediction as timeseries
  -d, --degree [degree]                                        degree for linear / polynomial regression  (default: 1)
  -y, --retry <number>                                [...]<jupyter_text>Setup the jupyter notebook<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import json
plt.rcParams['figure.figsize'] = [8, 8]<jupyter_output><empty_output><jupyter_text>Industrial Pump - TimeSeries DataWe will use the downloaded industrial pump data from mindsphere to try to fit the model which can predict flow problems. The mindsphere demo application is available athttps://dreamforce.mindconnect.rocksThis application can be used without mindsphere credentials. * username: <EMAIL>* password: <PASSWORD>!Following data was recorded over the time:- **Flow** - Water flow through the pump- **MotorCurrent** Motor Current- **PressureIn** Inlet pressure- **PressureOut** Outlet pressure- **StuffingBoxTemperature** The temperature of the pump![pumpschema](./images/Pump-Station-Scheme.jpg)The historic data was downloaded from MindSphere and  is available in the **data** directory of this notebook.  First look at the dataThe first file already has a typical behavior of the malfunctioning pump. The stuffing box temperature is increased and the flow is dropping.<jupyter_code>data = pd.read_json("./data/PUMP-011/PumpData/PumpData_0.json")
data.plot(x="_time")<jupyter_output><empty_output><jupyter_text>Training the modelsWe will use this data to train several polynomial regression models. Essentially we are trying to find a good fit for the **flow = f (MotorCurrent,PressureOut,StuffingBoxTemperature,PressureIn)** functionand we will try several polynomial functions (from linear to 4th-degree polynomial)The code below will use the CLI to call MindSphere trend-prediction API. We are using the content of the first file as the input and are creating 4 models using the CLI.See also : https://developer.mindsphere.io/apis/analytics-trendprediction/api-trendprediction-basics.html<jupyter_code>models = []
for degree in range(1,5):
    id = !mc trend-prediction \
        --mode train \
        --file "./data/PUMP-011/PumpData/PumpData_0.json" \
        --input "MotorCurrent,PressureOut,StuffingBoxTemperature,PressureIn" --output "Flow" \
        --degree $degree \
        --passkey $passkey
    
    modelid = str(id).split(' ')[4]
    models.append(modelid)


models<jupyter_output><empty_output><jupyter_text>Predicting the flow using trained modelsThe function below is using the CLI in the --predict mode to calculate the predictions with all models. We are passing the **modelid** parameter from the list above in order to use that trained model to invoke the prediction.<jupyter_code>def predictWith (models, filename):
    results = []

    for modelid in models:
        predictionList = !mc trend-prediction --mode predict --modelid $modelid \
        --input "MotorCurrent,PressureOut,StuffingBoxTemperature,PressureIn" --output "Flow" \
        --predictfile $filename --passkey $passkey

        result = ''.join(str(e) for e in predictionList)
        df = pd.read_json(result);
        prediction = pd.DataFrame.from_dict(df["timeSeries"][0])
        prediction["_time"] = pd.to_datetime(prediction["_time"])
        results.append(prediction)
    
    return results<jupyter_output><empty_output><jupyter_text>Testing the modelsNow is the time to test the trained models. We will apply each model to the json data stored in the **PumpData_\*.json** files and calculate the cumulative error of the prediction. The produced images show the discrepancies between the actual value and our prediction. The curves are close to each other but it is already obvious that some curves are closer to the prediction than the others.* Flow - Measured Flow (actual measured values)* Flow_1 - Linear regression model* Flow_2 - Quadratic regression model* Flow_3 - Cubic regression model* Flow_4 - Quartic (4th-degree polynomial) regression modelWe will also calculate the cumulative error (the distance of the prediction to the actual value) and pick the model with the smallest cumulative error.<jupyter_code>errors = {"degree_1": 0, "degree_2": 0, "degree_3": 0, "degree_4": 0}

filenames = [
    './data/PUMP-011/PumpData/PumpData_0.json',
    './data/PUMP-011/PumpData/PumpData_1.json',
    './data/PUMP-011/PumpData/PumpData_2.json']

# This uses all files for training but takes a lot of time...
# import glob
# filenames = enumerate(glob.glob('./data/PUMP-011/PumpData/PumpData_*.json'))

for fileindex, filename in enumerate(filenames):
    original = pd.read_json(filename)
    graphdata = original[['_time','Flow']]
    results = predictWith(models, filename)
    
    for idx, prediction in enumerate(results):
        graphdata = pd.merge(graphdata, prediction, on="_time", suffixes=("", "_" + str(idx+1) ));
        error = abs(sum(graphdata["Flow"]) - sum(graphdata["Flow_"+str(idx+1)]))
        errors["degree_" + str(idx+1)] += error
    
    if fileindex < 3:
        graphdata.plot(x="_time", title=filename); 
        
        
errors = pd.DataFrame.from_dict(errors, orient='index', columns=["error"])
errors.plot(kind="bar", title="cumulated errors", logy=True, table=True, xticks=[])<jupyter_output><empty_output><jupyter_text>Finding the best modelThe best model is the one with smallest cumulative error, in our case it is the cubic model which predicts the outcome with the smallest error.The model can be saved and executed also without mindsphere for example on the industrial edge.<jupyter_code>bestmodelid = models[int(str(errors.idxmin().values[0]).split("_")[1])] # The best model is the one with smallest error
model = !mc trend-prediction --mode get --modelid $bestmodelid --passkey $passkey

print (model.pop()) # remove the CLI message at the end 
modeljson = ''.join (str(e) for e in model);
import json 
print (json.dumps(json.loads(modeljson), indent=2))<jupyter_output>Sucessfully retrieved model with 8c27fa81-1064-43cf-99c9-1dea37b7c7cc id.
{
  "id": "8c27fa81-1064-43cf-99c9-1dea37b7c7cc",
  "metadataConfiguration": {
    "outputVariable": {
      "entityId": "cli-trend-prediction",
      "propertySetName": "cli-trend-prediction",
      "propertyName": "Flow"
    },
    "inputVariables": [
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "MotorCurrent"
      },
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "PressureOut"
      },
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "StuffingBoxTemperature"
      },
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "PressureIn"
      }
    ]
  },
  "intercept": 65724576.0827809,
  "coefficients": [...]<jupyter_text>Comparing with other models Linear modelHere is the linear model for comparison:<jupyter_code>linearmodelid = models[0] 
linearmodel = !mc trend-prediction --mode get --modelid $linearmodelid --passkey $passkey

print (linearmodel.pop()) # remove the CLI message at the end 
modeljson = ''.join (str(e) for e in linearmodel);
print (json.dumps(json.loads(modeljson), indent=2))<jupyter_output>Sucessfully retrieved model with c695ed5d-08bc-47b1-8e4e-fa0a3f29d242 id.
{
  "id": "c695ed5d-08bc-47b1-8e4e-fa0a3f29d242",
  "metadataConfiguration": {
    "outputVariable": {
      "entityId": "cli-trend-prediction",
      "propertySetName": "cli-trend-prediction",
      "propertyName": "Flow"
    },
    "inputVariables": [
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "MotorCurrent"
      },
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "PressureOut"
      },
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "StuffingBoxTemperature"
      },
      {
        "entityId": "cli-trend-prediction",
        "propertySetName": "cli-trend-prediction",
        "propertyName": "PressureIn"
      }
    ]
  },
  "intercept": -93.36314165683584,
  "coefficients"[...]<jupyter_text>Cleaning upThe ```mc trend-prediction --mode delete``` command can delete the created models. The code below deletes all created models:<jupyter_code>for modelid in models:
    !mc trend-prediction --mode delete --modelid $modelid --passkey $passkey

print("Done, thank you for testing mindsphere trend prediction API")<jupyter_output>Sucessfully deleted model with c695ed5d-08bc-47b1-8e4e-fa0a3f29d242 id
Sucessfully deleted model with 5dd3868a-8f16-4400-ac57-4c83d1fdec88 id
Sucessfully deleted model with 3a2a0c9f-bb89-40a2-9673-cea29853a8ee id
Sucessfully deleted model with 8c27fa81-1064-43cf-99c9-1dea37b7c7cc id
Done, thank you for testing mindsphere trend prediction API<jupyter_text>Cleaning up You may uncomment the code below to delete all models in the MindSphere. **This will delete all models! Use with care!**<jupyter_code># deleteall = !mc trend-prediction --mode list --passkey $passkey

# for s in deleteall[1:]:
#    id = str(s).split(' ')[0]
#    !mc trend-prediction --mode delete --modelid $id --passkey $passkey<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10926743159752869, 'fraction_numerical': 0.02091791703442189, 'mean_word_length': 3.7094763092269325, 'pattern_counts': {'":': 45, '<': 52, '<?xml version=': 0, '>': 51, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '417504', 'license': 'MIT', 'n_tokens_mistral': 3537, 'n_tokens_neox': 3376, 'n_words': 994, 'path': 'trend-prediction.ipynb', 'repo_name': 'mindsphere/analytics-examples', 'size': '11351'}