<jupyter_start><jupyter_text>How-to guide for Real-Time Forecasting use-case on Abacus.AI platformThis notebook provides you with a hands on environment to build a real-time forecasting model using the Abacus.AI Python Client Library.We'll be using the [Household Electricity Usage Dataset](https://s3.amazonaws.com/realityengines.exampledatasets/rtforecasting/household_electricity_usage.csv), which contains data about electricity usage in a specified household. 1. Install the Abacus.AI library.<jupyter_code>!pip install abacusai<jupyter_output><empty_output><jupyter_text>We'll also import pandas and pprint tools for neat visualization in this notebook.<jupyter_code>import pandas as pd # A tool we'll use to download and preview CSV files
import pprint # A tool to pretty print dictionary outputs
pp = pprint.PrettyPrinter(indent=2)<jupyter_output><empty_output><jupyter_text>2. Add your Abacus.AI [API Key](https://abacus.ai/app/profile/apikey) generated using the API dashboard as follows:<jupyter_code>#@title Abacus.AI API Key

api_key = ''  #@param {type: "string"}<jupyter_output><empty_output><jupyter_text>3. Import the Abacus.AI library and instantiate a client.<jupyter_code>from abacusai import ApiClient
client = ApiClient(api_key)<jupyter_output><empty_output><jupyter_text>1. Create a ProjectAbacus.AI projects are containers that have datasets and trained models. By specifying a business **Use Case**, Abacus.AI tailors the deep learning algorithms to produce the best performing model possible for your data.We'll call the `list_use_cases` method to retrieve a list of the available Use Cases currently available on the Abacus.AI platform.<jupyter_code>client.list_use_cases()<jupyter_output><empty_output><jupyter_text>In this notebook, we're going to create a real-time forecasting model using the Household Electricity Usage dataset. The 'ENERGY' use case is best tailored for this situation.<jupyter_code>#@title Abacus.AI Use Case

use_case = 'ENERGY'  #@param {type: "string"}<jupyter_output><empty_output><jupyter_text>By calling the `describe_use_case_requirements` method we can view what datasets are required for this use_case.<jupyter_code>for requirement in client.describe_use_case_requirements(use_case):
  pp.pprint(requirement.to_dict())<jupyter_output><empty_output><jupyter_text>Finally, let's create the project.<jupyter_code>real_time_project = client.create_project(name='Electricity Usage Forecasting', use_case=use_case)
real_time_project.to_dict()<jupyter_output><empty_output><jupyter_text>**Note: When feature_groups_enabled is True then the use case supports feature groups (collection of ML features). Feature groups are created at the organization level and can be tied to a project to further use it for training ML models**  2. Add Datasets to your ProjectAbacus.AI can read datasets directly from `AWS S3` or `Google Cloud Storage` buckets, otherwise you can also directly upload and store your datasets with Abacus.AI. For this notebook, we will have Abacus.AI read the datasets directly from a public S3 bucket's location.We are using one dataset for this notebook. We'll tell Abacus.AI how the dataset should be used when creating it by tagging the dataset with a special Abacus.AI **Dataset Type**.- [Household Electricity Usage Dataset](https://s3.amazonaws.com/realityengines.exampledatasets/rtforecasting/household_electricity_usage.csv) (**TIMESERIES**): This dataset contains information about electricity usage in specified households over a period of time.  Add the dataset to Abacus.AIFirst we'll use Pandas to preview the file, then add it to Abacus.AI.<jupyter_code>pd.read_csv('https://s3.amazonaws.com/realityengines.exampledatasets/rtforecasting/household_electricity_usage.csv')<jupyter_output><empty_output><jupyter_text>Using the Create Dataset API, we can tell Abacus.AI the public S3 URI of where to find the datasets. We will also give each dataset a Refresh Schedule, which tells Abacus.AI when it should refresh the dataset (take an updated/latest copy of the dataset).If you're unfamiliar with Cron Syntax, Crontab Guru can help translate the syntax back into natural language: [https://crontab.guru/0_12_\*_\*_\*](https://crontab.guru/0_12_*_*_*)**Note: This cron string will be evaluated in UTC time zone**<jupyter_code>real_time_dataset = client.create_dataset_from_file_connector(name='Household Electricity Usage',table_name='Household_Electricity_Usage',
                                     location='s3://realityengines.exampledatasets/rtforecasting/household_electricity_usage.csv',
                                     refresh_schedule='0 12 * * *')
datasets = [real_time_dataset]

for dataset in datasets:
    dataset.wait_for_inspection()<jupyter_output><empty_output><jupyter_text>3. Create Feature Groups and add them to your ProjectDatasets are created at the organization level and can be used to create feature groups as follows:<jupyter_code>feature_group = client.create_feature_group(table_name='real_time_forecasting',sql='SELECT * FROM Household_Electricity_Usage')<jupyter_output><empty_output><jupyter_text>Adding Feature Group to the project:<jupyter_code>client.add_feature_group_to_project(feature_group_id=feature_group.feature_group_id,project_id = real_time_project.project_id)<jupyter_output><empty_output><jupyter_text>Setting the Feature Group type according to the use case requirements:<jupyter_code>client.set_feature_group_type(feature_group_id=feature_group.feature_group_id, project_id = real_time_project.project_id, feature_group_type= "TIMESERIES")<jupyter_output><empty_output><jupyter_text>Check current Feature Group schema:<jupyter_code>client.get_feature_group_schema(feature_group_id=feature_group.feature_group_id)<jupyter_output><empty_output><jupyter_text>For each **Use Case**, there are special **Column Mappings** that must be applied to a column to fulfill use case requirements. We can find the list of available **Column Mappings** by calling the *Describe Use Case Requirements* API:<jupyter_code>client.describe_use_case_requirements(use_case)[0].allowed_feature_mappings
client.set_feature_mapping(project_id = real_time_project.project_id,feature_group_id= feature_group.feature_group_id, feature_name='value',feature_mapping='TARGET')
client.set_feature_mapping(project_id = real_time_project.project_id,feature_group_id= feature_group.feature_group_id, feature_name='time',feature_mapping='DATE')
client.set_feature_mapping(project_id = real_time_project.project_id,feature_group_id= feature_group.feature_group_id, feature_name='id',feature_mapping='ITEM_ID')<jupyter_output><empty_output><jupyter_text>For each required Feature Group Type within the use case, you must assign the Feature group to be used for training the model:<jupyter_code>client.use_feature_group_for_training(project_id=real_time_project.project_id, feature_group_id=feature_group.feature_group_id)<jupyter_output><empty_output><jupyter_text>Now that we've our feature groups assigned, we're almost ready to train a model!To be sure that our project is ready to go, let's call project.validate to confirm that all the project requirements have been met:<jupyter_code>real_time_project.validate()<jupyter_output><empty_output><jupyter_text>4. Train a Model For each **Use Case**, Abacus.AI has a bunch of options for training. We can call the *Get Training Config Options* API to see the available options.<jupyter_code>real_time_project.get_training_config_options()<jupyter_output><empty_output><jupyter_text>In this notebook, we'll just train with the default options, but definitely feel free to experiment, especially if you have familiarity with Machine Learning.<jupyter_code>real_time_model = real_time_project.train_model(training_config={})
real_time_model.to_dict()<jupyter_output><empty_output><jupyter_text>After we start training the model, we can call this blocking call that routinely checks the status of the model until it is trained and evaluated.<jupyter_code>real_time_model.wait_for_evaluation()<jupyter_output><empty_output><jupyter_text>**Note that model training might take some minutes to some hours depending upon the size of datasets, complexity of the models being trained and a variety of other factors**  **Checkpoint** [Optional]As model training can take an hours to complete, your page could time out or you might end up hitting the refresh button, this section helps you restore your progress:<jupyter_code>!pip install abacusai
import pandas as pd
import pprint
pp = pprint.PrettyPrinter(indent=2)
api_key = ''  #@param {type: "string"}
from abacusai import ApiClient
client = ApiClient(api_key)
real_time_project = next(project for project in client.list_projects() if project.name == 'Electricity Usage Forecasting')
real_time_model = real_time_project.list_models()[-1]
real_time_model.wait_for_evaluation()<jupyter_output><empty_output><jupyter_text>Evaluate your Model MetricsAfter your model is done training you can inspect the model's quality by reviewing the model's metrics<jupyter_code>pp.pprint(real_time_model.get_metrics().to_dict())<jupyter_output><empty_output><jupyter_text>To get a better understanding on what these metrics mean, visit our [documentation](https://abacus.ai/app/help/useCases/ENERGY/training) page.  5. Deploy Model After the model has been trained, we need to deploy the model to be able to start making predictions. Deploying a model will reserve cloud resources to host the model for Realtime and/or batch predictions.<jupyter_code>real_time_deployment = client.create_deployment(name='Electricity Usage Deployment',description='Electricity Usage Deployment',model_id=real_time_model.model_id)
real_time_deployment.wait_for_deployment()<jupyter_output><empty_output><jupyter_text>After the model is deployed, we need to create a deployment token for authenticating prediction requests. This token is only authorized to predict on deployments in this project, so it's safe to embed this token inside of a user-facing application or website.<jupyter_code>deployment_token = real_time_project.create_deployment_token().deployment_token
deployment_token<jupyter_output><empty_output><jupyter_text>6. Predict Now that you have an active deployment and a deployment token to authenticate requests, you can make the `get_forecast` API call below.This command will return a forecast under each percentile for the specified ITEM_ID. The forecast will be performed based on attributes specified in the dataset.<jupyter_code>ApiClient().get_forecast(deployment_token=deployment_token, 
               deployment_id=real_time_deployment.deployment_id, 
               query_data={"id":"MT_001"})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07417094815506772, 'fraction_numerical': 0.0029892573563755254, 'mean_word_length': 7.203831417624521, 'pattern_counts': {'":': 1, '<': 101, '<?xml version=': 0, '>': 101, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '13153', 'license': 'MIT', 'n_tokens_mistral': 3177, 'n_tokens_neox': 3083, 'n_words': 968, 'path': 'Use Cases/Real_time_Forecasting_Notebook.ipynb', 'repo_name': 'Ankit-RealityEngines/notebooks', 'size': '10705'}