<jupyter_start><jupyter_text>Feast Basic Customer Transactions Example This is a minimal example of using Feast. The point is to show users how to get data into Feast and how to retrieve features for online serving and model training.In this example we will1. Create a synthetic customer feature dataset2. Register a feature set to represent these features in Feast3. Ingest these features into Feast4. Create a feature query and retrieve online feature data5. Create a feature query and retrieve historical feature data  0. Configuration<jupyter_code>import os

# Feast Core acts as the central feature registry
FEAST_CORE_URL = os.getenv('FEAST_CORE_URL', 'localhost:6565')

# Feast Online Serving allows for the retrieval of real-time feature data
FEAST_ONLINE_SERVING_URL = os.getenv('FEAST_ONLINE_SERVING_URL', 'localhost:6566')

# Feast Batch Serving allows for the retrieval of historical feature data
FEAST_HISTORICAL_SERVING_URL = os.getenv('FEAST_HISTORICAL_SERVING_URL', 'localhost:6567')<jupyter_output><empty_output><jupyter_text>1. Install Feast SDK Install from PyPi<jupyter_code>!pip install feast<jupyter_output><empty_output><jupyter_text>2. Import necessary modules<jupyter_code>import pandas as pd
import numpy as np
from pytz import timezone, utc
from feast import Client, FeatureSet, Entity, ValueType
from feast.serving.ServingService_pb2 import GetOnlineFeaturesRequest
from feast.types.Value_pb2 import Value as Value
from google.protobuf.duration_pb2 import Duration
from datetime import datetime, timedelta
from random import randrange
import random<jupyter_output><empty_output><jupyter_text>3. Configure Feast services and connect the Feast clientConnect to Feast Core and Feast Online Serving<jupyter_code>client = Client(core_url=FEAST_CORE_URL, serving_url=FEAST_ONLINE_SERVING_URL)<jupyter_output><empty_output><jupyter_text>4. Create customer features Here we will create customer features for 5 customers over a month. Each customer will have a set of features for every day.<jupyter_code>days = [datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0).replace(tzinfo=utc) \
        - timedelta(day) for day in range(10)][::-1]

customers = [1001, 1002, 1003, 1004, 1005]
customer_features = pd.DataFrame(
    {
        "datetime": [day for day in days for customer in customers], # Datetime is required
        "customer_id": [customer for day in days for customer in customers], # Customer is the entity
        "daily_transactions": [np.random.rand() * 10 for _ in range(len(days) * len(customers))], # Feature 1
        "total_transactions": [np.random.randint(100) for _ in range(len(days) * len(customers))], # Feature 2
    }
)

print(customer_features.head(20))<jupyter_output>datetime  customer_id  daily_transactions  \
0  2020-06-17 00:00:00+00:00         1001            4.900417   
1  2020-06-17 00:00:00+00:00         1002            7.440329   
2  2020-06-17 00:00:00+00:00         1003            4.224760   
3  2020-06-17 00:00:00+00:00         1004            5.482722   
4  2020-06-17 00:00:00+00:00         1005            2.200896   
5  2020-06-18 00:00:00+00:00         1001            8.173628   
6  2020-06-18 00:00:00+00:00         1002            3.164327   
7  2020-06-18 00:00:00+00:00         1003            7.248387   
8  2020-06-18 00:00:00+00:00         1004            9.274397   
9  2020-06-18 00:00:00+00:00         1005            7.846449   
10 2020-06-19 00:00:00+00:00         1001            9.028874   
11 2020-06-19 00:00:00+00:00         1002            5.140390   
12 2020-06-19 00:00:00+00:00         1003            4.537877   
13 2020-06-19 00:00:00+00:00         1004            6.797491   
14 2020-06-19 00:00:00+00[...]<jupyter_text>5. Create feature set for customer features Now we will create a feature set for these features. Feature sets are essentially a schema that representfeature values. Feature sets allow Feast to both identify feature values and their structure. The following feature set contains no features yet.<jupyter_code>customer_fs = FeatureSet(
    "customer_transactions",
    entities=[Entity(name='customer_id', dtype=ValueType.INT64)]
)<jupyter_output><empty_output><jupyter_text>Here we are automatically inferring the schema from the provided dataset. The two features from the dataset will be added to the feature set<jupyter_code>customer_fs.infer_fields_from_df(customer_features, replace_existing_features=True)<jupyter_output>Feature daily_transactions (ValueType.DOUBLE) added from dataframe.
Feature total_transactions (ValueType.INT64) added from dataframe.<jupyter_text>6. Register feature set with Feast Core The apply() method will register the provided feature set with Feast Core (the feature registry).<jupyter_code>client.apply(customer_fs)
customer_fs = client.get_feature_set("customer_transactions")
print(customer_fs)<jupyter_output>{
  "spec": {
    "name": "customer_transactions",
    "entities": [
      {
        "name": "customer_id",
        "valueType": "INT64"
      }
    ],
    "features": [
      {
        "name": "daily_transactions",
        "valueType": "DOUBLE"
      },
      {
        "name": "total_transactions",
        "valueType": "INT64"
      }
    ],
    "source": {
      "type": "KAFKA",
      "kafkaSourceConfig": {
        "bootstrapServers": "localhost:9094",
        "topic": "feast-features"
      }
    },
    "project": "default"
  },
  "meta": {
    "createdTimestamp": "2020-06-26T12:27:17Z",
    "status": "STATUS_PENDING"
  }
}<jupyter_text>7. Ingest data into Feast for a feature set Next we will ingest/load data into Feast. This process populates all registered stores (BigQuery, Redis) with your feature data.<jupyter_code>client.ingest("customer_transactions", customer_features)<jupyter_output>Waiting for feature set to be ready for ingestion...<jupyter_text>8. Retrieve online features The example below retrieves online features for a single customer: "1001". Retrieval of features is not limited to a single feature set. Users can provide any features as long as they are present on the provided entities.<jupyter_code>online_features = client.get_online_features(
    feature_refs=[
        "daily_transactions",
        "total_transactions",
    ],
    entity_rows=[
        {
            "customer_id": 1001
        }
    ],
)
print(online_features.field_values)<jupyter_output>[fields {
  key: "customer_id"
  value {
    int64_val: 1001
  }
}
fields {
  key: "daily_transactions"
  value {
    double_val: 0.12021977894872915
  }
}
fields {
  key: "total_transactions"
  value {
    int64_val: 0
  }
}
statuses {
  key: "customer_id"
  value: PRESENT
}
statuses {
  key: "daily_transactions"
  value: PRESENT
}
statuses {
  key: "total_transactions"
  value: PRESENT
}
]<jupyter_text>To retrieve the fields in dictionary format, we can utilize `to_dict()` method.<jupyter_code>online_features.to_dict()<jupyter_output><empty_output><jupyter_text>9. Retrieve training features (Requires Google Cloud Platform)In order to retrieve historical feature data, the user must provide an `entity_rows` DataFrame. This DataFrame contains a combination of timestamps and entities (customers, in this case).The timestamps correlate to the event_time that a prediction needs to be made. At each one of these points in time you need to know the "current" feature values.We will randomly generate timestamps over the last 5 days and assign `customer_ids` to them.When these entity rows are sent to the Feast Serving API to retrieve feature values, along with a list of feature ids, Feast is then able to attach the correct feature values to each entity row. It will join the correct feature values at each point in time for each entity onto these entity rows.<jupyter_code>event_timestamps = [datetime.utcnow().replace(tzinfo=utc) - timedelta(days=randrange(5), hours=randrange(24), minutes=randrange(60)) for day in range(30)]

entity_rows = pd.DataFrame(
    {
        "datetime": event_timestamps,
        "customer_id": [customers[idx % len(customers)] for idx in range(len(event_timestamps))],
    }
)

print(entity_rows.head(10))<jupyter_output>datetime  customer_id
0 2020-06-15 10:35:10.918716+00:00         1001
1 2020-06-15 14:00:10.918758+00:00         1002
2 2020-06-17 08:59:10.918767+00:00         1003
3 2020-06-13 16:51:10.918774+00:00         1004
4 2020-06-17 06:14:10.918780+00:00         1005
5 2020-06-17 14:33:10.918786+00:00         1001
6 2020-06-14 23:15:10.918792+00:00         1002
7 2020-06-15 11:25:10.918798+00:00         1003
8 2020-06-18 09:04:10.918804+00:00         1004
9 2020-06-16 10:27:10.918810+00:00         1005<jupyter_text>10. Retrieve historical/batch features Next we will create a new client object, but this time we will configure it to connect to the Feast Batch Serving. This service will allow us to retrieve historical feature data.<jupyter_code>batch_client = Client(core_url=FEAST_CORE_URL, serving_url=FEAST_HISTORICAL_SERVING_URL)<jupyter_output><empty_output><jupyter_text>By calling the `get_historical_features` method we are able to retrieve a `job` object. This object can be used to retrieve the resulting training dataset that is exported by Feast. The dataset that is returned will contain feature values for each entity and timestamp combination in `entity_rows`.<jupyter_code>job = batch_client.get_historical_features(
                            feature_refs=[
                                "daily_transactions",
                                "total_transactions",
                               ],
                            entity_rows=entity_rows
                         )<jupyter_output><empty_output><jupyter_text>Once the job is complete, it is possible to retrieve the exported data (from Google Cloud Storage) and load it into memory as a Pandas Dataframe.<jupyter_code>df = job.to_dataframe()
print(df.head(50))<jupyter_output>event_timestamp  customer_id  daily_transactions  \
0  2020-06-13 23:45:10.918874+00:00         1001            1.879220   
1  2020-06-18 12:11:10.918845+00:00         1001            5.122846   
2  2020-06-17 20:46:10.918903+00:00         1001            2.145294   
3  2020-06-18 02:50:10.918816+00:00         1001            5.122846   
4  2020-06-15 10:35:10.918716+00:00         1001            5.758472   
5  2020-06-17 14:33:10.918786+00:00         1001            2.145294   
6  2020-06-14 23:15:10.918792+00:00         1002            5.467141   
7  2020-06-14 07:22:10.918851+00:00         1002            5.467141   
8  2020-06-17 23:40:10.918880+00:00         1002            3.338614   
9  2020-06-15 14:00:10.918758+00:00         1002            4.921264   
10 2020-06-15 18:22:10.918909+00:00         1002            4.921264   
11 2020-06-16 21:10:10.918822+00:00         1002            1.838186   
12 2020-06-18 05:47:10.918886+00:00         1003            2.70[...]<jupyter_text>The training dataset is staged on Google Cloud Storage and can be accessed directly if it is too large to load into memory<jupyter_code>job.get_avro_files()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08817689530685921, 'fraction_numerical': 0.12102888086642599, 'mean_word_length': 3.3251366120218577, 'pattern_counts': {'":': 26, '<': 58, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '72136', 'license': 'Apache-2.0', 'n_tokens_mistral': 4336, 'n_tokens_neox': 3500, 'n_words': 1020, 'path': 'examples/basic/basic.ipynb', 'repo_name': 'imjuanleonard/feast', 'size': '11080'}