<jupyter_start><jupyter_text>1 Discovering Butterfree - Feature Set BasicsWelcome to **Discovering Butterfree** tutorial series!This first tutorial will cover some basics of Butterfree library and you learn how to create your first feature set :rocket: :rocket:Before diving into the tutorial make sure you have a basic understanding of these main data concepts: **features**, **feature sets** and the **"Feature Store Architecture"**, you can read more about this [here](). Library Basics:Buterfree's main objective is to make feature engineering easy. The library provides a high-level API for declarative feature definitions. But behind these abstractions, Butterfree is essentially an **ETL (Extract - Transform - Load)** framework, so this reflects in terms of the organization of the project. Extract`from butterfree.extract import ...`Module with the entities responsible for extracting data into the pipeline. The module provides the following tools:* `readers`: data connectors. Currently Butterfree provides readers for files, tables registered in Spark Hive metastore, and Kafka topics.* `pre_processing`: a utility tool for making some transformations or re-arrange the structure of the reader's input data before the feature engineering.* `source`: a composition of `readers`. The entity responsible for merging datasets coming from the defined readers into a single dataframe input for the `Transform` stage. Transform`from butterfree.transform import ...`The main module of the library, responsible for feature engineering, in other words, all the transformations on the data. The module provides the following main tools:* `features`: the entity that defines what a feature is. Holds a transformation and metadata about the feature.* `transformations`: provides a set of components for transforming the data, with the possibility to use Spark native functions, aggregations, SQL expressions and others. * `feature_set`: an entity that defines a feature set. Holds features and the metadata around it. Load`from butterfree.load import ...`The module is responsible for saving the data in some data storage. The module provides the following tools:* `writers`: provide connections to data sources to write data. Currently Butterfree provides ways to save data on S3 registered as tables Spark Hive metastore and to Cassandra DB.* `sink`: a composition of writers. The entity responsible for triggering the writing jobs on a set of defined writers PipelinesPipelines are responsible for integrating all other modules (`extract`, `transform`, `load`) in order to define complete ETL jobs from source data to data storage destination.`from butterfree.pipelines import ...`* `feature_set_pipeline`: defines an ETL pipeline for creating feature sets.  Example:Simulating the following scenario:- We want to create a feature set with features about houses for rent (listings).- We are interested in houses only for the **Kanto** region.We have two sets of data:- Table: `listing_events`. Table with data about events of house listings.- File: `region.json`. Static file with data about the cities and regions.Our desire is to have result dataset with the following schema:| id | timestamp | rent | rent_over_area | bedrooms | bathrooms | area | bedrooms_over_area | bathrooms_over_area | latitude | longitude | h3 | city | region | - | - | - | - | - | - | - | - | - | - | - | - | - | - || int | timestamp | float | float | int | int | float | float | float | double | double | string | string | string |For more information about H3 geohash click [here](https://h3geo.org/docs/)The following code blocks will show how to generate this feature set using Butterfree library:<jupyter_code># setup spark
from pyspark import SparkContext, SparkConf
from pyspark.sql import session

conf = SparkConf().set('spark.driver.host','127.0.0.1')
sc = SparkContext(conf=conf)
spark = session.SparkSession(sc)
# fix working dir
import pathlib
import os

path = os.path.join(pathlib.Path().absolute(), '../..')
os.chdir(path)
# butterfree spark client
from butterfree.clients import SparkClient

spark_client = SparkClient()<jupyter_output><empty_output><jupyter_text>Showing test data<jupyter_code>listing_evengs_df = spark.read.json(f"{path}/examples/data/listing_events.json")
listing_evengs_df.createOrReplaceTempView("listing_events")  # creating listing_events table

print(">>> listing_events table:")
listing_evengs_df.toPandas()
print(">>> region.json file:")
spark.read.json(f"{path}/examples/data/region.json").toPandas()<jupyter_output>>>> region.json file:<jupyter_text>Extract- For the extract part, we need the `Source` entity and the `FileReader` and `TableReader` for the data we have.- We need to declare a query with the rule for joining the results of the readers too.- As proposed in the problem we can filter the region dataset to get only **Kanto** region.<jupyter_code>from butterfree.extract import Source
from butterfree.extract.readers import FileReader, TableReader
from butterfree.extract.pre_processing import filter

readers = [
    TableReader(id="listing_events", table="listing_events",),
    FileReader(id="region", path=f"{path}/examples/data/region.json", format="json",).with_(
        transformer=filter, condition="region == 'Kanto'"
    ),
]

query = """
select
    listing_events.*,
    region.city,
    region.lat,
    region.lng,
    region.region
from
    listing_events
    join region
      on listing_events.region_id = region.id
"""

source = Source(readers=readers, query=query)
# showing source result

source_df = source.construct(spark_client)
source_df.toPandas()<jupyter_output><empty_output><jupyter_text>Transform- At the transform part, a set of `Feature` objects is declared.- An Instance of `FeatureSet` is used to hold the features.- A `FeatureSet` can only be created when it is possible to define a unique tuple formed by key columns and a time reference. This is an **architectural requirement** for the data. So least one `KeyFeature` and one `TimestampFeature` is needed.- Every `Feature` needs a unique name, a description, and a data-type definition.<jupyter_code>from butterfree.transform import FeatureSet
from butterfree.transform.features import Feature, KeyFeature, TimestampFeature
from butterfree.transform.transformations import SQLExpressionTransform
from butterfree.transform.transformations.h3_transform import H3HashTransform
from butterfree.constants import DataType

keys = [
    KeyFeature(
        name="id",
        description="Unique identificator code for houses.",
        dtype=DataType.BIGINT,
    )
]

# from_ms = True because the data originally is not in a Timestamp format.
ts_feature = TimestampFeature(from_ms=True)

features = [
    Feature(
        name="rent",
        description="Rent value by month described in the listing.",
        dtype=DataType.FLOAT,
    ),
    Feature(
        name="rent_over_area",
        description="Rent value by month divided by the area of the house.",
        transformation=SQLExpressionTransform("rent / area"),
        dtype=DataType.FLOAT,
    ),
    Feature(
        name="bedrooms",
        description="Number of bedrooms of the house.",
        dtype=DataType.INTEGER,
    ),
    Feature(
        name="bathrooms",
        description="Number of bathrooms of the house.",
        dtype=DataType.INTEGER,
    ),
    Feature(
        name="area",
        description="Area of the house, in squared meters.",
        dtype=DataType.FLOAT,
    ),
    Feature(
        name="bedrooms_over_area",
        description="Number of bedrooms divided by the area.",
        transformation=SQLExpressionTransform("bedrooms / area"),
        dtype=DataType.FLOAT,
    ),
    Feature(
        name="bathrooms_over_area",
        description="Number of bathrooms divided by the area.",
        transformation=SQLExpressionTransform("bathrooms / area"),
        dtype=DataType.FLOAT,
    ),
    Feature(
        name="latitude",
        description="House location latitude.",
        from_column="lat",  # arg from_column is needed when changing column name
        dtype=DataType.DOUBLE,
    ),
    Feature(
        name="longitude",
        description="House location longitude.",
        from_column="lng",
        dtype=DataType.DOUBLE,
    ),
    Feature(
        name="h3",
        description="H3 hash geohash.",
        transformation=H3HashTransform(
            h3_resolutions=[10], lat_column="latitude", lng_column="longitude",
        ),
        dtype=DataType.STRING,
    ),
    Feature(name="city", description="House location city.", dtype=DataType.STRING,),
    Feature(
        name="region",
        description="House location region.",
        dtype=DataType.STRING,
    ),
]

feature_set = FeatureSet(
    name="house_listings",
    entity="house",  # entity: to which "business context" this feature set belongs
    description="Features describring a house listing.",
    keys=keys,
    timestamp=ts_feature,
    features=features,
)
# showing feature set result

feature_set_df = feature_set.construct(source_df, spark_client)
feature_set_df.toPandas()<jupyter_output><empty_output><jupyter_text>Load- For the load part we need `Writer` instances and a `Sink`.- writers define where to load the data.- The `Sink` gets the transformed data (feature set) and trigger the load to all the defined writers.- `debug_mode` will create a temporary view instead of trying to write in a real data store.<jupyter_code>from butterfree.load.writers import (
    HistoricalFeatureStoreWriter,
    OnlineFeatureStoreWriter,
)
from butterfree.load import Sink

writers = [HistoricalFeatureStoreWriter(debug_mode=True), OnlineFeatureStoreWriter(debug_mode=True)]
sink = Sink(writers=writers)<jupyter_output><empty_output><jupyter_text>Pipeline- The `Pipeline` entity wraps all the other defined elements.- `run` command will trigger the execution of the pipeline, end-to-end.<jupyter_code>from butterfree.pipelines import FeatureSetPipeline

pipeline = FeatureSetPipeline(source=source, feature_set=feature_set, sink=sink)
result_df = pipeline.run()<jupyter_output><empty_output><jupyter_text>Showing the results<jupyter_code>print(">>> Historical Feature house_listings feature set table:")
spark.table("historical_feature_store__house_listings").orderBy(
    "id", "timestamp"
).toPandas()
print(">>> Online Feature house_listings feature set table:")
spark.table("online_feature_store__house_listings").orderBy("id", "timestamp").toPandas()<jupyter_output>>>> Online Feature house_listings feature set table:
{'fraction_non_alphanumeric': 0.09165085388994307, 'fraction_numerical': 0.0018026565464895636, 'mean_word_length': 4.473001038421599, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '15170', 'license': 'Apache-2.0', 'n_tokens_mistral': 2807, 'n_tokens_neox': 2754, 'n_words': 1030, 'path': 'examples/simple_feature_set/simple_feature_set.ipynb', 'repo_name': 'jdvala/butterfree', 'size': '10540'}