title structured data classification with featurespace fchollethttps twitter comfchollet date created 20221109 last modified 20221109 description classify tabular data in a few lines of code accelerator gpu introduction this example demonstrates how to do structured data classification also known as tabular data classification starting from a raw csv file our data includes numerical features and integer categorical features and string categorical features we will use the utility keras utils featurespace to index preprocess and encode our features the code is adapted from the example structured data classification from scratchhttps keras ioexamplesstructureddatastructureddataclassificationfromscratch while the previous example managed its own lowlevel feature preprocessing and encoding with keras preprocessing layers in this example we delegate everything to featurespace making the workflow extremely quick and easy note that this example should be run with tensorflow 2 12 or higher before the release of tensorflow 2 12 you can use tfnightly the dataset our datasethttps archive ics uci edumldatasetsheartdisease is provided by the cleveland clinic foundation for heart disease it s a csv file with 303 rows each row contains information about a patient a sample and each column describes an attribute of the patient a feature we use the features to predict whether a patient has a heart disease binary classification here s the description of each feature column description feature type age age in years numerical sex 1 male 0 female categorical cp chest pain type 0 1 2 3 4 categorical trestbpd resting blood pressure in mm hg on admission numerical chol serum cholesterol in mgdl numerical fbs fasting blood sugar in 120 mgdl 1 true 0 false categorical restecg resting electrocardiogram results 0 1 2 categorical thalach maximum heart rate achieved numerical exang exercise induced angina 1 yes 0 no categorical oldpeak st depression induced by exercise relative to rest numerical slope slope of the peak exercise st segment numerical ca number of major vessels 03 colored by fluoroscopy both numerical categorical thal 3 normal 6 fixed defect 7 reversible defect categorical target diagnosis of heart disease 1 true 0 false target setup preparing the data let s download the data and load it into a pandas dataframe the dataset includes 303 samples with 14 columns per sample 13 features plus the target label here s a preview of a few samples the last column target indicates whether the patient has a heart disease 1 or not 0 let s split the data into a training and validation set let s generate tf data dataset objects for each dataframe each dataset yields a tuple input target where input is a dictionary of features and target is the value 0 or 1 let s batch the datasets configuring a featurespace to configure how each feature should be preprocessed we instantiate a keras utils featurespace and we pass to it a dictionary that maps the name of our features to a string that describes the feature type we have a few integer categorical features such as fbs one string categorical feature thal and a few numerical features which we d like to normalize except age which we d like to discretize into a number of bins we also use the crosses argument to capture feature interactions for some categorical features that is to say create additional features that represent value cooccurrences for these categorical features you can compute feature crosses like this for arbitrary sets of categorical features not just tuples of two features because the resulting cooccurences are hashed into a fixedsized vector you don t need to worry about whether the cooccurence space is too large categorical features encoded as integers categorical feature encoded as string numerical features to discretize numerical features to normalize we create additional features by hashing value cooccurrences for the following groups of categorical features the hashing space for these cooccurrences wil be 32dimensional our utility will onehot encode all categorical features and concat all features into a single vector one vector per sample further customizing a featurespace specifying the feature type via a string name is quick and easy but sometimes you may want to further configure the preprocessing of each feature for instance in our case our categorical features don t have a large set of possible values it s only a handful of values per feature e g 1 and 0 for the feature fbs and all possible values are represented in the training set as a result we don t need to reserve an index to represent out of vocabulary values for these features which would have been the default behavior below we just specify numoovindices0 in each of these features to tell the feature preprocessor to skip out of vocabulary indexing other customizations you have access to include specifying the number of bins for discretizing features of type floatdiscretized or the dimensionality of the hashing space for feature crossing categorical features encoded as integers categorical feature encoded as string numerical features to discretize numerical features to normalize specify feature cross with a custom crossing dim adapt the featurespace to the training data before we start using the featurespace to build a model we have to adapt it to the training data during adapt the featurespace will index the set of possible values for categorical features compute the mean and variance for numerical features to normalize compute the value boundaries for the different bins for numerical features to discretize note that adapt should be called on a tf data dataset which yields dicts of feature values no labels at this point the featurespace can be called on a dict of raw feature values and will return a single concatenate vector for each sample combining encoded features and feature crosses two ways to manage preprocessing as part of the tf data pipeline or in the model itself there are two ways in which you can leverage your featurespace asynchronous preprocessing in tf data you can make it part of your data pipeline before the model this enables asynchronous parallel preprocessing of the data on cpu before it hits the model do this if you re training on gpu or tpu or if you want to speed up preprocessing usually this is always the right thing to do during training synchronous preprocessing in the model you can make it part of your model this means that the model will expect dicts of raw feature values and the preprocessing batch will be done synchronously in a blocking manner before the rest of the forward pass do this if you want to have an endtoend model that can process raw feature values but keep in mind that your model will only be able to run on cpu since most types of feature preprocessing e g string preprocessing are not gpu or tpu compatible do not do this on gpu tpu or in performancesensitive settings in general you want to do inmodel preprocessing when you do inference on cpu in our case we will apply the featurespace in the tf data pipeline during training but we will do inference with an endtoend model that includes the featurespace let s create a training and validation dataset of preprocessed batches build a model time to build a model or rather two models a training model that expects preprocessed features one sample one vector an inference model that expects raw features one sample dict of raw feature values train the model let s train our model for 50 epochs note that feature preprocessing is happening as part of the tf data pipeline not as part of the model we quickly get to 80 validation accuracy inference on new data with the endtoend model now we can use our inference model which includes the featurespace to make predictions based on dicts of raw features values as follows introduction this example demonstrates how to do structured data classification also known as tabular data classification starting from a raw csv file our data includes numerical features and integer categorical features and string categorical features we will use the utility keras utils featurespace to index preprocess and encode our features the code is adapted from the example structured data classification from scratch https keras io examples structured_data structured_data_classification_from_scratch while the previous example managed its own low level feature preprocessing and encoding with keras preprocessing layers in this example we delegate everything to featurespace making the workflow extremely quick and easy note that this example should be run with tensorflow 2 12 or higher before the release of tensorflow 2 12 you can use tf nightly the dataset our dataset https archive ics uci edu ml datasets heart disease is provided by the cleveland clinic foundation for heart disease it s a csv file with 303 rows each row contains information about a patient a sample and each column describes an attribute of the patient a feature we use the features to predict whether a patient has a heart disease binary classification here s the description of each feature column description feature type age age in years numerical sex 1 male 0 female categorical cp chest pain type 0 1 2 3 4 categorical trestbpd resting blood pressure in mm hg on admission numerical chol serum cholesterol in mg dl numerical fbs fasting blood sugar in 120 mg dl 1 true 0 false categorical restecg resting electrocardiogram results 0 1 2 categorical thalach maximum heart rate achieved numerical exang exercise induced angina 1 yes 0 no categorical oldpeak st depression induced by exercise relative to rest numerical slope slope of the peak exercise st segment numerical ca number of major vessels 0 3 colored by fluoroscopy both numerical categorical thal 3 normal 6 fixed defect 7 reversible defect categorical target diagnosis of heart disease 1 true 0 false target setup preparing the data let s download the data and load it into a pandas dataframe the dataset includes 303 samples with 14 columns per sample 13 features plus the target label here s a preview of a few samples the last column target indicates whether the patient has a heart disease 1 or not 0 let s split the data into a training and validation set let s generate tf data dataset objects for each dataframe each dataset yields a tuple input target where input is a dictionary of features and target is the value 0 or 1 let s batch the datasets configuring a featurespace to configure how each feature should be preprocessed we instantiate a keras utils featurespace and we pass to it a dictionary that maps the name of our features to a string that describes the feature type we have a few integer categorical features such as fbs one string categorical feature thal and a few numerical features which we d like to normalize except age which we d like to discretize into a number of bins we also use the crosses argument to capture feature interactions for some categorical features that is to say create additional features that represent value co occurrences for these categorical features you can compute feature crosses like this for arbitrary sets of categorical features not just tuples of two features because the resulting co occurences are hashed into a fixed sized vector you don t need to worry about whether the co occurence space is too large categorical features encoded as integers categorical feature encoded as string numerical features to discretize numerical features to normalize we create additional features by hashing value co occurrences for the following groups of categorical features the hashing space for these co occurrences wil be 32 dimensional our utility will one hot encode all categorical features and concat all features into a single vector one vector per sample further customizing a featurespace specifying the feature type via a string name is quick and easy but sometimes you may want to further configure the preprocessing of each feature for instance in our case our categorical features don t have a large set of possible values it s only a handful of values per feature e g 1 and 0 for the feature fbs and all possible values are represented in the training set as a result we don t need to reserve an index to represent out of vocabulary values for these features which would have been the default behavior below we just specify num_oov_indices 0 in each of these features to tell the feature preprocessor to skip out of vocabulary indexing other customizations you have access to include specifying the number of bins for discretizing features of type float_discretized or the dimensionality of the hashing space for feature crossing categorical features encoded as integers categorical feature encoded as string numerical features to discretize numerical features to normalize specify feature cross with a custom crossing dim adapt the featurespace to the training data before we start using the featurespace to build a model we have to adapt it to the training data during adapt the featurespace will index the set of possible values for categorical features compute the mean and variance for numerical features to normalize compute the value boundaries for the different bins for numerical features to discretize note that adapt should be called on a tf data dataset which yields dicts of feature values no labels at this point the featurespace can be called on a dict of raw feature values and will return a single concatenate vector for each sample combining encoded features and feature crosses two ways to manage preprocessing as part of the tf data pipeline or in the model itself there are two ways in which you can leverage your featurespace asynchronous preprocessing in tf data you can make it part of your data pipeline before the model this enables asynchronous parallel preprocessing of the data on cpu before it hits the model do this if you re training on gpu or tpu or if you want to speed up preprocessing usually this is always the right thing to do during training synchronous preprocessing in the model you can make it part of your model this means that the model will expect dicts of raw feature values and the preprocessing batch will be done synchronously in a blocking manner before the rest of the forward pass do this if you want to have an end to end model that can process raw feature values but keep in mind that your model will only be able to run on cpu since most types of feature preprocessing e g string preprocessing are not gpu or tpu compatible do not do this on gpu tpu or in performance sensitive settings in general you want to do in model preprocessing when you do inference on cpu in our case we will apply the featurespace in the tf data pipeline during training but we will do inference with an end to end model that includes the featurespace let s create a training and validation dataset of preprocessed batches build a model time to build a model or rather two models a training model that expects preprocessed features one sample one vector an inference model that expects raw features one sample dict of raw feature values train the model let s train our model for 50 epochs note that feature preprocessing is happening as part of the tf data pipeline not as part of the model we quickly get to 80 validation accuracy inference on new data with the end to end model now we can use our inference model which includes the featurespace to make predictions based on dicts of raw features values as follows
import os
os.environ["KERAS_BACKEND"] = "tensorflow"

import tensorflow as tf
import pandas as pd
import keras
from keras.utils import FeatureSpace



file_url = (
    "http://storage.googleapis.com/download.tensorflow.org/data/heart.csv"
)
dataframe = pd.read_csv(file_url)



print(dataframe.shape)



dataframe.head()



val_dataframe = dataframe.sample(frac=0.2, random_state=1337)
train_dataframe = dataframe.drop(val_dataframe.index)

print(
    "Using %d samples for training and %d for validation"
    % (len(train_dataframe), len(val_dataframe))
)




def dataframe_to_dataset(dataframe):
    dataframe = dataframe.copy()
    labels = dataframe.pop("target")
    ds = tf.data.Dataset.from_tensor_slices((dict(dataframe), labels))
    ds = ds.shuffle(buffer_size=len(dataframe))
    return ds


train_ds = dataframe_to_dataset(train_dataframe)
val_ds = dataframe_to_dataset(val_dataframe)



for x, y in train_ds.take(1):
    print("Input:", x)
    print("Target:", y)



train_ds = train_ds.batch(32)
val_ds = val_ds.batch(32)



feature_space = FeatureSpace(
    features={
        
        "sex": "integer_categorical",
        "cp": "integer_categorical",
        "fbs": "integer_categorical",
        "restecg": "integer_categorical",
        "exang": "integer_categorical",
        "ca": "integer_categorical",
        
        "thal": "string_categorical",
        
        "age": "float_discretized",
        
        "trestbps": "float_normalized",
        "chol": "float_normalized",
        "thalach": "float_normalized",
        "oldpeak": "float_normalized",
        "slope": "float_normalized",
    },
    
    
    
    crosses=[("sex", "age"), ("thal", "ca")],
    
    
    crossing_dim=32,
    
    
    
    output_mode="concat",
)



feature_space = FeatureSpace(
    features={
        
        "sex": FeatureSpace.integer_categorical(num_oov_indices=0),
        "cp": FeatureSpace.integer_categorical(num_oov_indices=0),
        "fbs": FeatureSpace.integer_categorical(num_oov_indices=0),
        "restecg": FeatureSpace.integer_categorical(num_oov_indices=0),
        "exang": FeatureSpace.integer_categorical(num_oov_indices=0),
        "ca": FeatureSpace.integer_categorical(num_oov_indices=0),
        
        "thal": FeatureSpace.string_categorical(num_oov_indices=0),
        
        "age": FeatureSpace.float_discretized(num_bins=30),
        
        "trestbps": FeatureSpace.float_normalized(),
        "chol": FeatureSpace.float_normalized(),
        "thalach": FeatureSpace.float_normalized(),
        "oldpeak": FeatureSpace.float_normalized(),
        "slope": FeatureSpace.float_normalized(),
    },
    
    crosses=[
        FeatureSpace.cross(feature_names=("sex", "age"), crossing_dim=64),
        FeatureSpace.cross(
            feature_names=("thal", "ca"),
            crossing_dim=16,
        ),
    ],
    output_mode="concat",
)



train_ds_with_no_labels = train_ds.map(lambda x, _: x)
feature_space.adapt(train_ds_with_no_labels)



for x, _ in train_ds.take(1):
    preprocessed_x = feature_space(x)
    print("preprocessed_x.shape:", preprocessed_x.shape)
    print("preprocessed_x.dtype:", preprocessed_x.dtype)





preprocessed_train_ds = train_ds.map(
    lambda x, y: (feature_space(x), y), num_parallel_calls=tf.data.AUTOTUNE
)
preprocessed_train_ds = preprocessed_train_ds.prefetch(tf.data.AUTOTUNE)

preprocessed_val_ds = val_ds.map(
    lambda x, y: (feature_space(x), y), num_parallel_calls=tf.data.AUTOTUNE
)
preprocessed_val_ds = preprocessed_val_ds.prefetch(tf.data.AUTOTUNE)



dict_inputs = feature_space.get_inputs()
encoded_features = feature_space.get_encoded_features()

x = keras.layers.Dense(32, activation="relu")(encoded_features)
x = keras.layers.Dropout(0.5)(x)
predictions = keras.layers.Dense(1, activation="sigmoid")(x)

training_model = keras.Model(inputs=encoded_features, outputs=predictions)
training_model.compile(
    optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"]
)

inference_model = keras.Model(inputs=dict_inputs, outputs=predictions)



training_model.fit(
    preprocessed_train_ds,
    epochs=20,
    validation_data=preprocessed_val_ds,
    verbose=2,
)





sample = {
    "age": 60,
    "sex": 1,
    "cp": 1,
    "trestbps": 145,
    "chol": 233,
    "fbs": 1,
    "restecg": 2,
    "thalach": 150,
    "exang": 0,
    "oldpeak": 2.3,
    "slope": 3,
    "ca": 0,
    "thal": "fixed",
}

input_dict = {
    name: tf.convert_to_tensor([value]) for name, value in sample.items()
}
predictions = inference_model.predict(input_dict)

print(
    f"This particular patient had a {100 * predictions[0][0]:.2f}% probability "
    "of having a heart disease, as evaluated by our model."
)