title the functional api fchollethttps twitter comfchollet date created 20190301 last modified 20200412 description complete guide to the functional api accelerator gpu setup introduction the keras functional api is a way to create models that are more flexible than the keras sequential api the functional api can handle models with nonlinear topology shared layers and even multiple inputs or outputs the main idea is that a deep learning model is usually a directed acyclic graph dag of layers so the functional api is a way to build graphs of layers consider the following model div classkdefaultcodeblock input 784dimensional vectors dense 64 units relu activation dense 64 units relu activation dense 10 units softmax activation output logits of a probability distribution over 10 classes div this is a basic graph with three layers to build this model using the functional api start by creating an input node the shape of the data is set as a 784dimensional vector the batch size is always omitted since only the shape of each sample is specified if for example you have an image input with a shape of 32 32 3 you would use just for demonstration purposes the inputs that is returned contains information about the shape and dtype of the input data that you feed to your model here s the shape here s the dtype you create a new node in the graph of layers by calling a layer on this inputs object the layer call action is like drawing an arrow from inputs to this layer you created you re passing the inputs to the dense layer and you get x as the output let s add a few more layers to the graph of layers at this point you can create a model by specifying its inputs and outputs in the graph of layers let s check out what the model summary looks like you can also plot the model as a graph and optionally display the input and output shapes of each layer in the plotted graph this figure and the code are almost identical in the code version the connection arrows are replaced by the call operation a graph of layers is an intuitive mental image for a deep learning model and the functional api is a way to create models that closely mirrors this training evaluation and inference training evaluation and inference work exactly in the same way for models built using the functional api as for sequential models the model class offers a builtin training loop the fit method and a builtin evaluation loop the evaluate method note that you can easily customize these loopsguidescustomizingwhathappensinfit to implement training routines beyond supervised learning e g ganshttps keras ioexamplesgenerativedcganoverridingtrainstep here load the mnist image data reshape it into vectors fit the model on the data while monitoring performance on a validation split then evaluate the model on the test data for further reading see the training and evaluationguidestrainingwithbuiltinmethods guide save and serialize saving the model and serialization work the same way for models built using the functional api as they do for sequential models the standard way to save a functional model is to call model save to save the entire model as a single file you can later recreate the same model from this file even if the code that built the model is no longer available this saved file includes the model architecture model weight values that were learned during training model training config if any as passed to compile optimizer and its state if any to restart training where you left off recreate the exact same model purely from the file for details read the model serialization saving guidesserializationandsaving guide use the same graph of layers to define multiple models in the functional api models are specifying their inputs and outputs in a graph of layers that means that a single graph of layers can be used to generate multiple models in the example below you use the same stack of layers to instantiate two models an encoder model that turns image inputs into 16dimensional vectors and an endtoend autoencoder model for training here the decoding architecture is strictly symmetrical to the encoding architecture so the output shape is the same as the input shape 28 28 1 the reverse of a conv2d layer is a conv2dtranspose layer and the reverse of a maxpooling2d layer is an upsampling2d layer all models are callable just like layers you can treat any model as if it were a layer by invoking it on an input or on the output of another layer by calling a model you aren t just reusing the architecture of the model you re also reusing its weights to see this in action here s a different take on the autoencoder example that creates an encoder model a decoder model and chains them in two calls to obtain the autoencoder model as you can see the model can be nested a model can contain submodels since a model is just like a layer a common use case for model nesting is ensembling for example here s how to ensemble a set of models into a single model that averages their predictions manipulate complex graph topologies models with multiple inputs and outputs the functional api makes it easy to manipulate multiple inputs and outputs this cannot be handled with the sequential api for example if you re building a system for ranking customer issue tickets by priority and routing them to the correct department then the model will have three inputs the title of the ticket text input the text body of the ticket text input and any tags added by the user categorical input this model will have two outputs the priority score between 0 and 1 scalar sigmoid output and the department that should handle the ticket softmax output over the set of departments you can build this model in a few lines with the functional api embed each word in the title into a 64dimensional vector embed each word in the text into a 64dimensional vector reduce sequence of embedded words in the title into a single 128dimensional vector reduce sequence of embedded words in the body into a single 32dimensional vector merge all available features into a single large vector via concatenation stick a logistic regression for priority prediction on top of the features stick a department classifier on top of the features instantiate an endtoend model predicting both priority and department now plot the model when compiling this model you can assign different losses to each output you can even assign different weights to each loss to modulate their contribution to the total training loss since the output layers have different names you could also specify the losses and loss weights with the corresponding layer names train the model by passing lists of numpy arrays of inputs and targets dummy input data dummy target data when calling fit with a dataset object it should yield either a tuple of lists like titledata bodydata tagsdata prioritytargets depttargets or a tuple of dictionaries like title titledata body bodydata tags tagsdata priority prioritytargets department depttargets for more detailed explanation refer to the training and evaluationguidestrainingwithbuiltinmethods guide a toy resnet model in addition to models with multiple inputs and outputs the functional api makes it easy to manipulate nonlinear connectivity topologies these are models with layers that are not connected sequentially which the sequential api cannot handle a common use case for this is residual connections let s build a toy resnet model for cifar10 to demonstrate this plot the model now train the model we restrict the data to the first 1000 samples so as to limit execution time on colab try to train on the entire dataset until convergence shared layers another good use for the functional api are models that use shared layers shared layers are layer instances that are reused multiple times in the same model they learn features that correspond to multiple paths in the graphoflayers shared layers are often used to encode inputs from similar spaces say two different pieces of text that feature similar vocabulary they enable sharing of information across these different inputs and they make it possible to train such a model on less data if a given word is seen in one of the inputs that will benefit the processing of all inputs that pass through the shared layer to share a layer in the functional api call the same layer instance multiple times for instance here s an embedding layer shared across two different text inputs embedding for 1000 unique words mapped to 128dimensional vectors variablelength sequence of integers variablelength sequence of integers reuse the same layer to encode both inputs extract and reuse nodes in the graph of layers because the graph of layers you are manipulating is a static data structure it can be accessed and inspected and this is how you are able to plot functional models as images this also means that you can access the activations of intermediate layers nodes in the graph and reuse them elsewhere which is very useful for something like feature extraction let s look at an example this is a vgg19 model with weights pretrained on imagenet and these are the intermediate activations of the model obtained by querying the graph data structure use these features to create a new featureextraction model that returns the values of the intermediate layer activations this comes in handy for tasks like neural style transferhttps keras ioexamplesgenerativeneuralstyletransfer among other things extend the api using custom layers keras includes a wide range of builtin layers for example convolutional layers conv1d conv2d conv3d conv2dtranspose pooling layers maxpooling1d maxpooling2d maxpooling3d averagepooling1d rnn layers gru lstm convlstm2d batchnormalization dropout embedding etc but if you don t find what you need it s easy to extend the api by creating your own layers all layers subclass the layer class and implement call method that specifies the computation done by the layer build method that creates the weights of the layer this is just a style convention since you can create weights in init as well to learn more about creating layers from scratch read custom layers and modelsguidesmakingnewlayersandmodelsviasubclassing guide the following is a basic implementation of keras layers dense for serialization support in your custom layer define a getconfig method that returns the constructor arguments of the layer instance optionally implement the class method fromconfigcls config which is used when recreating a layer instance given its config dictionary the default implementation of fromconfig is python def fromconfigcls config return clsconfig when to use the functional api should you use the keras functional api to create a new model or just subclass the model class directly in general the functional api is higherlevel easier and safer and has a number of features that subclassed models do not support however model subclassing provides greater flexibility when building models that are not easily expressible as directed acyclic graphs of layers for example you could not implement a treernn with the functional api and would have to subclass model directly for an indepth look at the differences between the functional api and model subclassing read what are symbolic and imperative apis in tensorflow 2 0 https blog tensorflow org201901whataresymbolicandimperativeapis html functional api strengths the following properties are also true for sequential models which are also data structures but are not true for subclassed models which are python bytecode not data structures less verbose there is no super init no def callself etc compare python inputs keras inputshape32 x layers dense64 activation relu inputs outputs layers dense10x mlp keras modelinputs outputs with the subclassed version python class mlpkeras model def initself kwargs super initkwargs self dense1 layers dense64 activation relu self dense2 layers dense10 def callself inputs x self dense1inputs return self dense2x instantiate the model mlp mlp necessary to create the model s state the model doesn t have a state until it s called at least once mlpops zeros1 32 model validation while defining its connectivity graph in the functional api the input specification shape and dtype is created in advance using input every time you call a layer the layer checks that the specification passed to it matches its assumptions and it will raise a helpful error message if not this guarantees that any model you can build with the functional api will run all debugging other than convergencerelated debugging happens statically during the model construction and not at execution time this is similar to type checking in a compiler a functional model is plottable and inspectable you can plot the model as a graph and you can easily access intermediate nodes in this graph for example to extract and reuse the activations of intermediate layers as seen in a previous example python featureslist layer output for layer in vgg19 layers featextractionmodel keras modelinputsvgg19 input outputsfeatureslist a functional model can be serialized or cloned because a functional model is a data structure rather than a piece of code it is safely serializable and can be saved as a single file that allows you to recreate the exact same model without having access to any of the original code see the serialization saving guideguidesserializationandsaving to serialize a subclassed model it is necessary for the implementer to specify a getconfig and fromconfig method at the model level functional api weakness it does not support dynamic architectures the functional api treats models as dags of layers this is true for most deep learning architectures but not all for example recursive networks or tree rnns do not follow this assumption and cannot be implemented in the functional api mixandmatch api styles choosing between the functional api or model subclassing isn t a binary decision that restricts you into one category of models all models in the keras api can interact with each other whether they re sequential models functional models or subclassed models that are written from scratch you can always use a functional model or sequential model as part of a subclassed model or layer define a functional model our previouslydefined functional model you can use any subclassed layer or model in the functional api as long as it implements a call method that follows one of the following patterns callself inputs kwargs where inputs is a tensor or a nested structure of tensors e g a list of tensors and where kwargs are nontensor arguments noninputs callself inputs trainingnone kwargs where training is a boolean indicating whether the layer should behave in training mode and inference mode callself inputs masknone kwargs where mask is a boolean mask tensor useful for rnns for instance callself inputs trainingnone masknone kwargs of course you can have both masking and trainingspecific behavior at the same time additionally if you implement the getconfig method on your custom layer or model the functional models you create will still be serializable and cloneable here s a quick example of a custom rnn written from scratch being used in a functional model note that you specify a static batch size for the inputs with the batchshape arg because the inner computation of customrnn requires a static batch size when you create the state zeros tensor setup introduction the keras functional api is a way to create models that are more flexible than the keras sequential api the functional api can handle models with non linear topology shared layers and even multiple inputs or outputs the main idea is that a deep learning model is usually a directed acyclic graph dag of layers so the functional api is a way to build graphs of layers consider the following model div class k default codeblock input 784 dimensional vectors dense 64 units relu activation dense 64 units relu activation dense 10 units softmax activation output logits of a probability distribution over 10 classes div this is a basic graph with three layers to build this model using the functional api start by creating an input node the shape of the data is set as a 784 dimensional vector the batch size is always omitted since only the shape of each sample is specified if for example you have an image input with a shape of 32 32 3 you would use just for demonstration purposes the inputs that is returned contains information about the shape and dtype of the input data that you feed to your model here s the shape here s the dtype you create a new node in the graph of layers by calling a layer on this inputs object the layer call action is like drawing an arrow from inputs to this layer you created you re passing the inputs to the dense layer and you get x as the output let s add a few more layers to the graph of layers at this point you can create a model by specifying its inputs and outputs in the graph of layers let s check out what the model summary looks like you can also plot the model as a graph and optionally display the input and output shapes of each layer in the plotted graph this figure and the code are almost identical in the code version the connection arrows are replaced by the call operation a graph of layers is an intuitive mental image for a deep learning model and the functional api is a way to create models that closely mirrors this training evaluation and inference training evaluation and inference work exactly in the same way for models built using the functional api as for sequential models the model class offers a built in training loop the fit method and a built in evaluation loop the evaluate method note that you can easily customize these loops guides customizing_what_happens_in_fit to implement training routines beyond supervised learning e g gans https keras io examples generative dcgan_overriding_train_step here load the mnist image data reshape it into vectors fit the model on the data while monitoring performance on a validation split then evaluate the model on the test data for further reading see the training and evaluation guides training_with_built_in_methods guide save and serialize saving the model and serialization work the same way for models built using the functional api as they do for sequential models the standard way to save a functional model is to call model save to save the entire model as a single file you can later recreate the same model from this file even if the code that built the model is no longer available this saved file includes the model architecture model weight values that were learned during training model training config if any as passed to compile optimizer and its state if any to restart training where you left off recreate the exact same model purely from the file for details read the model serialization saving guides serialization_and_saving guide use the same graph of layers to define multiple models in the functional api models are specifying their inputs and outputs in a graph of layers that means that a single graph of layers can be used to generate multiple models in the example below you use the same stack of layers to instantiate two models an encoder model that turns image inputs into 16 dimensional vectors and an end to end autoencoder model for training here the decoding architecture is strictly symmetrical to the encoding architecture so the output shape is the same as the input shape 28 28 1 the reverse of a conv2d layer is a conv2dtranspose layer and the reverse of a maxpooling2d layer is an upsampling2d layer all models are callable just like layers you can treat any model as if it were a layer by invoking it on an input or on the output of another layer by calling a model you aren t just reusing the architecture of the model you re also reusing its weights to see this in action here s a different take on the autoencoder example that creates an encoder model a decoder model and chains them in two calls to obtain the autoencoder model as you can see the model can be nested a model can contain sub models since a model is just like a layer a common use case for model nesting is ensembling for example here s how to ensemble a set of models into a single model that averages their predictions manipulate complex graph topologies models with multiple inputs and outputs the functional api makes it easy to manipulate multiple inputs and outputs this cannot be handled with the sequential api for example if you re building a system for ranking customer issue tickets by priority and routing them to the correct department then the model will have three inputs the title of the ticket text input the text body of the ticket text input and any tags added by the user categorical input this model will have two outputs the priority score between 0 and 1 scalar sigmoid output and the department that should handle the ticket softmax output over the set of departments you can build this model in a few lines with the functional api number of unique issue tags size of vocabulary obtained when preprocessing text data number of departments for predictions variable length sequence of ints variable length sequence of ints binary vectors of size num_tags embed each word in the title into a 64 dimensional vector embed each word in the text into a 64 dimensional vector reduce sequence of embedded words in the title into a single 128 dimensional vector reduce sequence of embedded words in the body into a single 32 dimensional vector merge all available features into a single large vector via concatenation stick a logistic regression for priority prediction on top of the features stick a department classifier on top of the features instantiate an end to end model predicting both priority and department now plot the model when compiling this model you can assign different losses to each output you can even assign different weights to each loss to modulate their contribution to the total training loss since the output layers have different names you could also specify the losses and loss weights with the corresponding layer names train the model by passing lists of numpy arrays of inputs and targets dummy input data dummy target data when calling fit with a dataset object it should yield either a tuple of lists like title_data body_data tags_data priority_targets dept_targets or a tuple of dictionaries like title title_data body body_data tags tags_data priority priority_targets department dept_targets for more detailed explanation refer to the training and evaluation guides training_with_built_in_methods guide a toy resnet model in addition to models with multiple inputs and outputs the functional api makes it easy to manipulate non linear connectivity topologies these are models with layers that are not connected sequentially which the sequential api cannot handle a common use case for this is residual connections let s build a toy resnet model for cifar10 to demonstrate this plot the model now train the model we restrict the data to the first 1000 samples so as to limit execution time on colab try to train on the entire dataset until convergence shared layers another good use for the functional api are models that use shared layers shared layers are layer instances that are reused multiple times in the same model they learn features that correspond to multiple paths in the graph of layers shared layers are often used to encode inputs from similar spaces say two different pieces of text that feature similar vocabulary they enable sharing of information across these different inputs and they make it possible to train such a model on less data if a given word is seen in one of the inputs that will benefit the processing of all inputs that pass through the shared layer to share a layer in the functional api call the same layer instance multiple times for instance here s an embedding layer shared across two different text inputs embedding for 1000 unique words mapped to 128 dimensional vectors variable length sequence of integers variable length sequence of integers reuse the same layer to encode both inputs extract and reuse nodes in the graph of layers because the graph of layers you are manipulating is a static data structure it can be accessed and inspected and this is how you are able to plot functional models as images this also means that you can access the activations of intermediate layers nodes in the graph and reuse them elsewhere which is very useful for something like feature extraction let s look at an example this is a vgg19 model with weights pretrained on imagenet and these are the intermediate activations of the model obtained by querying the graph data structure use these features to create a new feature extraction model that returns the values of the intermediate layer activations this comes in handy for tasks like neural style transfer https keras io examples generative neural_style_transfer among other things extend the api using custom layers keras includes a wide range of built in layers for example convolutional layers conv1d conv2d conv3d conv2dtranspose pooling layers maxpooling1d maxpooling2d maxpooling3d averagepooling1d rnn layers gru lstm convlstm2d batchnormalization dropout embedding etc but if you don t find what you need it s easy to extend the api by creating your own layers all layers subclass the layer class and implement call method that specifies the computation done by the layer build method that creates the weights of the layer this is just a style convention since you can create weights in __init__ as well to learn more about creating layers from scratch read custom layers and models guides making_new_layers_and_models_via_subclassing guide the following is a basic implementation of keras layers dense for serialization support in your custom layer define a get_config method that returns the constructor arguments of the layer instance optionally implement the class method from_config cls config which is used when recreating a layer instance given its config dictionary the default implementation of from_config is python def from_config cls config return cls config when to use the functional api should you use the keras functional api to create a new model or just subclass the model class directly in general the functional api is higher level easier and safer and has a number of features that subclassed models do not support however model subclassing provides greater flexibility when building models that are not easily expressible as directed acyclic graphs of layers for example you could not implement a tree rnn with the functional api and would have to subclass model directly for an in depth look at the differences between the functional api and model subclassing read what are symbolic and imperative apis in tensorflow 2 0 https blog tensorflow org 2019 01 what are symbolic and imperative apis html functional api strengths the following properties are also true for sequential models which are also data structures but are not true for subclassed models which are python bytecode not data structures less verbose there is no super __init__ no def call self etc compare python inputs keras input shape 32 x layers dense 64 activation relu inputs outputs layers dense 10 x mlp keras model inputs outputs with the subclassed version python class mlp keras model def __init__ self kwargs super __init__ kwargs self dense_1 layers dense 64 activation relu self dense_2 layers dense 10 def call self inputs x self dense_1 inputs return self dense_2 x instantiate the model mlp mlp necessary to create the model s state the model doesn t have a state until it s called at least once _ mlp ops zeros 1 32 model validation while defining its connectivity graph in the functional api the input specification shape and dtype is created in advance using input every time you call a layer the layer checks that the specification passed to it matches its assumptions and it will raise a helpful error message if not this guarantees that any model you can build with the functional api will run all debugging other than convergence related debugging happens statically during the model construction and not at execution time this is similar to type checking in a compiler a functional model is plottable and inspectable you can plot the model as a graph and you can easily access intermediate nodes in this graph for example to extract and reuse the activations of intermediate layers as seen in a previous example python features_list layer output for layer in vgg19 layers feat_extraction_model keras model inputs vgg19 input outputs features_list a functional model can be serialized or cloned because a functional model is a data structure rather than a piece of code it is safely serializable and can be saved as a single file that allows you to recreate the exact same model without having access to any of the original code see the serialization saving guide guides serialization_and_saving to serialize a subclassed model it is necessary for the implementer to specify a get_config and from_config method at the model level functional api weakness it does not support dynamic architectures the functional api treats models as dags of layers this is true for most deep learning architectures but not all for example recursive networks or tree rnns do not follow this assumption and cannot be implemented in the functional api mix and match api styles choosing between the functional api or model subclassing isn t a binary decision that restricts you into one category of models all models in the keras api can interact with each other whether they re sequential models functional models or subclassed models that are written from scratch you can always use a functional model or sequential model as part of a subclassed model or layer define a functional model our previously defined functional model you can use any subclassed layer or model in the functional api as long as it implements a call method that follows one of the following patterns call self inputs kwargs where inputs is a tensor or a nested structure of tensors e g a list of tensors and where kwargs are non tensor arguments non inputs call self inputs training none kwargs where training is a boolean indicating whether the layer should behave in training mode and inference mode call self inputs mask none kwargs where mask is a boolean mask tensor useful for rnns for instance call self inputs training none mask none kwargs of course you can have both masking and training specific behavior at the same time additionally if you implement the get_config method on your custom layer or model the functional models you create will still be serializable and cloneable here s a quick example of a custom rnn written from scratch being used in a functional model note that you specify a static batch size for the inputs with the batch_shape arg because the inner computation of customrnn requires a static batch size when you create the state zeros tensor
import numpy as np
import keras
from keras import layers
from keras import ops



inputs = keras.Input(shape=(784,))




img_inputs = keras.Input(shape=(32, 32, 3))



inputs.shape



inputs.dtype



dense = layers.Dense(64, activation="relu")
x = dense(inputs)



x = layers.Dense(64, activation="relu")(x)
outputs = layers.Dense(10)(x)



model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")



model.summary()



keras.utils.plot_model(model, "my_first_model.png")



keras.utils.plot_model(
    model, "my_first_model_with_shape_info.png", show_shapes=True
)





(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

x_train = x_train.reshape(60000, 784).astype("float32") / 255
x_test = x_test.reshape(10000, 784).astype("float32") / 255

model.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.RMSprop(),
    metrics=["accuracy"],
)

history = model.fit(
    x_train, y_train, batch_size=64, epochs=2, validation_split=0.2
)

test_scores = model.evaluate(x_test, y_test, verbose=2)
print("Test loss:", test_scores[0])
print("Test accuracy:", test_scores[1])





model.save("my_model.keras")
del model

model = keras.models.load_model("my_model.keras")





encoder_input = keras.Input(shape=(28, 28, 1), name="img")
x = layers.Conv2D(16, 3, activation="relu")(encoder_input)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.Conv2D(16, 3, activation="relu")(x)
encoder_output = layers.GlobalMaxPooling2D()(x)

encoder = keras.Model(encoder_input, encoder_output, name="encoder")
encoder.summary()

x = layers.Reshape((4, 4, 1))(encoder_output)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
x = layers.Conv2DTranspose(32, 3, activation="relu")(x)
x = layers.UpSampling2D(3)(x)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
decoder_output = layers.Conv2DTranspose(1, 3, activation="relu")(x)

autoencoder = keras.Model(encoder_input, decoder_output, name="autoencoder")
autoencoder.summary()





encoder_input = keras.Input(shape=(28, 28, 1), name="original_img")
x = layers.Conv2D(16, 3, activation="relu")(encoder_input)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.Conv2D(16, 3, activation="relu")(x)
encoder_output = layers.GlobalMaxPooling2D()(x)

encoder = keras.Model(encoder_input, encoder_output, name="encoder")
encoder.summary()

decoder_input = keras.Input(shape=(16,), name="encoded_img")
x = layers.Reshape((4, 4, 1))(decoder_input)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
x = layers.Conv2DTranspose(32, 3, activation="relu")(x)
x = layers.UpSampling2D(3)(x)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
decoder_output = layers.Conv2DTranspose(1, 3, activation="relu")(x)

decoder = keras.Model(decoder_input, decoder_output, name="decoder")
decoder.summary()

autoencoder_input = keras.Input(shape=(28, 28, 1), name="img")
encoded_img = encoder(autoencoder_input)
decoded_img = decoder(encoded_img)
autoencoder = keras.Model(autoencoder_input, decoded_img, name="autoencoder")
autoencoder.summary()




def get_model():
    inputs = keras.Input(shape=(128,))
    outputs = layers.Dense(1)(inputs)
    return keras.Model(inputs, outputs)


model1 = get_model()
model2 = get_model()
model3 = get_model()

inputs = keras.Input(shape=(128,))
y1 = model1(inputs)
y2 = model2(inputs)
y3 = model3(inputs)
outputs = layers.average([y1, y2, y3])
ensemble_model = keras.Model(inputs=inputs, outputs=outputs)



num_tags = 12  
num_words = 10000  
num_departments = 4  

title_input = keras.Input(
    shape=(None,), name="title"
)  
body_input = keras.Input(
    shape=(None,), name="body"
)  
tags_input = keras.Input(
    shape=(num_tags,), name="tags"
)  


title_features = layers.Embedding(num_words, 64)(title_input)

body_features = layers.Embedding(num_words, 64)(body_input)


title_features = layers.LSTM(128)(title_features)

body_features = layers.LSTM(32)(body_features)


x = layers.concatenate([title_features, body_features, tags_input])


priority_pred = layers.Dense(1, name="priority")(x)

department_pred = layers.Dense(num_departments, name="department")(x)


model = keras.Model(
    inputs=[title_input, body_input, tags_input],
    outputs={"priority": priority_pred, "department": department_pred},
)



keras.utils.plot_model(
    model, "multi_input_and_output_model.png", show_shapes=True
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[
        keras.losses.BinaryCrossentropy(from_logits=True),
        keras.losses.CategoricalCrossentropy(from_logits=True),
    ],
    loss_weights=[1.0, 0.2],
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={
        "priority": keras.losses.BinaryCrossentropy(from_logits=True),
        "department": keras.losses.CategoricalCrossentropy(from_logits=True),
    },
    loss_weights={"priority": 1.0, "department": 0.2},
)




title_data = np.random.randint(num_words, size=(1280, 10))
body_data = np.random.randint(num_words, size=(1280, 100))
tags_data = np.random.randint(2, size=(1280, num_tags)).astype("float32")


priority_targets = np.random.random(size=(1280, 1))
dept_targets = np.random.randint(2, size=(1280, num_departments))

model.fit(
    {"title": title_data, "body": body_data, "tags": tags_data},
    {"priority": priority_targets, "department": dept_targets},
    epochs=2,
    batch_size=32,
)





inputs = keras.Input(shape=(32, 32, 3), name="img")
x = layers.Conv2D(32, 3, activation="relu")(inputs)
x = layers.Conv2D(64, 3, activation="relu")(x)
block_1_output = layers.MaxPooling2D(3)(x)

x = layers.Conv2D(64, 3, activation="relu", padding="same")(block_1_output)
x = layers.Conv2D(64, 3, activation="relu", padding="same")(x)
block_2_output = layers.add([x, block_1_output])

x = layers.Conv2D(64, 3, activation="relu", padding="same")(block_2_output)
x = layers.Conv2D(64, 3, activation="relu", padding="same")(x)
block_3_output = layers.add([x, block_2_output])

x = layers.Conv2D(64, 3, activation="relu")(block_3_output)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(256, activation="relu")(x)
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(10)(x)

model = keras.Model(inputs, outputs, name="toy_resnet")
model.summary()



keras.utils.plot_model(model, "mini_resnet.png", show_shapes=True)



(x_train, y_train), (x_test, y_test) = keras.datasets.cifar10.load_data()

x_train = x_train.astype("float32") / 255.0
x_test = x_test.astype("float32") / 255.0
y_train = keras.utils.to_categorical(y_train, 10)
y_test = keras.utils.to_categorical(y_test, 10)

model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["acc"],
)


model.fit(
    x_train[:1000],
    y_train[:1000],
    batch_size=64,
    epochs=1,
    validation_split=0.2,
)




shared_embedding = layers.Embedding(1000, 128)


text_input_a = keras.Input(shape=(None,), dtype="int32")


text_input_b = keras.Input(shape=(None,), dtype="int32")


encoded_input_a = shared_embedding(text_input_a)
encoded_input_b = shared_embedding(text_input_b)



vgg19 = keras.applications.VGG19()



features_list = [layer.output for layer in vgg19.layers]



feat_extraction_model = keras.Model(inputs=vgg19.input, outputs=features_list)

img = np.random.random((1, 224, 224, 3)).astype("float32")
extracted_features = feat_extraction_model(img)






class CustomDense(layers.Layer):
    def __init__(self, units=32):
        super().__init__()
        self.units = units

    def build(self, input_shape):
        self.w = self.add_weight(
            shape=(input_shape[-1], self.units),
            initializer="random_normal",
            trainable=True,
        )
        self.b = self.add_weight(
            shape=(self.units,), initializer="random_normal", trainable=True
        )

    def call(self, inputs):
        return ops.matmul(inputs, self.w) + self.b


inputs = keras.Input((4,))
outputs = CustomDense(10)(inputs)

model = keras.Model(inputs, outputs)




class CustomDense(layers.Layer):
    def __init__(self, units=32):
        super().__init__()
        self.units = units

    def build(self, input_shape):
        self.w = self.add_weight(
            shape=(input_shape[-1], self.units),
            initializer="random_normal",
            trainable=True,
        )
        self.b = self.add_weight(
            shape=(self.units,), initializer="random_normal", trainable=True
        )

    def call(self, inputs):
        return ops.matmul(inputs, self.w) + self.b

    def get_config(self):
        return {"units": self.units}


inputs = keras.Input((4,))
outputs = CustomDense(10)(inputs)

model = keras.Model(inputs, outputs)
config = model.get_config()

new_model = keras.Model.from_config(
    config, custom_objects={"CustomDense": CustomDense}
)







units = 32
timesteps = 10
input_dim = 5


inputs = keras.Input((None, units))
x = layers.GlobalAveragePooling1D()(inputs)
outputs = layers.Dense(1)(x)
model = keras.Model(inputs, outputs)


class CustomRNN(layers.Layer):
    def __init__(self):
        super().__init__()
        self.units = units
        self.projection_1 = layers.Dense(units=units, activation="tanh")
        self.projection_2 = layers.Dense(units=units, activation="tanh")
        
        self.classifier = model

    def call(self, inputs):
        outputs = []
        state = ops.zeros(shape=(inputs.shape[0], self.units))
        for t in range(inputs.shape[1]):
            x = inputs[:, t, :]
            h = self.projection_1(x)
            y = h + self.projection_2(state)
            state = y
            outputs.append(y)
        features = ops.stack(outputs, axis=1)
        print(features.shape)
        return self.classifier(features)


rnn_model = CustomRNN()
_ = rnn_model(ops.zeros((1, timesteps, input_dim)))



units = 32
timesteps = 10
input_dim = 5
batch_size = 16


class CustomRNN(layers.Layer):
    def __init__(self):
        super().__init__()
        self.units = units
        self.projection_1 = layers.Dense(units=units, activation="tanh")
        self.projection_2 = layers.Dense(units=units, activation="tanh")
        self.classifier = layers.Dense(1)

    def call(self, inputs):
        outputs = []
        state = ops.zeros(shape=(inputs.shape[0], self.units))
        for t in range(inputs.shape[1]):
            x = inputs[:, t, :]
            h = self.projection_1(x)
            y = h + self.projection_2(state)
            state = y
            outputs.append(y)
        features = ops.stack(outputs, axis=1)
        return self.classifier(features)





inputs = keras.Input(batch_shape=(batch_size, timesteps, input_dim))
x = layers.Conv1D(32, 3)(inputs)
outputs = CustomRNN()(x)

model = keras.Model(inputs, outputs)

rnn_model = CustomRNN()
_ = rnn_model(ops.zeros((1, 10, 5)))