title training evaluation with the builtin methods fchollethttps twitter comfchollet date created 20190301 last modified 20230325 description complete guide to training evaluation with fit and evaluate accelerator gpu setup we import torch tf so as to use torch dataloaders tf data datasets introduction this guide covers training evaluation and prediction inference models when using builtin apis for training validation such as model fit model evaluate and model predict if you are interested in leveraging fit while specifying your own training step function see the customizing what happens in fit guideguidescustomizingwhathappensinfit if you are interested in writing your own training evaluation loops from scratch see the guide writing a training loop from scratchguideswritingatrainingloopfromscratch in general whether you are using builtin loops or writing your own model training evaluation works strictly in the same way across every kind of keras model sequential models models built with the functional api and models written from scratch via model subclassing this guide doesn t cover distributed training which is covered in our guide to multigpu distributed traininghttps keras ioguidesdistributedtraining api overview a first endtoend example when passing data to the builtin training loops of a model you should either use numpy arrays if your data is small and fits in memory subclasses of keras utils pydataset tf data dataset objects pytorch dataloader instances in the next few paragraphs we ll use the mnist dataset as numpy arrays in order to demonstrate how to use optimizers losses and metrics afterwards we ll take a close look at each of the other options let s consider the following model here we build in with the functional api but it could be a sequential model or a subclassed model as well here s what the typical endtoend workflow looks like consisting of training validation on a holdout set generated from the original training data evaluation on the test data we ll use mnist data for this example preprocess the data these are numpy arrays reserve 10 000 samples for validation we specify the training configuration optimizer loss metrics loss function to minimize list of metrics to monitor we call fit which will train the model by slicing the data into batches of size batchsize and repeatedly iterating over the entire dataset for a given number of epochs we pass some validation for monitoring validation loss and metrics at the end of each epoch the returned history object holds a record of the loss values and metric values during training we evaluate the model on the test data via evaluate evaluate the model on the test data using evaluate generate predictions probabilities the output of the last layer on new data using predict now let s review each piece of this workflow in detail the compile method specifying a loss metrics and an optimizer to train a model with fit you need to specify a loss function an optimizer and optionally some metrics to monitor you pass these to the model as arguments to the compile method the metrics argument should be a list your model can have any number of metrics if your model has multiple outputs you can specify different losses and metrics for each output and you can modulate the contribution of each output to the total loss of the model you will find more details about this in the passing data to multiinput multioutput models section note that if you re satisfied with the default settings in many cases the optimizer loss and metrics can be specified via string identifiers as a shortcut for later reuse let s put our model definition and compile step in functions we will call them several times across different examples in this guide many builtin optimizers losses and metrics are available in general you won t have to create your own losses metrics or optimizers from scratch because what you need is likely to be already part of the keras api optimizers sgd with or without momentum rmsprop adam etc losses meansquarederror kldivergence cosinesimilarity etc metrics auc precision recall etc custom losses if you need to create a custom loss keras provides three ways to do so the first method involves creating a function that accepts inputs ytrue and ypred the following example shows a loss function that computes the mean squared error between the real data and the predictions we need to onehot encode the labels to use mse if you need a loss function that takes in parameters beside ytrue and ypred you can subclass the keras losses loss class and implement the following two methods initself accept parameters to pass during the call of your loss function callself ytrue ypred use the targets ytrue and the model predictions ypred to compute the model s loss let s say you want to use mean squared error but with an added term that will deincentivize prediction values far from 0 5 we assume that the categorical targets are onehot encoded and take values between 0 and 1 this creates an incentive for the model not to be too confident which may help reduce overfitting we won t know if it works until we try here s how you would do it custom metrics if you need a metric that isn t part of the api you can easily create custom metrics by subclassing the keras metrics metric class you will need to implement 4 methods initself in which you will create state variables for your metric updatestateself ytrue ypred sampleweightnone which uses the targets ytrue and the model predictions ypred to update the state variables resultself which uses the state variables to compute the final results resetstateself which reinitializes the state of the metric state update and results computation are kept separate in updatestate and result respectively because in some cases the results computation might be very expensive and would only be done periodically here s a simple example showing how to implement a categoricaltruepositives metric that counts how many samples were correctly classified as belonging to a given class the state of the metric will be reset at the start of each epoch handling losses and metrics that don t fit the standard signature the overwhelming majority of losses and metrics can be computed from ytrue and ypred where ypred is an output of your model but not all of them for instance a regularization loss may only require the activation of a layer there are no targets in this case and this activation may not be a model output in such cases you can call self addlosslossvalue from inside the call method of a custom layer losses added in this way get added to the main loss during training the one passed to compile here s a simple example that adds activity regularization note that activity regularization is builtin in all keras layers this layer is just for the sake of providing a concrete example insert activity regularization as a layer the displayed loss will be much higher than before due to the regularization component note that when you pass losses via addloss it becomes possible to call compile without a loss function since the model already has a loss to minimize consider the following logisticendpoint layer it takes as inputs targets logits and it tracks a crossentropy loss via addloss compute the trainingtime loss value and add it to the layer using self addloss return the inferencetime prediction tensor for predict you can use it in a model with two inputs input data targets compiled without a loss argument like this for more information about training multiinput models see the section passing data to multiinput multioutput models automatically setting apart a validation holdout set in the first endtoend example you saw we used the validationdata argument to pass a tuple of numpy arrays xval yval to the model for evaluating a validation loss and validation metrics at the end of each epoch here s another option the argument validationsplit allows you to automatically reserve part of your training data for validation the argument value represents the fraction of the data to be reserved for validation so it should be set to a number higher than 0 and lower than 1 for instance validationsplit0 2 means use 20 of the data for validation and validationsplit0 6 means use 60 of the data for validation the way the validation is computed is by taking the last x samples of the arrays received by the fit call before any shuffling note that you can only use validationsplit when training with numpy data training evaluation using tf data datasets in the past few paragraphs you ve seen how to handle losses metrics and optimizers and you ve seen how to use the validationdata and validationsplit arguments in fit when your data is passed as numpy arrays another option is to use an iteratorlike such as a tf data dataset a pytorch dataloader or a keras pydataset let s take look at the former the tf data api is a set of utilities in tensorflow 2 0 for loading and preprocessing data in a way that s fast and scalable for a complete guide about creating datasets see the tf data documentationhttps www tensorflow orgguidedata you can use tf data to train your keras models regardless of the backend you re using whether it s jax pytorch or tensorflow you can pass a dataset instance directly to the methods fit evaluate and predict first let s create a training dataset instance for the sake of our example we ll use the same mnist data as before shuffle and slice the dataset now we get a test dataset since the dataset already takes care of batching we don t pass a batchsize argument you can also evaluate or predict on a dataset note that the dataset is reset at the end of each epoch so it can be reused of the next epoch if you want to run training only on a specific number of batches from this dataset you can pass the stepsperepoch argument which specifies how many training steps the model should run using this dataset before moving on to the next epoch prepare the training dataset only use the 100 batches per epoch that s 64 100 samples you can also pass a dataset instance as the validationdata argument in fit prepare the training dataset prepare the validation dataset at the end of each epoch the model will iterate over the validation dataset and compute the validation loss and validation metrics if you want to run validation only on a specific number of batches from this dataset you can pass the validationsteps argument which specifies how many validation steps the model should run with the validation dataset before interrupting validation and moving on to the next epoch prepare the training dataset prepare the validation dataset only run validation using the first 10 batches of the dataset using the validationsteps argument note that the validation dataset will be reset after each use so that you will always be evaluating on the same samples from epoch to epoch the argument validationsplit generating a holdout set from the training data is not supported when training from dataset objects since this feature requires the ability to index the samples of the datasets which is not possible in general with the dataset api training evaluation using pydataset instances keras utils pydataset is a utility that you can subclass to obtain a python generator with two important properties it works well with multiprocessing it can be shuffled e g when passing shuffletrue in fit a pydataset must implement two methods getitem len the method getitem should return a complete batch if you want to modify your dataset between epochs you may implement onepochend here s a quick example to fit the model pass the dataset instead as the x argument no need for a y argument since the dataset includes the targets and pass the validation dataset as the validationdata argument and no need for the batchsize argument since the dataset is already batched evaluating the model is just as easy importantly pydataset objects support three common constructor arguments that handle the parallel processing configuration workers number of workers to use in multithreading or multiprocessing typically you d set it to the number of cores on your cpu usemultiprocessing whether to use python multiprocessing for parallelism setting this to true means that your dataset will be replicated in multiple forked processes this is necessary to gain computelevel rather than io level benefits from parallelism however it can only be set to true if your dataset can be safely pickled maxqueuesize maximum number of batches to keep in the queue when iterating over the dataset in a multithreaded or multipricessed setting you can reduce this value to reduce the cpu memory consumption of your dataset it defaults to 10 by default multiprocessing is disabled usemultiprocessingfalse and only one thread is used you should make sure to only turn on usemultiprocessing if your code is running inside a python if name main block in order to avoid issues here s a 4thread nonmultiprocessed example training evaluation using pytorch dataloader objects all builtin training and evaluation apis are also compatible with torch utils data dataset and torch utils data dataloader objects regardless of whether you re using the pytorch backend or the jax or tensorflow backends let s take a look at a simple example unlike pydataset which are batchcentric pytorch dataset objects are samplecentric the len method returns the number of samples and the getitem method returns a specific sample to use a pytorch dataset you need to wrap it into a dataloader which takes care of batching and shuffling now you can use them in the keras api just like any other iterator using sample weighting and class weighting with the default settings the weight of a sample is decided by its frequency in the dataset there are two methods to weight the data independent of sample frequency class weights sample weights class weights this is set by passing a dictionary to the classweight argument to model fit this dictionary maps class indices to the weight that should be used for samples belonging to this class this can be used to balance classes without resampling or to train a model that gives more importance to a particular class for instance if class 0 is half as represented as class 1 in your data you could use model fit classweight0 1 1 0 5 here s a numpy example where we use class weights or sample weights to give more importance to the correct classification of class 5 which is the digit 5 in the mnist dataset set weight 2 for class 5 making this class 2x more important sample weights for fine grained control or if you are not building a classifier you can use sample weights when training from numpy data pass the sampleweight argument to model fit when training from tf data or any other sort of iterator yield inputbatch labelbatch sampleweightbatch tuples a sample weights array is an array of numbers that specify how much weight each sample in a batch should have in computing the total loss it is commonly used in imbalanced classification problems the idea being to give more weight to rarelyseen classes when the weights used are ones and zeros the array can be used as a mask for the loss function entirely discarding the contribution of certain samples to the total loss here s a matching dataset example create a dataset that includes sample weights 3rd element in the return tuple shuffle and slice the dataset passing data to multiinput multioutput models in the previous examples we were considering a model with a single input a tensor of shape 764 and a single output a prediction tensor of shape 10 but what about models that have multiple inputs or outputs consider the following model which has an image input of shape 32 32 3 that s height width channels and a time series input of shape none 10 that s timesteps features our model will have two outputs computed from the combination of these inputs a score of shape 1 and a probability distribution over five classes of shape 5 let s plot this model so you can clearly see what we re doing here note that the shapes shown in the plot are batch shapes rather than persample shapes at compilation time we can specify different losses to different outputs by passing the loss functions as a list if we only passed a single loss function to the model the same loss function would be applied to every output which is not appropriate here likewise for metrics since we gave names to our output layers we could also specify peroutput losses and metrics via a dict we recommend the use of explicit names and dicts if you have more than 2 outputs it s possible to give different weights to different outputspecific losses for instance one might wish to privilege the score loss in our example by giving to 2x the importance of the class loss using the lossweights argument you could also choose not to compute a loss for certain outputs if these outputs are meant for prediction but not for training list loss version or dict loss version passing data to a multiinput or multioutput model in fit works in a similar way as specifying a loss function in compile you can pass lists of numpy arrays with 1 1 mapping to the outputs that received a loss function or dicts mapping output names to numpy arrays generate dummy numpy data fit on lists alternatively fit on dicts here s the dataset use case similarly as what we did for numpy arrays the dataset should return a tuple of dicts using callbacks callbacks in keras are objects that are called at different points during training at the start of an epoch at the end of a batch at the end of an epoch etc they can be used to implement certain behaviors such as doing validation at different points during training beyond the builtin perepoch validation checkpointing the model at regular intervals or when it exceeds a certain accuracy threshold changing the learning rate of the model when training seems to be plateauing doing finetuning of the top layers when training seems to be plateauing sending email or instant message notifications when training ends or where a certain performance threshold is exceeded etc callbacks can be passed as a list to your call to fit stop training when valloss is no longer improving no longer improving being defined as no better than 1e2 less no longer improving being further defined as for at least 2 epochs many builtin callbacks are available there are many builtin callbacks already available in keras such as modelcheckpoint periodically save the model earlystopping stop training when training is no longer improving the validation metrics tensorboard periodically write model logs that can be visualized in tensorboardhttps www tensorflow orgtensorboard more details in the section visualization csvlogger streams loss and metrics data to a csv file etc see the callbacks documentationapicallbacks for the complete list writing your own callback you can create a custom callback by extending the base class keras callbacks callback a callback has access to its associated model through the class property self model make sure to read the complete guide to writing custom callbacksguideswritingyourowncallbacks here s a simple example saving a list of perbatch loss values during training checkpointing models when you re training model on relatively large datasets it s crucial to save checkpoints of your model at frequent intervals the easiest way to achieve this is with the modelcheckpoint callback path where to save the model the two parameters below mean that we will overwrite the current checkpoint if and only if the valloss score has improved the saved model name will include the current epoch the modelcheckpoint callback can be used to implement faulttolerance the ability to restart training from the last saved state of the model in case training gets randomly interrupted here s a basic example prepare a directory to store all the checkpoints either restore the latest model or create a fresh one if there is no checkpoint available this callback saves the model every 100 batches we include the training loss in the saved model name you call also write your own callback for saving and restoring models for a complete guide on serialization and saving see the guide to saving and serializing modelsguidesserializationandsaving using learning rate schedules a common pattern when training deep learning models is to gradually reduce the learning as training progresses this is generally known as learning rate decay the learning decay schedule could be static fixed in advance as a function of the current epoch or the current batch index or dynamic responding to the current behavior of the model in particular the validation loss passing a schedule to an optimizer you can easily use a static learning rate decay schedule by passing a schedule object as the learningrate argument in your optimizer several builtin schedules are available exponentialdecay piecewiseconstantdecay polynomialdecay and inversetimedecay using callbacks to implement a dynamic learning rate schedule a dynamic learning rate schedule for instance decreasing the learning rate when the validation loss is no longer improving cannot be achieved with these schedule objects since the optimizer does not have access to validation metrics however callbacks do have access to all metrics including validation metrics you can thus achieve this pattern by using a callback that modifies the current learning rate on the optimizer in fact this is even builtin as the reducelronplateau callback visualizing loss and metrics during training with tensorboard the best way to keep an eye on your model during training is to use tensorboardhttps www tensorflow orgtensorboard a browserbased application that you can run locally that provides you with live plots of the loss and metrics for training and evaluation optionally visualizations of the histograms of your layer activations optionally 3d visualizations of the embedding spaces learned by your embedding layers if you have installed tensorflow with pip you should be able to launch tensorboard from the command line tensorboard logdirfullpathtoyourlogs using the tensorboard callback the easiest way to use tensorboard with a keras model and the fit method is the tensorboard callback in the simplest case just specify where you want the callback to write logs and you re good to go for more information see the documentation for the tensorboard callbackhttps keras ioapicallbackstensorboard setup we import torch tf so as to use torch dataloaders tf data datasets introduction this guide covers training evaluation and prediction inference models when using built in apis for training validation such as model fit model evaluate and model predict if you are interested in leveraging fit while specifying your own training step function see the customizing what happens in fit guide guides customizing_what_happens_in_fit if you are interested in writing your own training evaluation loops from scratch see the guide writing a training loop from scratch guides writing_a_training_loop_from_scratch in general whether you are using built in loops or writing your own model training evaluation works strictly in the same way across every kind of keras model sequential models models built with the functional api and models written from scratch via model subclassing this guide doesn t cover distributed training which is covered in our guide to multi gpu distributed training https keras io guides distributed_training api overview a first end to end example when passing data to the built in training loops of a model you should either use numpy arrays if your data is small and fits in memory subclasses of keras utils pydataset tf data dataset objects pytorch dataloader instances in the next few paragraphs we ll use the mnist dataset as numpy arrays in order to demonstrate how to use optimizers losses and metrics afterwards we ll take a close look at each of the other options let s consider the following model here we build in with the functional api but it could be a sequential model or a subclassed model as well here s what the typical end to end workflow looks like consisting of training validation on a holdout set generated from the original training data evaluation on the test data we ll use mnist data for this example preprocess the data these are numpy arrays reserve 10 000 samples for validation we specify the training configuration optimizer loss metrics optimizer loss function to minimize list of metrics to monitor we call fit which will train the model by slicing the data into batches of size batch_size and repeatedly iterating over the entire dataset for a given number of epochs we pass some validation for monitoring validation loss and metrics at the end of each epoch the returned history object holds a record of the loss values and metric values during training we evaluate the model on the test data via evaluate evaluate the model on the test data using evaluate generate predictions probabilities the output of the last layer on new data using predict now let s review each piece of this workflow in detail the compile method specifying a loss metrics and an optimizer to train a model with fit you need to specify a loss function an optimizer and optionally some metrics to monitor you pass these to the model as arguments to the compile method the metrics argument should be a list your model can have any number of metrics if your model has multiple outputs you can specify different losses and metrics for each output and you can modulate the contribution of each output to the total loss of the model you will find more details about this in the passing data to multi input multi output models section note that if you re satisfied with the default settings in many cases the optimizer loss and metrics can be specified via string identifiers as a shortcut for later reuse let s put our model definition and compile step in functions we will call them several times across different examples in this guide many built in optimizers losses and metrics are available in general you won t have to create your own losses metrics or optimizers from scratch because what you need is likely to be already part of the keras api optimizers sgd with or without momentum rmsprop adam etc losses meansquarederror kldivergence cosinesimilarity etc metrics auc precision recall etc custom losses if you need to create a custom loss keras provides three ways to do so the first method involves creating a function that accepts inputs y_true and y_pred the following example shows a loss function that computes the mean squared error between the real data and the predictions we need to one hot encode the labels to use mse if you need a loss function that takes in parameters beside y_true and y_pred you can subclass the keras losses loss class and implement the following two methods __init__ self accept parameters to pass during the call of your loss function call self y_true y_pred use the targets y_true and the model predictions y_pred to compute the model s loss let s say you want to use mean squared error but with an added term that will de incentivize prediction values far from 0 5 we assume that the categorical targets are one hot encoded and take values between 0 and 1 this creates an incentive for the model not to be too confident which may help reduce overfitting we won t know if it works until we try here s how you would do it custom metrics if you need a metric that isn t part of the api you can easily create custom metrics by subclassing the keras metrics metric class you will need to implement 4 methods __init__ self in which you will create state variables for your metric update_state self y_true y_pred sample_weight none which uses the targets y_true and the model predictions y_pred to update the state variables result self which uses the state variables to compute the final results reset_state self which reinitializes the state of the metric state update and results computation are kept separate in update_state and result respectively because in some cases the results computation might be very expensive and would only be done periodically here s a simple example showing how to implement a categoricaltruepositives metric that counts how many samples were correctly classified as belonging to a given class the state of the metric will be reset at the start of each epoch handling losses and metrics that don t fit the standard signature the overwhelming majority of losses and metrics can be computed from y_true and y_pred where y_pred is an output of your model but not all of them for instance a regularization loss may only require the activation of a layer there are no targets in this case and this activation may not be a model output in such cases you can call self add_loss loss_value from inside the call method of a custom layer losses added in this way get added to the main loss during training the one passed to compile here s a simple example that adds activity regularization note that activity regularization is built in in all keras layers this layer is just for the sake of providing a concrete example pass through layer insert activity regularization as a layer the displayed loss will be much higher than before due to the regularization component note that when you pass losses via add_loss it becomes possible to call compile without a loss function since the model already has a loss to minimize consider the following logisticendpoint layer it takes as inputs targets logits and it tracks a crossentropy loss via add_loss compute the training time loss value and add it to the layer using self add_loss return the inference time prediction tensor for predict you can use it in a model with two inputs input data targets compiled without a loss argument like this no loss argument for more information about training multi input models see the section passing data to multi input multi output models automatically setting apart a validation holdout set in the first end to end example you saw we used the validation_data argument to pass a tuple of numpy arrays x_val y_val to the model for evaluating a validation loss and validation metrics at the end of each epoch here s another option the argument validation_split allows you to automatically reserve part of your training data for validation the argument value represents the fraction of the data to be reserved for validation so it should be set to a number higher than 0 and lower than 1 for instance validation_split 0 2 means use 20 of the data for validation and validation_split 0 6 means use 60 of the data for validation the way the validation is computed is by taking the last x samples of the arrays received by the fit call before any shuffling note that you can only use validation_split when training with numpy data training evaluation using tf data datasets in the past few paragraphs you ve seen how to handle losses metrics and optimizers and you ve seen how to use the validation_data and validation_split arguments in fit when your data is passed as numpy arrays another option is to use an iterator like such as a tf data dataset a pytorch dataloader or a keras pydataset let s take look at the former the tf data api is a set of utilities in tensorflow 2 0 for loading and preprocessing data in a way that s fast and scalable for a complete guide about creating datasets see the tf data documentation https www tensorflow org guide data you can use tf data to train your keras models regardless of the backend you re using whether it s jax pytorch or tensorflow you can pass a dataset instance directly to the methods fit evaluate and predict first let s create a training dataset instance for the sake of our example we ll use the same mnist data as before shuffle and slice the dataset now we get a test dataset since the dataset already takes care of batching we don t pass a batch_size argument you can also evaluate or predict on a dataset note that the dataset is reset at the end of each epoch so it can be reused of the next epoch if you want to run training only on a specific number of batches from this dataset you can pass the steps_per_epoch argument which specifies how many training steps the model should run using this dataset before moving on to the next epoch prepare the training dataset only use the 100 batches per epoch that s 64 100 samples you can also pass a dataset instance as the validation_data argument in fit prepare the training dataset prepare the validation dataset at the end of each epoch the model will iterate over the validation dataset and compute the validation loss and validation metrics if you want to run validation only on a specific number of batches from this dataset you can pass the validation_steps argument which specifies how many validation steps the model should run with the validation dataset before interrupting validation and moving on to the next epoch prepare the training dataset prepare the validation dataset only run validation using the first 10 batches of the dataset using the validation_steps argument note that the validation dataset will be reset after each use so that you will always be evaluating on the same samples from epoch to epoch the argument validation_split generating a holdout set from the training data is not supported when training from dataset objects since this feature requires the ability to index the samples of the datasets which is not possible in general with the dataset api training evaluation using pydataset instances keras utils pydataset is a utility that you can subclass to obtain a python generator with two important properties it works well with multiprocessing it can be shuffled e g when passing shuffle true in fit a pydataset must implement two methods __getitem__ __len__ the method __getitem__ should return a complete batch if you want to modify your dataset between epochs you may implement on_epoch_end here s a quick example to fit the model pass the dataset instead as the x argument no need for a y argument since the dataset includes the targets and pass the validation dataset as the validation_data argument and no need for the batch_size argument since the dataset is already batched evaluating the model is just as easy importantly pydataset objects support three common constructor arguments that handle the parallel processing configuration workers number of workers to use in multithreading or multiprocessing typically you d set it to the number of cores on your cpu use_multiprocessing whether to use python multiprocessing for parallelism setting this to true means that your dataset will be replicated in multiple forked processes this is necessary to gain compute level rather than i o level benefits from parallelism however it can only be set to true if your dataset can be safely pickled max_queue_size maximum number of batches to keep in the queue when iterating over the dataset in a multithreaded or multipricessed setting you can reduce this value to reduce the cpu memory consumption of your dataset it defaults to 10 by default multiprocessing is disabled use_multiprocessing false and only one thread is used you should make sure to only turn on use_multiprocessing if your code is running inside a python if __name__ __main__ block in order to avoid issues here s a 4 thread non multiprocessed example training evaluation using pytorch dataloader objects all built in training and evaluation apis are also compatible with torch utils data dataset and torch utils data dataloader objects regardless of whether you re using the pytorch backend or the jax or tensorflow backends let s take a look at a simple example unlike pydataset which are batch centric pytorch dataset objects are sample centric the __len__ method returns the number of samples and the __getitem__ method returns a specific sample to use a pytorch dataset you need to wrap it into a dataloader which takes care of batching and shuffling now you can use them in the keras api just like any other iterator using sample weighting and class weighting with the default settings the weight of a sample is decided by its frequency in the dataset there are two methods to weight the data independent of sample frequency class weights sample weights class weights this is set by passing a dictionary to the class_weight argument to model fit this dictionary maps class indices to the weight that should be used for samples belonging to this class this can be used to balance classes without resampling or to train a model that gives more importance to a particular class for instance if class 0 is half as represented as class 1 in your data you could use model fit class_weight 0 1 1 0 5 here s a numpy example where we use class weights or sample weights to give more importance to the correct classification of class 5 which is the digit 5 in the mnist dataset set weight 2 for class 5 making this class 2x more important sample weights for fine grained control or if you are not building a classifier you can use sample weights when training from numpy data pass the sample_weight argument to model fit when training from tf data or any other sort of iterator yield input_batch label_batch sample_weight_batch tuples a sample weights array is an array of numbers that specify how much weight each sample in a batch should have in computing the total loss it is commonly used in imbalanced classification problems the idea being to give more weight to rarely seen classes when the weights used are ones and zeros the array can be used as a mask for the loss function entirely discarding the contribution of certain samples to the total loss here s a matching dataset example create a dataset that includes sample weights 3rd element in the return tuple shuffle and slice the dataset passing data to multi input multi output models in the previous examples we were considering a model with a single input a tensor of shape 764 and a single output a prediction tensor of shape 10 but what about models that have multiple inputs or outputs consider the following model which has an image input of shape 32 32 3 that s height width channels and a time series input of shape none 10 that s timesteps features our model will have two outputs computed from the combination of these inputs a score of shape 1 and a probability distribution over five classes of shape 5 let s plot this model so you can clearly see what we re doing here note that the shapes shown in the plot are batch shapes rather than per sample shapes at compilation time we can specify different losses to different outputs by passing the loss functions as a list if we only passed a single loss function to the model the same loss function would be applied to every output which is not appropriate here likewise for metrics since we gave names to our output layers we could also specify per output losses and metrics via a dict we recommend the use of explicit names and dicts if you have more than 2 outputs it s possible to give different weights to different output specific losses for instance one might wish to privilege the score loss in our example by giving to 2x the importance of the class loss using the loss_weights argument you could also choose not to compute a loss for certain outputs if these outputs are meant for prediction but not for training list loss version or dict loss version passing data to a multi input or multi output model in fit works in a similar way as specifying a loss function in compile you can pass lists of numpy arrays with 1 1 mapping to the outputs that received a loss function or dicts mapping output names to numpy arrays generate dummy numpy data fit on lists alternatively fit on dicts here s the dataset use case similarly as what we did for numpy arrays the dataset should return a tuple of dicts using callbacks callbacks in keras are objects that are called at different points during training at the start of an epoch at the end of a batch at the end of an epoch etc they can be used to implement certain behaviors such as doing validation at different points during training beyond the built in per epoch validation checkpointing the model at regular intervals or when it exceeds a certain accuracy threshold changing the learning rate of the model when training seems to be plateauing doing fine tuning of the top layers when training seems to be plateauing sending email or instant message notifications when training ends or where a certain performance threshold is exceeded etc callbacks can be passed as a list to your call to fit stop training when val_loss is no longer improving no longer improving being defined as no better than 1e 2 less no longer improving being further defined as for at least 2 epochs many built in callbacks are available there are many built in callbacks already available in keras such as modelcheckpoint periodically save the model earlystopping stop training when training is no longer improving the validation metrics tensorboard periodically write model logs that can be visualized in tensorboard https www tensorflow org tensorboard more details in the section visualization csvlogger streams loss and metrics data to a csv file etc see the callbacks documentation api callbacks for the complete list writing your own callback you can create a custom callback by extending the base class keras callbacks callback a callback has access to its associated model through the class property self model make sure to read the complete guide to writing custom callbacks guides writing_your_own_callbacks here s a simple example saving a list of per batch loss values during training checkpointing models when you re training model on relatively large datasets it s crucial to save checkpoints of your model at frequent intervals the easiest way to achieve this is with the modelcheckpoint callback path where to save the model the two parameters below mean that we will overwrite the current checkpoint if and only if the val_loss score has improved the saved model name will include the current epoch only save a model if val_loss has improved the modelcheckpoint callback can be used to implement fault tolerance the ability to restart training from the last saved state of the model in case training gets randomly interrupted here s a basic example prepare a directory to store all the checkpoints either restore the latest model or create a fresh one if there is no checkpoint available this callback saves the model every 100 batches we include the training loss in the saved model name you call also write your own callback for saving and restoring models for a complete guide on serialization and saving see the guide to saving and serializing models guides serialization_and_saving using learning rate schedules a common pattern when training deep learning models is to gradually reduce the learning as training progresses this is generally known as learning rate decay the learning decay schedule could be static fixed in advance as a function of the current epoch or the current batch index or dynamic responding to the current behavior of the model in particular the validation loss passing a schedule to an optimizer you can easily use a static learning rate decay schedule by passing a schedule object as the learning_rate argument in your optimizer several built in schedules are available exponentialdecay piecewiseconstantdecay polynomialdecay and inversetimedecay using callbacks to implement a dynamic learning rate schedule a dynamic learning rate schedule for instance decreasing the learning rate when the validation loss is no longer improving cannot be achieved with these schedule objects since the optimizer does not have access to validation metrics however callbacks do have access to all metrics including validation metrics you can thus achieve this pattern by using a callback that modifies the current learning rate on the optimizer in fact this is even built in as the reducelronplateau callback visualizing loss and metrics during training with tensorboard the best way to keep an eye on your model during training is to use tensorboard https www tensorflow org tensorboard a browser based application that you can run locally that provides you with live plots of the loss and metrics for training and evaluation optionally visualizations of the histograms of your layer activations optionally 3d visualizations of the embedding spaces learned by your embedding layers if you have installed tensorflow with pip you should be able to launch tensorboard from the command line tensorboard logdir full_path_to_your_logs using the tensorboard callback the easiest way to use tensorboard with a keras model and the fit method is the tensorboard callback in the simplest case just specify where you want the callback to write logs and you re good to go how often to log histogram visualizations how often to log embedding visualizations how often to write logs default once per epoch for more information see the documentation for the tensorboard callback https keras io api callbacks tensorboard
import torch
import tensorflow as tf

import os
import numpy as np
import keras
from keras import layers
from keras import ops





inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
x = layers.Dense(64, activation="relu", name="dense_2")(x)
outputs = layers.Dense(10, activation="softmax", name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)



(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()


x_train = x_train.reshape(60000, 784).astype("float32") / 255
x_test = x_test.reshape(10000, 784).astype("float32") / 255

y_train = y_train.astype("float32")
y_test = y_test.astype("float32")


x_val = x_train[-10000:]
y_val = y_train[-10000:]
x_train = x_train[:-10000]
y_train = y_train[:-10000]



model.compile(
    optimizer=keras.optimizers.RMSprop(),  
    
    loss=keras.losses.SparseCategoricalCrossentropy(),
    
    metrics=[keras.metrics.SparseCategoricalAccuracy()],
)



print("Fit model on training data")
history = model.fit(
    x_train,
    y_train,
    batch_size=64,
    epochs=2,
    
    
    
    validation_data=(x_val, y_val),
)



history.history




print("Evaluate on test data")
results = model.evaluate(x_test, y_test, batch_size=128)
print("test loss, test acc:", results)



print("Generate predictions for 3 samples")
predictions = model.predict(x_test[:3])
print("predictions shape:", predictions.shape)





model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[keras.metrics.SparseCategoricalAccuracy()],
)



model.compile(
    optimizer="rmsprop",
    loss="sparse_categorical_crossentropy",
    metrics=["sparse_categorical_accuracy"],
)




def get_uncompiled_model():
    inputs = keras.Input(shape=(784,), name="digits")
    x = layers.Dense(64, activation="relu", name="dense_1")(inputs)
    x = layers.Dense(64, activation="relu", name="dense_2")(x)
    outputs = layers.Dense(10, activation="softmax", name="predictions")(x)
    model = keras.Model(inputs=inputs, outputs=outputs)
    return model


def get_compiled_model():
    model = get_uncompiled_model()
    model.compile(
        optimizer="rmsprop",
        loss="sparse_categorical_crossentropy",
        metrics=["sparse_categorical_accuracy"],
    )
    return model







def custom_mean_squared_error(y_true, y_pred):
    return ops.mean(ops.square(y_true - y_pred), axis=-1)


model = get_uncompiled_model()
model.compile(optimizer=keras.optimizers.Adam(), loss=custom_mean_squared_error)


y_train_one_hot = ops.one_hot(y_train, num_classes=10)
model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)




class CustomMSE(keras.losses.Loss):
    def __init__(self, regularization_factor=0.1, name="custom_mse"):
        super().__init__(name=name)
        self.regularization_factor = regularization_factor

    def call(self, y_true, y_pred):
        mse = ops.mean(ops.square(y_true - y_pred), axis=-1)
        reg = ops.mean(ops.square(0.5 - y_pred), axis=-1)
        return mse + reg * self.regularization_factor


model = get_uncompiled_model()
model.compile(optimizer=keras.optimizers.Adam(), loss=CustomMSE())

y_train_one_hot = ops.one_hot(y_train, num_classes=10)
model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)





class CategoricalTruePositives(keras.metrics.Metric):
    def __init__(self, name="categorical_true_positives", **kwargs):
        super().__init__(name=name, **kwargs)
        self.true_positives = self.add_variable(
            shape=(), name="ctp", initializer="zeros"
        )

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_pred = ops.reshape(ops.argmax(y_pred, axis=1), (-1, 1))
        values = ops.cast(y_true, "int32") == ops.cast(y_pred, "int32")
        values = ops.cast(values, "float32")
        if sample_weight is not None:
            sample_weight = ops.cast(sample_weight, "float32")
            values = ops.multiply(values, sample_weight)
        self.true_positives.assign_add(ops.sum(values))

    def result(self):
        return self.true_positives

    def reset_state(self):
        
        self.true_positives.assign(0.0)


model = get_uncompiled_model()
model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(),
    metrics=[CategoricalTruePositives()],
)
model.fit(x_train, y_train, batch_size=64, epochs=3)




class ActivityRegularizationLayer(layers.Layer):
    def call(self, inputs):
        self.add_loss(ops.sum(inputs) * 0.1)
        return inputs  


inputs = keras.Input(shape=(784,), name="digits")
x = layers.Dense(64, activation="relu", name="dense_1")(inputs)


x = ActivityRegularizationLayer()(x)

x = layers.Dense(64, activation="relu", name="dense_2")(x)
outputs = layers.Dense(10, name="predictions")(x)

model = keras.Model(inputs=inputs, outputs=outputs)
model.compile(
    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)



model.fit(x_train, y_train, batch_size=64, epochs=1)




class LogisticEndpoint(keras.layers.Layer):
    def __init__(self, name=None):
        super().__init__(name=name)
        self.loss_fn = keras.losses.BinaryCrossentropy(from_logits=True)

    def call(self, targets, logits, sample_weights=None):
        
        
        loss = self.loss_fn(targets, logits, sample_weights)
        self.add_loss(loss)

        
        return ops.softmax(logits)




inputs = keras.Input(shape=(3,), name="inputs")
targets = keras.Input(shape=(10,), name="targets")
logits = keras.layers.Dense(10)(inputs)
predictions = LogisticEndpoint(name="predictions")(targets, logits)

model = keras.Model(inputs=[inputs, targets], outputs=predictions)
model.compile(optimizer="adam")  

data = {
    "inputs": np.random.random((3, 3)),
    "targets": np.random.random((3, 10)),
}
model.fit(data)





model = get_compiled_model()
model.fit(x_train, y_train, batch_size=64, validation_split=0.2, epochs=1)



model = get_compiled_model()



train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))

train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)


test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
test_dataset = test_dataset.batch(64)



model.fit(train_dataset, epochs=3)


print("Evaluate")
result = model.evaluate(test_dataset)
dict(zip(model.metrics_names, result))



model = get_compiled_model()


train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)


model.fit(train_dataset, epochs=3, steps_per_epoch=100)



model = get_compiled_model()


train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)


val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))
val_dataset = val_dataset.batch(64)

model.fit(train_dataset, epochs=1, validation_data=val_dataset)



model = get_compiled_model()


train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)


val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))
val_dataset = val_dataset.batch(64)

model.fit(
    train_dataset,
    epochs=1,
    
    
    validation_data=val_dataset,
    validation_steps=10,
)






class ExamplePyDataset(keras.utils.PyDataset):
    def __init__(self, x, y, batch_size, **kwargs):
        super().__init__(**kwargs)
        self.x = x
        self.y = y
        self.batch_size = batch_size

    def __len__(self):
        return int(np.ceil(len(self.x) / float(self.batch_size)))

    def __getitem__(self, idx):
        batch_x = self.x[idx * self.batch_size : (idx + 1) * self.batch_size]
        batch_y = self.y[idx * self.batch_size : (idx + 1) * self.batch_size]
        return batch_x, batch_y


train_py_dataset = ExamplePyDataset(x_train, y_train, batch_size=32)
val_py_dataset = ExamplePyDataset(x_val, y_val, batch_size=32)



model = get_compiled_model()
model.fit(
    train_py_dataset, batch_size=64, validation_data=val_py_dataset, epochs=1
)



model.evaluate(val_py_dataset)



train_py_dataset = ExamplePyDataset(x_train, y_train, batch_size=32, workers=4)
val_py_dataset = ExamplePyDataset(x_val, y_val, batch_size=32, workers=4)

model = get_compiled_model()
model.fit(
    train_py_dataset, batch_size=64, validation_data=val_py_dataset, epochs=1
)




class ExampleTorchDataset(torch.utils.data.Dataset):
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __len__(self):
        return len(self.x)

    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]


train_torch_dataset = ExampleTorchDataset(x_train, y_train)
val_torch_dataset = ExampleTorchDataset(x_val, y_val)



train_dataloader = torch.utils.data.DataLoader(
    train_torch_dataset, batch_size=32, shuffle=True
)
val_dataloader = torch.utils.data.DataLoader(
    val_torch_dataset, batch_size=32, shuffle=True
)



model = get_compiled_model()
model.fit(
    train_dataloader, batch_size=64, validation_data=val_dataloader, epochs=1
)
model.evaluate(val_dataloader)







class_weight = {
    0: 1.0,
    1: 1.0,
    2: 1.0,
    3: 1.0,
    4: 1.0,
    
    
    5: 2.0,
    6: 1.0,
    7: 1.0,
    8: 1.0,
    9: 1.0,
}

print("Fit with class weight")
model = get_compiled_model()
model.fit(x_train, y_train, class_weight=class_weight, batch_size=64, epochs=1)



sample_weight = np.ones(shape=(len(y_train),))
sample_weight[y_train == 5] = 2.0

print("Fit with sample weight")
model = get_compiled_model()
model.fit(
    x_train, y_train, sample_weight=sample_weight, batch_size=64, epochs=1
)



sample_weight = np.ones(shape=(len(y_train),))
sample_weight[y_train == 5] = 2.0



train_dataset = tf.data.Dataset.from_tensor_slices(
    (x_train, y_train, sample_weight)
)


train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

model = get_compiled_model()
model.fit(train_dataset, epochs=1)



image_input = keras.Input(shape=(32, 32, 3), name="img_input")
timeseries_input = keras.Input(shape=(None, 10), name="ts_input")

x1 = layers.Conv2D(3, 3)(image_input)
x1 = layers.GlobalMaxPooling2D()(x1)

x2 = layers.Conv1D(3, 3)(timeseries_input)
x2 = layers.GlobalMaxPooling1D()(x2)

x = layers.concatenate([x1, x2])

score_output = layers.Dense(1, name="score_output")(x)
class_output = layers.Dense(5, name="class_output")(x)

model = keras.Model(
    inputs=[image_input, timeseries_input], outputs=[score_output, class_output]
)



keras.utils.plot_model(
    model, "multi_input_and_output_model.png", show_shapes=True
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[
        keras.losses.MeanSquaredError(),
        keras.losses.CategoricalCrossentropy(),
    ],
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[
        keras.losses.MeanSquaredError(),
        keras.losses.CategoricalCrossentropy(),
    ],
    metrics=[
        [
            keras.metrics.MeanAbsolutePercentageError(),
            keras.metrics.MeanAbsoluteError(),
        ],
        [keras.metrics.CategoricalAccuracy()],
    ],
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={
        "score_output": keras.losses.MeanSquaredError(),
        "class_output": keras.losses.CategoricalCrossentropy(),
    },
    metrics={
        "score_output": [
            keras.metrics.MeanAbsolutePercentageError(),
            keras.metrics.MeanAbsoluteError(),
        ],
        "class_output": [keras.metrics.CategoricalAccuracy()],
    },
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={
        "score_output": keras.losses.MeanSquaredError(),
        "class_output": keras.losses.CategoricalCrossentropy(),
    },
    metrics={
        "score_output": [
            keras.metrics.MeanAbsolutePercentageError(),
            keras.metrics.MeanAbsoluteError(),
        ],
        "class_output": [keras.metrics.CategoricalAccuracy()],
    },
    loss_weights={"score_output": 2.0, "class_output": 1.0},
)




model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[None, keras.losses.CategoricalCrossentropy()],
)


model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss={"class_output": keras.losses.CategoricalCrossentropy()},
)



model.compile(
    optimizer=keras.optimizers.RMSprop(1e-3),
    loss=[
        keras.losses.MeanSquaredError(),
        keras.losses.CategoricalCrossentropy(),
    ],
)


img_data = np.random.random_sample(size=(100, 32, 32, 3))
ts_data = np.random.random_sample(size=(100, 20, 10))
score_targets = np.random.random_sample(size=(100, 1))
class_targets = np.random.random_sample(size=(100, 5))


model.fit(
    [img_data, ts_data], [score_targets, class_targets], batch_size=32, epochs=1
)


model.fit(
    {"img_input": img_data, "ts_input": ts_data},
    {"score_output": score_targets, "class_output": class_targets},
    batch_size=32,
    epochs=1,
)



train_dataset = tf.data.Dataset.from_tensor_slices(
    (
        {"img_input": img_data, "ts_input": ts_data},
        {"score_output": score_targets, "class_output": class_targets},
    )
)
train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)

model.fit(train_dataset, epochs=1)



model = get_compiled_model()

callbacks = [
    keras.callbacks.EarlyStopping(
        
        monitor="val_loss",
        
        min_delta=1e-2,
        
        patience=2,
        verbose=1,
    )
]
model.fit(
    x_train,
    y_train,
    epochs=20,
    batch_size=64,
    callbacks=callbacks,
    validation_split=0.2,
)




class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs):
        self.per_batch_losses = []

    def on_batch_end(self, batch, logs):
        self.per_batch_losses.append(logs.get("loss"))




model = get_compiled_model()

callbacks = [
    keras.callbacks.ModelCheckpoint(
        
        
        
        
        
        filepath="mymodel_{epoch}.keras",
        save_best_only=True,  
        monitor="val_loss",
        verbose=1,
    )
]
model.fit(
    x_train,
    y_train,
    epochs=2,
    batch_size=64,
    callbacks=callbacks,
    validation_split=0.2,
)




checkpoint_dir = "./ckpt"
if not os.path.exists(checkpoint_dir):
    os.makedirs(checkpoint_dir)


def make_or_restore_model():
    
    
    checkpoints = [
        checkpoint_dir + "/" + name for name in os.listdir(checkpoint_dir)
    ]
    if checkpoints:
        latest_checkpoint = max(checkpoints, key=os.path.getctime)
        print("Restoring from", latest_checkpoint)
        return keras.models.load_model(latest_checkpoint)
    print("Creating a new model")
    return get_compiled_model()


model = make_or_restore_model()
callbacks = [
    
    
    keras.callbacks.ModelCheckpoint(
        filepath=checkpoint_dir + "/model-loss={loss:.2f}.keras", save_freq=100
    )
]
model.fit(x_train, y_train, epochs=1, callbacks=callbacks)





initial_learning_rate = 0.1
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True
)

optimizer = keras.optimizers.RMSprop(learning_rate=lr_schedule)







keras.callbacks.TensorBoard(
    log_dir="/full_path_to_your_logs",
    histogram_freq=0,  
    embeddings_freq=0,  
    update_freq="epoch",
)