<jupyter_start><jupyter_text><!-- ---title: How to convert pure PyTorch code to Igniteweight: 2downloads: truesidebar: truetags:  - training loop  - validation loop  - metrics  - checkpoints  - progress bar  - logging--- --> How to convert pure PyTorch code to Ignite  In this guide, we will show how PyTorch code components can be converted into compact and flexible PyTorch-Ignite code. ![Convert PyTorch to Ignite](assets/convert-pytorch2ignite.gif) Since Ignite focuses on the training and validation pipeline, the code for models, datasets, optimizers, etc will remain user-defined and in pure PyTorch.<jupyter_code>model = ...
train_loader = ...
val_loader = ...
optimizer = ...
criterion = ...<jupyter_output><empty_output><jupyter_text>Training Loop to `trainer`A typical PyTorch training loop processes a single batch of data, passes it through the `model`, calculates `loss`, etc as below:```pythonfor batch in train_loader:    model.train()    inputs, targets = batch    optimizer.zero_grad()    outputs = model(inputs)    loss = criterion(outputs, targets)    loss.backward()    optimizer.step()``` To convert the above code into Ignite we need to move the code or steps taken to process a single batch of data while training under a function (`train_step()` below). This function will take `engine` and `batch` (current batch of data) as arguments and can return any data (usually the loss) that can be accessed via `engine.state.output`. We pass this function to `Engine` which creates a `trainer` object.<jupyter_code>from ignite.engine import Engine


def train_step(engine, batch):
    model.train()
    inputs, targets = batch
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()
    return loss.item()


trainer = Engine(train_step)<jupyter_output><empty_output><jupyter_text>There are other [helper methods](https://pytorch.org/ignite/engine.htmlhelper-methods-to-define-supervised-trainer-and-evaluator) that directly create the `trainer` object without writing a custom function for some common use cases like [supervised training](https://pytorch.org/ignite/generated/ignite.engine.create_supervised_trainer.htmlignite.engine.create_supervised_trainer) and [truncated backprop through time](https://pytorch.org/ignite/contrib/engines.htmlignite.contrib.engines.tbptt.create_supervised_tbptt_trainer).  Validation Loop to `evaluator`The validation loop typically makes predictions (`y_pred` below) on the `val_loader` batch by batch and uses them to calculate evaluation metrics (Accuracy, Intersection over Union, etc) as below:```pythonmodel.eval()num_correct = 0num_examples = 0for batch in val_loader:    x, y = batch    y_pred = model(x)    correct = torch.eq(torch.round(y_pred).type(y.type()), y).view(-1)    num_correct = torch.sum(correct).item()    num_examples = correct.shape[0]    print(f"Epoch: {epoch},  Accuracy: {num_correct / num_examples}")``` We will convert this to Ignite in two steps by separating the validation and metrics logic.We will move the model evaluation logic under another function (`validation_step()` below) which receives the same parameters as `train_step()` and processes a single batch of data to return some output (usually the predicted and actual value which can be used to calculate metrics) stored in `engine.state.output`. Another instance (called `evaluator` below) of `Engine` is created by passing the `validation_step()` function.<jupyter_code>def validation_step(engine, batch):
    model.eval()
    with torch.no_grad():
        x, y = batch
        y_pred = model(x)

    return y_pred, y
    
    
evaluator = Engine(validation_step)<jupyter_output><empty_output><jupyter_text>Similar to the training loop, there are [helper methods](https://pytorch.org/ignite/engine.htmlhelper-methods-to-define-supervised-trainer-and-evaluator) to avoid writing this custom evaluation function like [`create_supervised_evaluator`](https://pytorch.org/ignite/generated/ignite.engine.create_supervised_evaluator.htmlignite.engine.create_supervised_evaluator).**Note**: You can create different evaluators for training, validation, and testing if they serve different purposes. A common practice is to have two separate evaluators for training and validation, since the results of the validation evaluator are helpful in determining the best model to save after training.  Switch to built-in MetricsThen we can replace the code for calculating metrics like accuracy and instead use several [out-of-the-box metrics](https://pytorch.org/ignite/metrics.htmlcomplete-list-of-metrics) that Ignite provides or write a custom one (refer [here](https://pytorch.org/ignite/metrics.htmlhow-to-create-a-custom-metric)). The metrics will be computed using the `evaluator`'s output. Finally, we attach these metrics to the `evaluator` by providing a key name ("accuracy" below) so they can be accessed via `engine.state.metrics[key_name]`.<jupyter_code>from ignite.metrics import Accuracy

Accuracy().attach(evaluator, "accuracy")<jupyter_output><empty_output><jupyter_text>Organizing code into Events and HandlersNext, we need to identify any code that is triggered when an event occurs. Examples of events can be the start of an iteration, completion of an epoch, or even the start of backprop. We already provide some predefined events (complete list [here](https://pytorch.org/ignite/generated/ignite.engine.events.Events.htmlignite.engine.events.Events)) however we can also create custom ones (refer [here](https://pytorch.org/ignite/concepts.htmlcustom-events)). We move the event-specific code to different handlers (named functions, lambdas, class functions) which are attached to these events and executed whenever a specific event happens. Here are some common handlers:  Running `evaluator`We can convert the code that runs the `evaluator` on the training/validation/test dataset after `validate_every` epoch:```pythonif epoch % validate_every == 0:     Validation logic```by attaching a handler to a built-in event `EPOCH_COMPLETED` like:<jupyter_code>from ignite.engine import Events

validate_every = 10


@trainer.on(Events.EPOCH_COMPLETED(every=validate_every))
def run_validation():
    evaluator.run(val_loader)<jupyter_output><empty_output><jupyter_text>Logging metricsSimilarly, we can log the validation metrics in another handler or combine it with the above handler.<jupyter_code>@trainer.on(Events.EPOCH_COMPLETED(every=validate_every))
def log_validation():
    metrics = evaluator.state.metrics
    print(f"Epoch: {trainer.state.epoch},  Accuracy: {metrics['accuracy']}")<jupyter_output><empty_output><jupyter_text>Progress BarWe use a built-in wrapper around `tqdm` called [`ProgressBar()`](https://pytorch.org/ignite/generated/ignite.contrib.handlers.tqdm_logger.htmlmodule-ignite.contrib.handlers.tqdm_logger).<jupyter_code>from ignite.contrib.handlers import ProgressBar

ProgressBar().attach(trainer)<jupyter_output><empty_output><jupyter_text>CheckpointingInstead of saving all models after `checkpoint_every` epoch:```pythonif epoch % checkpoint_every == 0:    checkpoint(model, optimizer, "checkpoint_dir")```we can smartly save the best `n_saved` models (depending on `evaluator.state.metrics`), and the state of `optimizer` and `trainer` via the built-in [`Checkpoint()`](https://pytorch.org/ignite/generated/ignite.handlers.checkpoint.Checkpoint.htmlcheckpoint).<jupyter_code>from ignite.handlers import Checkpoint

checkpoint_every = 5
checkpoint_dir = ...


checkpointer = Checkpoint(
    to_save={'model': model, 'optimizer': optimizer, 'trainer': trainer},
    save_handler=checkpoint_dir, n_saved=2
)
trainer.add_event_handler(
    Events.EPOCH_COMPLETED(every=checkpoint_every), checkpointer
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09595894804361771, 'fraction_numerical': 0.0015394483643361128, 'mean_word_length': 5.601185436071126, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '561760', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2239, 'n_tokens_neox': 2211, 'n_words': 728, 'path': 'how-to-guides/02-convert-pytorch-to-ignite.ipynb', 'repo_name': 'H4dr1en/examples', 'size': '7795'}