<commit_before># Trim insignificant weights

This document provides an overview on model pruning to help you determine how it
fits with your usecase. To dive right into the code, see the
[Pruning with Keras](pruning_with_keras.ipynb) tutorial
and the [API docs](../../api_docs/python). For additional details on how to use
the Keras API, a deep dive into pruning, and documentation on more advanced
usage patterns, see the
[Train sparse models](train_sparse_models.md) guide.

## Overview

Magnitude-based weight pruning gradually zeroes out model weights during the
training process to achieve model sparsity. Sparse models are easier to
compress, and we can skip the zeroes during inference for latency improvements.

This technique brings improvements via model compression. In the future,
framework support for this technique will provide latency improvements. We've
seen up to 6x improvements in model compression with minimal loss of accuracy.

The technique is used in production in various speech applications, such as
speech recognition and text-to-speech, and has been experimented on across
various vision and translation models.

<!-- TODO: TF 1.14 ... TODO: roadmap ok? -->

Users can apply this technique using APIs for Keras on Tensorflow 1.x for
versions 1.14+ in both graph and eager execution.

Note: The pruning API is only compatible with `tf.distribute` when using graph
execution. It is on our roadmap to bring full support to TF 2.x and eager
execution.

## Results

### Image Classification

<figure>
  <table>
    <tr>
      <th>Model</th>
      <th>Non-sparse Top-1 Accuracy </th>
      <th>Sparse Accuracy </th>
      <th>Sparsity </th>
    </tr>
    <tr>
      <td rowspan=3>InceptionV3</td>
      <td rowspan=3>78.1%</td>
      <td>78.0%</td>
      <td>50%</td>
    </tr>
    <tr>
      <td>76.1%</td><td>75%</td>
    </tr>
    <tr><commit_msg>Internal<commit_after># Trim insignificant weights

This document provides an overview on model pruning to help you determine how it
fits with your usecase. To dive right into the code, see the
[Pruning with Keras](pruning_with_keras.ipynb) tutorial
and the [API docs](../../api_docs/python). For additional details on how to use
the Keras API, a deep dive into pruning, and documentation on more advanced
usage patterns, see the
[Train sparse models](train_sparse_models.md) guide.

## Overview

Magnitude-based weight pruning gradually zeroes out model weights during the
training process to achieve model sparsity. Sparse models are easier to
compress, and we can skip the zeroes during inference for latency improvements.

This technique brings improvements via model compression. In the future,
framework support for this technique will provide latency improvements. We've
seen up to 6x improvements in model compression with minimal loss of accuracy.

The technique is being evaluated in various speech applications, such as
speech recognition and text-to-speech, and has been experimented on across
various vision and translation models.

Users can apply this technique using APIs for Keras on Tensorflow 1.x for
versions 1.14+ and nightly in both graph and eager execution.

Note: The pruning API is only compatible with `tf.distribute` when using graph
execution. It is on our roadmap to bring full support to TF 2.x and eager
execution.

## Results

### Image Classification

<figure>
  <table>
    <tr>
      <th>Model</th>
      <th>Non-sparse Top-1 Accuracy </th>
      <th>Sparse Accuracy </th>
      <th>Sparsity </th>
    </tr>
    <tr>
      <td rowspan=3>InceptionV3</td>
      <td rowspan=3>78.1%</td>
      <td>78.0%</td>
      <td>50%</td>
    </tr>
    <tr>
      <td>76.1%</td><td>75%</td>
    </tr>
    <tr>
{'fraction_non_alphanumeric': 0.08315334773218143, 'fraction_numerical': 0.013228941684665227, 'mean_word_length': 3.75, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2012014', 'n_tokens_mistral': 1119, 'n_tokens_neox': 1052, 'n_words': 482}