<jupyter_start><jupyter_text>TF RankingIn this Notebook, we run through a simplified example to highlight some of the features of the TF Ranking library and demonstrate an end-to-end execution.The general recipe is a short list of four main steps:1.   Compose a function to **read** input data and prepare a Tensorflow Dataset;2.   Define a **scoring** function that, given a (set of) query-document feature vector(s), produces a score indicating the query's level of relevance to the document;3.   Create a **loss** function that measures how far off the produced scores from step (2) are from the ground truth; and,4.   Define evaluation **metrics**.A final step makes use of standard Tensorflow API to create, train, and evaluate a model.We have included in the TF Ranking library a default implementation of data readers (in the `tensorflow_ranking.data` module), loss functions (in `tensorflow_ranking.losses`), and popular evaluation metrics (in `tensorflow_ranking.metrics`) that may be further tailored to your needs as we shall show later in this Notebook. PreparationIn what follows, we will assume the existence of a dataset that is split into training and test sets and that are stored at `data/train.txt` and `data/test.txt` respectively. We further assume that the dataset is in the LibSVM format and lines in the training and test files are sorted by query ID -- an assumption that holds for many popular learning-to-rank benchmark datasets.We have included in our release a toy (randomly generated) dataset in the `data/` directory. However, to learn a more interesting model, you may copy your dataset of choice to the `data/` directory. Please ensure the format of your dataset conforms to the requirements above. Alternatively, you may edit this Notebook to plug in a customized input pipeline for a non-comformant dataset.  Get Started with TF Ranking       Run in Google Colab        View source on GitHub    Dependencies and Global VariablesLet us start by importing libraries that will be used throughout this Notebook. We also enable the "eager execution" mode for convenience and demonstration purposes.<jupyter_code>! pip install tensorflow_ranking
import tensorflow as tf
import tensorflow_ranking as tfr

tf.enable_eager_execution()
tf.executing_eagerly()<jupyter_output><empty_output><jupyter_text>Next, we will download a dummy dataset in LibSVM format.Note that you can replace these datasets with public or custom datasets. We also define some global parameters.<jupyter_code>! wget -O "/tmp/train.txt" "https://raw.githubusercontent.com/tensorflow/ranking/master/tensorflow_ranking/examples/data/train.txt"
! wget -O "/tmp/test.txt" "https://raw.githubusercontent.com/tensorflow/ranking/master/tensorflow_ranking/examples/data/test.txt"
# Store the paths to files containing training and test instances.
# As noted above, we will assume the data is in the LibSVM format
# and that the content of each file is sorted by query ID.
_TRAIN_DATA_PATH="/tmp/train.txt"
_TEST_DATA_PATH="/tmp/test.txt"

# Define a loss function. To find a complete list of available
# loss functions or to learn how to add your own custom function
# please refer to the tensorflow_ranking.losses module.
_LOSS="pairwise_logistic_loss"

# In the TF-Ranking framework, a training instance is represented
# by a Tensor that contains features from a list of documents
# associated with a single query. For simplicity, we fix the shape
# of these Tensors to a maximum list size and call it "list_size,"
# the maximum number of documents per query in the dataset.
# In this demo, we take the following approach:
#   * If a query has fewer documents, its Tensor will be padded
#     appropriately.
#   * If a query has more documents, we shuffle its list of
#     documents and trim the list down to the prescribed list_size.
_LIST_SIZE=100

# The total number of features per query-document pair.
# We set this number to the number of features in the MSLR-Web30K
# dataset.
_NUM_FEATURES=136

# Parameters to the scoring function.
_BATCH_SIZE=32
_HIDDEN_LAYER_DIMS=["20", "10"]<jupyter_output><empty_output><jupyter_text>Input PipelineThe first step to construct an input pipeline that reads your dataset and produces a `tensorflow.data.Dataset` object. In this example, we will invoke a LibSVM parser that is included in the `tensorflow_ranking.data` module to generate a `Dataset` from a given file.We parameterize this function by a `path` argument so that the function can be used to read both training and test data files.<jupyter_code>def input_fn(path):
  train_dataset = tf.data.Dataset.from_generator(
      tfr.data.libsvm_generator(path, _NUM_FEATURES, _LIST_SIZE),
      output_types=(
          {str(k): tf.float32 for k in range(1,_NUM_FEATURES+1)},
          tf.float32
      ),
      output_shapes=(
          {str(k): tf.TensorShape([_LIST_SIZE, 1])
            for k in range(1,_NUM_FEATURES+1)},
          tf.TensorShape([_LIST_SIZE])
      )
  )

  train_dataset = train_dataset.shuffle(1000).repeat().batch(_BATCH_SIZE)
  return train_dataset.make_one_shot_iterator().get_next()<jupyter_output><empty_output><jupyter_text>Scoring FunctionNext, we turn to the scoring function which is arguably at the heart of a TF Ranking model. The idea is to compute a relevance score for a (set of) query-document pair(s). The TF-Ranking model will use training data to learn this function.Here we formulate a scoring function using a feed forward network. The function takes the features of a single example (i.e., query-document pair) and produces a relevance score.<jupyter_code>def example_feature_columns():
  """Returns the example feature columns."""
  feature_names = [
      "%d" % (i + 1) for i in range(0, _NUM_FEATURES)
  ]
  return {
      name: tf.feature_column.numeric_column(
          name, shape=(1,), default_value=0.0) for name in feature_names
  }

def make_score_fn():
  """Returns a scoring function to build `EstimatorSpec`."""

  def _score_fn(context_features, group_features, mode, params, config):
    """Defines the network to score a documents."""
    del params
    del config
    # Define input layer.
    example_input = [
        tf.layers.flatten(group_features[name])
        for name in sorted(example_feature_columns())
    ]
    input_layer = tf.concat(example_input, 1)

    cur_layer = input_layer
    for i, layer_width in enumerate(int(d) for d in _HIDDEN_LAYER_DIMS):
      cur_layer = tf.layers.dense(
          cur_layer,
          units=layer_width,
          activation="tanh")

    logits = tf.layers.dense(cur_layer, units=1)
    return logits

  return _score_fn<jupyter_output><empty_output><jupyter_text>Evaluation MetricsWe have provided an implementation of popular Information Retrieval evalution metrics in the TF Ranking library.<jupyter_code>def eval_metric_fns():
  """Returns a dict from name to metric functions.

  This can be customized as follows. Care must be taken when handling padded
  lists.

  def _auc(labels, predictions, features):
    is_label_valid = tf_reshape(tf.greater_equal(labels, 0.), [-1, 1])
    clean_labels = tf.boolean_mask(tf.reshape(labels, [-1, 1], is_label_valid)
    clean_pred = tf.boolean_maks(tf.reshape(predictions, [-1, 1], is_label_valid)
    return tf.metrics.auc(clean_labels, tf.sigmoid(clean_pred), ...)
  metric_fns["auc"] = _auc

  Returns:
    A dict mapping from metric name to a metric function with above signature.
  """
  metric_fns = {}
  metric_fns.update({
      "metric/ndcg@%d" % topn: tfr.metrics.make_ranking_metric_fn(
          tfr.metrics.RankingMetricKey.NDCG, topn=topn)
      for topn in [1, 3, 5, 10]
  })

  return metric_fns<jupyter_output><empty_output><jupyter_text>Putting It All TogetherWe are now ready to put all of the components above together and create an `Estimator` that can be used to train and evaluate a model.<jupyter_code>def get_estimator(hparams):
  """Create a ranking estimator.

  Args:
    hparams: (tf.contrib.training.HParams) a hyperparameters object.

  Returns:
    tf.learn `Estimator`.
  """
  def _train_op_fn(loss):
    """Defines train op used in ranking head."""
    return tf.contrib.layers.optimize_loss(
        loss=loss,
        global_step=tf.train.get_global_step(),
        learning_rate=hparams.learning_rate,
        optimizer="Adagrad")

  ranking_head = tfr.head.create_ranking_head(
      loss_fn=tfr.losses.make_loss_fn(_LOSS),
      eval_metric_fns=eval_metric_fns(),
      train_op_fn=_train_op_fn)

  return tf.estimator.Estimator(
      model_fn=tfr.model.make_groupwise_ranking_fn(
          group_score_fn=make_score_fn(),
          group_size=1,
          transform_fn=None,
          ranking_head=ranking_head),
      params=hparams)<jupyter_output><empty_output><jupyter_text>Let us instantiate and initialize the `Estimator` we defined above.<jupyter_code>hparams = tf.contrib.training.HParams(learning_rate=0.05)
ranker = get_estimator(hparams)<jupyter_output><empty_output><jupyter_text>Now that we have a correctly initialized `Estimator`, we will train a model using the training data. We encourage you to experiment with different number of steps here and below.<jupyter_code>ranker.train(input_fn=lambda: input_fn(_TRAIN_DATA_PATH), steps=100)<jupyter_output><empty_output><jupyter_text>Finally, let us evaluate our model on the test set.<jupyter_code>ranker.evaluate(input_fn=lambda: input_fn(_TEST_DATA_PATH), steps=100)<jupyter_output><empty_output><jupyter_text>VisualizationThe train and evaluation steps above by default store checkpoints, metrics, and other useful information about your network to a temporary directory on disk. We encourage you to visualize this data using [Tensorboard](http://www.tensorflow.org/guide/summaries_and_tensorboard). In particular, you can launch Tensorboard and point it to where your model data is stored as follows:First, let's find out the path to the log directory created by the process above.<jupyter_code>ranker.model_dir<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07738452309538092, 'fraction_numerical': 0.00609878024395121, 'mean_word_length': 4.487109160724081, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '24252', 'license': 'Apache-2.0', 'n_tokens_mistral': 2962, 'n_tokens_neox': 2864, 'n_words': 1084, 'path': 'tensorflow_ranking/examples/tf_ranking_libsvm.ipynb', 'repo_name': 'johnvorsten/ranking', 'size': '10002'}