<jupyter_start><jupyter_text>Copyright 2020 The TensorFlow Authors.<jupyter_code>#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.<jupyter_output><empty_output><jupyter_text>FaceSSD Fairness Indicators Example Colab       View on TensorFlow.org        Run in Google Colab        View on GitHub        Download notebook   OverviewIn this activity, you'll use [Fairness Indicators](https://www.tensorflow.org/tfx/guide/fairness_indicators) to explore the [FaceSSD predictions on Labeled Faces in the Wild dataset](https://modelcards.withgoogle.com/face-detection). Fairness Indicators is a suite of tools built on top of [TensorFlow Model Analysis](https://www.tensorflow.org/tfx/model_analysis/get_started) that enable regular evaluation of fairness metrics in product pipelines.About the DatasetIn this exercise, you'll work with the FaceSSD prediction dataset, approximately 200k different image predictions and groundtruths generated by FaceSSD API.About the Tools[TensorFlow Model Analysis](https://www.tensorflow.org/tfx/model_analysis/get_started) is a library for evaluating both TensorFlow and non-TensorFlow machine learning models. It allows users to evaluate their models on large amounts of data in a distributed manner, computing in-graph and other metrics over different slices of data and visualize in notebooks.[TensorFlow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started) is one tool you can use to analyze your data. You can use it to find potential problems in your data, such as missing values and data imbalances, that can lead to Fairness disparities.With [Fairness Indicators](https://www.tensorflow.org/tfx/guide/fairness_indicators), users will be able to: * Evaluate model performance, sliced across defined groups of users* Feel confident about results with confidence intervals and evaluations at multiple thresholds  ImportingRun the following code to install the fairness_indicators library. This package contains the tools we'll be using in this exercise. Restart Runtime may be requested but is not necessary.<jupyter_code>!pip install fairness-indicators
import os
import tempfile
import apache_beam as beam
import numpy as np
import pandas as pd
from datetime import datetime

import tensorflow_hub as hub
import tensorflow as tf
import tensorflow_model_analysis as tfma
import tensorflow_data_validation as tfdv
from tensorflow_model_analysis.addons.fairness.post_export_metrics import fairness_indicators
from tensorflow_model_analysis.addons.fairness.view import widget_view
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_predict as agnostic_predict
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_evaluate_graph
from tensorflow_model_analysis.model_agnostic_eval import model_agnostic_extractor

from witwidget.notebook.visualization import WitConfigBuilder
from witwidget.notebook.visualization import WitWidget<jupyter_output><empty_output><jupyter_text>Download and Understand the Data [Labeled Faces in the Wild](http://vis-www.cs.umass.edu/lfw/) is a public benchmark dataset for face verification, also known as pair matching. LFW contains more than 13,000 images of faces collected from the web.We ran FaceSSD predictions on this dataset to predict whether a face is present in a given image. In this Colab, we will slice data according to gender to observe if there are any significant differences between model performance for different gender groups.If there is more than one face in an image, gender is labeled as "MISSING".We've hosted the dataset on Google Cloud Platform for convenience. Run the following code to download the data from GCP, the data will take about a minute to download and analyze.<jupyter_code>data_location = tf.keras.utils.get_file('lfw_dataset.tf', 'https://storage.googleapis.com/facessd_dataset/lfw_dataset.tfrecord')

stats = tfdv.generate_statistics_from_tfrecord(data_location=data_location)
tfdv.visualize_statistics(stats)<jupyter_output><empty_output><jupyter_text>Defining Constants<jupyter_code>BASE_DIR = tempfile.gettempdir()

tfma_eval_result_path = os.path.join(BASE_DIR, 'tfma_eval_result')

compute_confidence_intervals = True

slice_key = 'object/groundtruth/Gender'
label_key = 'object/groundtruth/face'
prediction_key = 'object/prediction/face'

feature_map = {
    slice_key:
        tf.io.FixedLenFeature([], tf.string, default_value=['none']),
    label_key:
        tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
    prediction_key:
        tf.io.FixedLenFeature([], tf.float32, default_value=[0.0]),
}<jupyter_output><empty_output><jupyter_text>Model Agnostic Config for TFMA<jupyter_code>model_agnostic_config = agnostic_predict.ModelAgnosticConfig(
    label_keys=[label_key],
    prediction_keys=[prediction_key],
    feature_spec=feature_map)

model_agnostic_extractors = [
    model_agnostic_extractor.ModelAgnosticExtractor(
        model_agnostic_config=model_agnostic_config, desired_batch_size=3),
    tfma.extractors.slice_key_extractor.SliceKeyExtractor(
          [tfma.slicer.SingleSliceSpec(),
           tfma.slicer.SingleSliceSpec(columns=[slice_key])])
]<jupyter_output><empty_output><jupyter_text>Fairness Callbacks and Computing Fairness Metrics<jupyter_code># Helper class for counting examples in beam PCollection
class CountExamples(beam.CombineFn):
    def __init__(self, message):
      self.message = message

    def create_accumulator(self):
      return 0

    def add_input(self, current_sum, element):
      return current_sum + 1

    def merge_accumulators(self, accumulators): 
      return sum(accumulators)

    def extract_output(self, final_sum):
      if final_sum:
        print("%s: %d"%(self.message, final_sum))
metrics_callbacks = [
  tfma.post_export_metrics.fairness_indicators(
      thresholds=[0.1, 0.3, 0.5, 0.7, 0.9],
      labels_key=label_key,
      target_prediction_keys=[prediction_key]),
  tfma.post_export_metrics.auc(
      curve='PR',
      labels_key=label_key,
      target_prediction_keys=[prediction_key]),
]

eval_shared_model = tfma.types.EvalSharedModel(
    add_metrics_callbacks=metrics_callbacks,
    construct_fn=model_agnostic_evaluate_graph.make_construct_fn(
        add_metrics_callbacks=metrics_callbacks,
        config=model_agnostic_config))

with beam.Pipeline() as pipeline:
  # Read data.
  data = (
      pipeline
      | 'ReadData' >> beam.io.ReadFromTFRecord(data_location))

  # Count all examples.
  data_count = (
      data | 'Count number of examples' >> beam.CombineGlobally(
          CountExamples('Before filtering "Gender:MISSING"')))

  # If there are more than one face in image, the gender feature is 'MISSING'
  # and we are filtering that image out.
  def filter_missing_gender(element):
    example = tf.train.Example.FromString(element)
    if example.features.feature[slice_key].bytes_list.value[0] != b'MISSING':
      yield element

  filtered_data = (
      data
      | 'Filter Missing Gender' >> beam.ParDo(filter_missing_gender))
  
  # Count after filtering "Gender:MISSING".
  filtered_data_count = (
      filtered_data | 'Count number of examples after filtering'
      >> beam.CombineGlobally(
          CountExamples('After filtering "Gender:MISSING"')))

  # Because LFW data set has always faces by default, we are adding
  # labels as 1.0 for all images.
  def add_face_groundtruth(element):
    example = tf.train.Example.FromString(element)
    example.features.feature[label_key].float_list.value[:] = [1.0]
    yield example.SerializeToString()

  final_data = (
      filtered_data
      | 'Add Face Groundtruth' >> beam.ParDo(add_face_groundtruth))

  # Run TFMA.
  _ = (
      final_data
      | 'ExtractEvaluateAndWriteResults' >>
       tfma.ExtractEvaluateAndWriteResults(
                 eval_shared_model=eval_shared_model,
                 compute_confidence_intervals=compute_confidence_intervals,
                 output_path=tfma_eval_result_path,
                 extractors=model_agnostic_extractors))

eval_result = tfma.load_eval_result(output_path=tfma_eval_result_path)<jupyter_output><empty_output><jupyter_text>Render Fairness IndicatorsRender the Fairness Indicators widget with the exported evaluation results.Below you will see bar charts displaying performance of each slice of the data on selected metrics. You can adjust the baseline comparison slice as well as the displayed threshold(s) using the drop down menus at the top of the visualization.A relevant metric for this use case is true positive rate, also known as recall. Use the selector on the left hand side to choose the graph for true_positive_rate. These metric values match the values displayed on the [model card](https://modelcards.withgoogle.com/face-detection).For some photos, gender is labeled as young instead of male or female, if the person in the photo is too young to be accurately annotated.<jupyter_code>widget_view.render_fairness_indicator(eval_result=eval_result,
                                      slicing_column=slice_key)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07053291536050156, 'fraction_numerical': 0.00438871473354232, 'mean_word_length': 4.900739827373613, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 41, 'https://': 9, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '260654', 'license': 'Apache-2.0', 'n_tokens_mistral': 2770, 'n_tokens_neox': 2648, 'n_words': 836, 'path': 'fairness_indicators/documentation/tutorials/Facessd_Fairness_Indicators_Example_Colab.ipynb', 'repo_name': 'isabella232/fairness-indicators', 'size': '9570'}