<jupyter_start><jupyter_text>A practical guide towards explainability and bias evaluation in machine learning<NAME>Chief Scientist, The Institute for Ethical AI & Machine LearningDirector of ML Engineering, Seldon Technologies  Today 1) Hands on example: Company "Hype-ML" automating loan approval process  2) Terminology + why it's not about "just removing bias" 3) Data Analysis 4) Model Evaluation 4) Production Monitoring  Hype-ML - A NEW Project has come in!* Insurance company has a process where domain expert approves/rejects loan applications* They receive over 1m applications and want to automate the process Business wants it NOW!* They heard their competitor is using "Machine Learning" and business says we need to use that  The team had a look at how this worked  The team asked for DATA Business gave them an excel sheet with 25 rowsThe team pushed back, and after a while they finally got a dataset with ~8000 rows  And so it begun... The Hype-ML journey towards greatness...<jupyter_code>df_data, df_display = get_dataset_1()

df_display.head()

X = df_data.drop(label_column, axis=1).copy()
y = df_data[label_column].astype(int).values.copy()

X_train, X_valid, y_train, y_valid = \
        train_test_split(X, y, test_size=0.2, random_state=7)

print("Training size:", y_train.shape, "Testing size: ", y_valid.shape)<jupyter_output>Training size: (6400,) Testing size:  (1600,)<jupyter_text><jupyter_code># 1 layer, 100 neurons model, with softmax (0-1 probabilities)
model = build_model(X)

model.fit(f_in(X_train), y_train, epochs=10,
    batch_size=512, shuffle=True, validation_data=(f_in(X_valid), y_valid),
    callbacks=[PlotLossesKeras()], verbose=0, validation_split=0.05,)
score = model.evaluate(f_in(X_valid), y_valid, verbose=1)
print("Error %.4f: " % score[0])
print("Accuracy %.4f: " % (score[1]*100))<jupyter_output>1600/1600 [==============================] - 0s 41us/step
Error 0.0707: 
Accuracy 98.7500:<jupyter_text>Accuracy is ~98%! What a better result on a Friday evening! Time for PROD?  We push to PROD! A few weeks go by...  BUZZFEED NEWS: HYPE-ML DEPLOYS RACIST SEXIST AI!!!! ...but... but we followed the instructions in the internet!  Time to diagnose! We ask business to gather the 110 applications in production... ...and label it to understand what went wrong<jupyter_code>X_prod, y_prod = get_production_dataset()

print(X_prod.shape)
X_prod.head()
score = model.evaluate(f_in(X_prod), y_prod, verbose=1)
probabilities = model.predict(f_in(X_prod))
pred = f_out(probabilities)
print("Accuracy %.4f: " % (score[1]*100))
xai.confusion_matrix_plot(y_prod, pred)
xai.roc_plot(y_prod, pred) 
fig, ax = plt.subplots(1,2)
a = sn.countplot(y_valid, ax=ax[0]); a.set_title("TRAINING DATA"); a.set_xticklabels(["Rejected", "Approved"])
a = sn.countplot(y_prod, ax=ax[1]); a.set_title("PRODUCTION"); a.set_xticklabels(["Rejected", "Approved"])<jupyter_output><empty_output><jupyter_text>Undesired bias and explainability* Has become popular due to several high profile incidents:    * Amazon's "sexist" recruitment tool    * Microsoft's "racist" chatbot    * Negative discrimination in automated sentencing    * Black box models with complex patterns that can't be interpretable Organisations cannot take on unknown risks  This challenge goes beyond the algorithms A large ethical decision should not just fall on the shoulders of a single data scientist  The answer is not just about "removing bias"* Any non trivial decision (i.e. more than 1 option) holds a bias, without exceptions.* It's impossible to "just remove bias", as the whole purpose of ML is to discriminate towards the right answer* Societal bias carries an inherent bias - what may be "racist" for one person, may not be for another group or geography Let's see what "undesired bias" looks like  Split into two conceptual pieces 1) Statistical bias  The "error" between from where you are to where you CAN be. 2) A-priori bias  The "error" between where you CAN be to where you SHOULD be.  Statistical bias: Errors in project decisions Sub-optimal choices of accuracy metrics / cost functions Sub-optimal machine learning models chosen for the task  Lack of infrastructure or metrics required to monitor model performance in production Lack of human-in-the-loop where necessary Not using resources at disposal (e.g. domain experts, tools, etc).  A-priori bias: Bias introduced through societal or scope diversions Sub-optimal business objectives Lack of understanding of the project  Incomplete resources (data, time, domain experts, etc) Incorrectly labelled data (accident vs otherwise) Lack of relevant skillset Societal shifts in perception  Explainability is key* To identify and evaluate undersired biases* For definitions of regulatory demands (GDPR)* For compliance of processes* To identify and reduce risks (FP vs FN)  Key Point: Explainability !== Interpretability*  Having a model that can be interpreted doesn't mean it can be explained* Explainability requires us to go beyond the algorithms* Undesired bias cannot be tackled without explainability  Remember our workflow? Let's add three new steps:  Augmenting the data science workflow  The Explainability Tradeoff Introducing fail-safe mechanisms, removing features, and using simpler models may have an impact on accuracy Not all usecases demand the same level of scrutiny The ones that are more critical do require a more strict process Similar to enterprise software, the overhead to offer accountability and governance is introduced  1) Data Analysis  Points to cover1.1) Data imbalances1.2) Upsampling / downsampling1.3) Correlations1.4) Train / test set1.5) Further techniques  XAI - eXplainable AI  A set of tools to explain machine learning data https://github.com/EthicalML/XAI  Let's get the new training dataset<jupyter_code>X, y, X_train, X_valid, y_train, y_valid, X_display, y_display, df, df_display \
    = get_dataset_2()
df_display.head()<jupyter_output><empty_output><jupyter_text>1.1) Data imbalances We can visualise the imbalances by looking at the number of examples for each class<jupyter_code>im = xai.imbalance_plot(df_display, "gender", threshold=0.55, categorical_cols=["gender"])<jupyter_output><empty_output><jupyter_text>We can evaluate imbalances by the product of multiple categories<jupyter_code>im = xai.imbalance_plot(df_display, "gender", "loan" , categorical_cols=["loan", "gender"])<jupyter_output><empty_output><jupyter_text>For numeric datasets we can break it down in bins<jupyter_code>im = xai.imbalance_plot(df_display, "age" , bins=10)<jupyter_output>WARNING:root:No categorical_cols passed so inferred using np.object, np.int8 and np.bool: Index(['workclass', 'education', 'marital-status', 'occupation',
       'relationship', 'ethnicity', 'gender', 'native-country', 'loan'],
      dtype='object'). If you see an error these are not correct, please provide them as a string array as: categorical_cols=['col1', 'col2', ...]<jupyter_text>1.2) Upsampling / Downsampling<jupyter_code>im = xai.balance(df_display, "ethnicity", "loan", categorical_cols=["ethnicity", "loan"],
                upsample=0.5, downsample=0.5, bins=5)<jupyter_output><empty_output><jupyter_text>1.3 Correlations hidden in data We can identify potential correlations across variables through a dendogram visualiation<jupyter_code>corr = xai.correlations(df_display, include_categorical=True)<jupyter_output>/home/alejandro/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/scipy/stats/stats.py:245: RuntimeWarning: The input array could not be properly checked for nan values. nan values will be ignored.
  "values. nan values will be ignored.", RuntimeWarning)<jupyter_text>1.4) Balanced train/testing sets<jupyter_code>X_train_balanced, y_train_balanced, X_valid_balanced, y_valid_balanced, train_idx, test_idx = \
    xai.balanced_train_test_split(
            X, y, "gender", 
            min_per_group=300,
            max_per_group=300,
            categorical_cols=["gender", "loan"])

X_valid_balanced["loan"] = y_valid_balanced
im = xai.imbalance_plot(X_valid_balanced, "gender", "loan", categorical_cols=["gender", "loan"])<jupyter_output><empty_output><jupyter_text>1.5 Shoutout to other tools and techniqueshttps://github.com/EthicalML/awesome-production-machine-learningindustrial-strength-visualisation-libraries![](images/dataviz.jpg)  2) Model evaluation  Points to cover2.1) Standard model evaluation metrics2.2) Global model explanation techniques2.3) Black box local model explanation techniques2.4) Other libraries available  Alibi - Black Box Model Explanations A set of proven scientific techniques to explain ML models as black boxes https://github.com/SeldonIO/Alibi  Model Evaluation Metrics: White / Black Box![](images/whiteblackbox.jpg)  Model Evaluation Metrics: Global vs Local![](images/globallocal.jpg)  2.1) Standard model evaluation metrics<jupyter_code># Let's start by building our model with our newly balanced dataset
model = build_model(X)
model.fit(f_in(X_train), y_train, epochs=20, batch_size=512, shuffle=True, validation_data=(f_in(X_valid), y_valid), callbacks=[PlotLossesKeras()], verbose=0, validation_split=0.05,)
probabilities = model.predict(f_in(X_valid))
pred = f_out(probabilities)
xai.confusion_matrix_plot(y_valid, pred)
im = xai.roc_plot(y_valid, pred)
im = xai.roc_plot(y_valid, pred, df=X_valid, cross_cols=["gender"], categorical_cols=["gender"])
im = xai.metrics_plot(y_valid, pred)
im = xai.metrics_plot(y_valid, pred, df=X_valid, cross_cols=["gender"], categorical_cols="gender")<jupyter_output><empty_output><jupyter_text>2.2) Global black box model evalutaion metrics<jupyter_code>imp = xai.feature_importance(X_valid, y_valid, lambda x, y: model.evaluate(f_in(x), y, verbose=0)[1], repeat=1)<jupyter_output>/home/alejandro/miniconda3/envs/reddit-classification/lib/python3.7/site-packages/xai/__init__.py:1127: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  x[c] = tmp<jupyter_text>2.3) Local black box model evaluation metrics Overview of methods![](images/alibi-table.jpg)  Anchors  Consists of if-then rules, called the anchors, which sufficiently guarantee the explanation locally and try to maximize the area for which the explanation holds. (ArXiv: Anchors: High-Precision Model-Agnostic Explanations)<jupyter_code>from alibi.explainers import AnchorTabular

explainer = AnchorTabular(
    loan_model_alibi.predict, 
    feature_names_alibi, 
    categorical_names=category_map_alibi)

explainer.fit(
    X_train_alibi, 
    disc_perc=[25, 50, 75])

print("Explainer built")
X_test_alibi[:1]
explanation = explainer.explain(X_test_alibi[:1], threshold=0.95)

print('Anchor: %s' % (' AND '.join(explanation['names'])))
print('Precision: %.2f' % explanation['precision'])
print('Coverage: %.2f' % explanation['coverage'])<jupyter_output>Anchor: Marital Status = Separated AND Sex = Female AND Capital Gain <= 0.00
Precision: 0.97
Coverage: 0.10<jupyter_text>Counterfactual Explanations The counterfactual explanation of an outcome or a situation Y takes the form “If X had not occured, Y would not have occured” ![](images/counterfactuals7.jpg)  1.5 Shoutout to other tools and techniqueshttps://github.com/EthicalML/awesome-production-machine-learningexplaining-black-box-models-and-datasets![](images/modevallibs.jpg)  3) Production Monitoring  Key points to cover1) Design patterns for explainers2) Live demo of explainers3) Leveraging humans for explainers  Seldon Core - Production ML in K8s A language agnostic ML serving & monitoring framework in Kubernetes https://github.com/SeldonIO/seldon-core  3.1) Design patterns for explainers![](images/deployment-overview.jpg)  Setup Seldon in your kubernetes cluster<jupyter_code>%%bash
kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
helm init
kubectl rollout status deploy/tiller-deploy -n kube-system
helm install seldon-core-operator --name seldon-core-operator --repo https://storage.googleapis.com/seldon-charts
helm install seldon-core-analytics --name seldon-core-analytics --repo https://storage.googleapis.com/seldon-charts
helm install stable/ambassador --name ambassador
from sklearn.preprocessing import LabelEncoder, StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer

# feature transformation pipeline
ordinal_features = [x for x in range(len(alibi_feature_names)) if x not in list(alibi_category_map.keys())]
ordinal_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),
                                      ('scaler', StandardScaler())])

categorical_features = list(alibi_category_map.keys())
categorical_transformer = Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),
                                          ('onehot', OneHotEncoder(handle_unknown='ignore'))])

preprocessor = ColumnTransformer(transformers=[('num', ordinal_transformer, ordinal_features),
                                               ('cat', categorical_transformer, categorical_features)])
preprocessor.fit(alibi_data)
from sklearn.ensemble import RandomForestClassifier

np.random.seed(0)
clf = RandomForestClassifier(n_estimators=50)
clf.fit(preprocessor.transform(X_train_alibi), y_train_alibi)
!mkdir -p pipeline/pipeline_steps/loanclassifier/<jupyter_output><empty_output><jupyter_text>Save the model artefacts so we can deploy them<jupyter_code>import dill

with open("pipeline/pipeline_steps/loanclassifier/preprocessor.dill", "wb") as prep_f:
    dill.dump(preprocessor, prep_f)
    
with open("pipeline/pipeline_steps/loanclassifier/model.dill", "wb") as model_f:
    dill.dump(clf, model_f)<jupyter_output><empty_output><jupyter_text>Build a Model wrapper that uses the trained models through a predict function<jupyter_code>%%writefile pipeline/pipeline_steps/loanclassifier/Model.py
import dill

class Model:
    def __init__(self, *args, **kwargs):
        
        with open("preprocessor.dill", "rb") as prep_f:
            self.preprocessor = dill.load(prep_f)
        with open("model.dill", "rb") as model_f:
            self.clf = dill.load(model_f)
        
    def predict(self, X, feature_names=[]):
        X_prep = self.preprocessor.transform(X)
        proba = self.clf.predict_proba(X_prep)
        return proba<jupyter_output><empty_output><jupyter_text>Add the dependencies for the wrapper to work<jupyter_code>%%writefile pipeline/pipeline_steps/loanclassifier/requirements.txt
scikit-learn==0.20.1
dill==0.2.9
scikit-image==0.15.0
scikit-learn==0.20.1
scipy==1.1.0
numpy==1.15.4
!mkdir pipeline/pipeline_steps/loanclassifier/.s2i
%%writefile pipeline/pipeline_steps/loanclassifier/.s2i/environment
MODEL_NAME=Model
API_TYPE=REST
SERVICE_TYPE=MODEL
PERSISTENCE=0<jupyter_output><empty_output><jupyter_text>Use the source2image command to containerize code<jupyter_code>!s2i build pipeline/pipeline_steps/loanclassifier seldonio/seldon-core-s2i-python3:0.8 loanclassifier:0.1<jupyter_output><empty_output><jupyter_text>Define the graph of your pipeline with individual models<jupyter_code>%%writefile pipeline/pipeline_steps/loanclassifier/loanclassifiermodel.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: loanclassifier
spec:
  name: loanclassifier
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: loanclassifier:0.1
          name: model
    graph:
      children: []
      name: model
      type: MODEL
      endpoint:
        type: REST
    name: loanclassifier
    replicas: 1<jupyter_output><empty_output><jupyter_text>Deploy your model!<jupyter_code>!kubectl apply -f pipeline/pipeline_steps/loanclassifier/loanclassifiermodel.yaml<jupyter_output><empty_output><jupyter_text>Now we can send data through the REST API<jupyter_code>X_test_alibi[:1]
%%bash
curl -X POST -H 'Content-Type: application/json' \
    -d "{'data': {'names': ['text'], 'ndarray': [[52,  4,  0,  2,  8,  4,  2,  0,  0,  0, 60,  9]]}}" \
    http://localhost:80/seldon/default/loanclassifier/api/v0.1/predictions<jupyter_output>{
  "meta": {
    "puid": "96cmdkc4k1c6oassvpnpasqbgf",
    "tags": {
    },
    "routing": {
    },
    "requestPath": {
      "model": "loanclassifier:0.1"
    },
    "metrics": []
  },
  "data": {
    "names": ["t:0", "t:1"],
    "ndarray": [[0.86, 0.14]]
  }
}<jupyter_text>We can also reach it with the Python Client<jupyter_code>from seldon_core.seldon_client import SeldonClient

batch = X_test_alibi[:1]

sc = SeldonClient(
    gateway="ambassador", 
    gateway_endpoint="localhost:80",
    deployment_name="loanclassifier",
    payload_type="ndarray",
    namespace="default",
    transport="rest")

client_prediction = sc.predict(data=batch)

print(client_prediction.response)<jupyter_output>meta {
  puid: "hv4dnmr8m3ckgrhtnc48rs7mjg"
  requestPath {
    key: "model"
    value: "loanclassifier:0.1"
  }
}
data {
  names: "t:0"
  names: "t:1"
  ndarray {
    values {
      list_value {
        values {
          number_value: 0.86
        }
        values {
          number_value: 0.14
        }
      }
    }
  }
}<jupyter_text>Now we can create an explainer for our model<jupyter_code>from alibi.explainers import AnchorTabular

predict_fn = lambda x: clf.predict(preprocessor.transform(x))
explainer = AnchorTabular(predict_fn, alibi_feature_names, categorical_names=alibi_category_map)
explainer.fit(X_train_alibi, disc_perc=[25, 50, 75])

explanation = explainer.explain(X_test_alibi[0], threshold=0.95)

print('Anchor: %s' % (' AND '.join(explanation['names'])))
print('Precision: %.2f' % explanation['precision'])
print('Coverage: %.2f' % explanation['coverage'])

def predict_remote_fn(X):
    from seldon_core.seldon_client import SeldonClient
    from seldon_core.utils import get_data_from_proto
    
    kwargs = {
        "gateway": "ambassador", 
        "deployment_name": "loanclassifier",
        "payload_type": "ndarray",
        "namespace": "default",
        "transport": "rest"
    }
    
    try:
        kwargs["gateway_endpoint"] = "localhost:80"
        sc = SeldonClient(**kwargs)
        prediction = sc.predict(data=X)
    except:
        # If we are inside the container, we need to reach the ambassador service directly
        kwargs["gateway_endpoint"] = "ambassador:80"
        sc = SeldonClient(**kwargs)
        prediction = sc.predict(data=X)
    
    y = get_data_from_proto(prediction.response)
    return y<jupyter_output><empty_output><jupyter_text>But now we can use the remote model we have in production<jupyter_code># Summary of the predict_remote_fn
def predict_remote_fn(X):
    ....
    sc = SeldonClient(...)
    prediction = sc.predict(data=X)
    y = get_data_from_proto(prediction.response)
    return y<jupyter_output><empty_output><jupyter_text>And train our explainer to use the remote function<jupyter_code>from seldon_core.utils import get_data_from_proto

explainer = AnchorTabular(predict_remote_fn, alibi_feature_names, categorical_names=alibi_category_map)
explainer.fit(X_train_alibi, disc_perc=[25, 50, 75])

explanation = explainer.explain(X_test_alibi[idx], threshold=0.95)

print('Anchor: %s' % (' AND '.join(explanation['names'])))
print('Precision: %.2f' % explanation['precision'])
print('Coverage: %.2f' % explanation['coverage'])<jupyter_output>Anchor: Marital Status = Separated AND Sex = Female
Precision: 0.97
Coverage: 0.11<jupyter_text>To containerise our explainer, save the trained binary<jupyter_code>import dill

with open("pipeline/pipeline_steps/loanclassifier-explainer/explainer.dill", "wb") as x_f:
    dill.dump(explainer, x_f)<jupyter_output><empty_output><jupyter_text>Expose it through a wrapper<jupyter_code>%%writefile pipeline/pipeline_steps/loanclassifier-explainer/Explainer.py
import dill
import json
import numpy as np

class Explainer:
    def __init__(self, *args, **kwargs):
        
        with open("explainer.dill", "rb") as x_f:
            self.explainer = dill.load(x_f)
        
    def predict(self, X, feature_names=[]):
        print("Received: " + str(X))
        explanation = self.explainer.explain(X)
        print("Predicted: " + str(explanation))
        return json.dumps(explanation, cls=NumpyEncoder)

    
    
    
class NumpyEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, (
        np.int_, np.intc, np.intp, np.int8, np.int16, np.int32, np.int64, np.uint8, np.uint16, np.uint32, np.uint64)):
            return int(obj)
        elif isinstance(obj, (np.float_, np.float16, np.float32, np.float64)):
            return float(obj)
        elif isinstance(obj, (np.ndarray,)):
            return obj.tolist()
        return json.JSONEncoder.default(self, obj)<jupyter_output><empty_output><jupyter_text>Add config files to build image with script<jupyter_code>!s2i build pipeline/pipeline_steps/loanclassifier-explainer seldonio/seldon-core-s2i-python3:0.8 loanclassifier-explainer:0.1
!mkdir -p pipeline/pipeline_steps/loanclassifier-explainer
%%writefile pipeline/pipeline_steps/loanclassifier-explainer/loanclassifiermodel-explainer.yaml
apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
  labels:
    app: seldon
  name: loanclassifier-explainer
spec:
  name: loanclassifier-explainer
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - image: loanclassifier-explainer:0.1
          name: model-explainer
    graph:
      children: []
      name: model-explainer
      type: MODEL
      endpoint:
        type: REST
    name: loanclassifier-explainer
    replicas: 1<jupyter_output><empty_output><jupyter_text>Deploy your remote explainer<jupyter_code>!kubectl apply -f pipeline/pipeline_steps/loanclassifier-explainer/loanclassifiermodel-explainer.yaml<jupyter_output><empty_output><jupyter_text>Now we can request explanations throught the REST API<jupyter_code>%%bash
curl -X POST -H 'Content-Type: application/json' \
    -d "{'data': {'names': ['text'], 'ndarray': [[52,  4,  0,  2,  8,  4,  2,  0,  0,  0, 60, 9]] }}" \
    http://localhost:80/seldon/default/loanclassifier-explainer/api/v0.1/predictions<jupyter_output>{
  "meta": {
    "puid": "ohbll5bcpu9gg7jjj1unll4155",
    "tags": {
    },
    "routing": {
    },
    "requestPath": {
      "model-explainer": "loanclassifier-explainer:0.1"
    },
    "metrics": []
  },
  "strData": "{\"names\": [\"Marital Status = Separated\", \"Sex = Female\"], \"precision\": 0.9629629629629629, \"coverage\": 0.1078, \"raw\": {\"feature\": [3, 7], \"mean\": [0.9002808988764045, 0.9629629629629629], \"precision\": [0.9002808988764045, 0.9629629629629629], \"coverage\": [0.1821, 0.1078], \"examples\": [{\"covered\": [[46, 4, 4, 2, 2, 1, 4, 1, 0, 0, 45, 9], [24, 4, 1, 2, 6, 3, 2, 1, 0, 0, 40, 9], [39, 4, 4, 2, 4, 1, 4, 1, 4650, 0, 44, 9], [40, 4, 0, 2, 5, 4, 4, 0, 0, 0, 32, 9], [39, 4, 1, 2, 8, 0, 4, 1, 3103, 0, 50, 9], [45, 4, 1, 2, 6, 5, 4, 0, 0, 0, 42, 9], [41, 4, 1, 2, 5, 1, 4, 1, 0, 0, 40, 9], [40, 4, 4, 2, 2, 0, 4, 1, 0, 0, 40, 9], [58, 4, 3, 2, 2, 2, 4, 0, 0, 0, 45, 5], [23, 4, 1, 2, 5, 1, 4, 1, 0, 0, 50, 9]], \"covered_true\": [[33, 4, 4, 2, 2, 0, 4, 1, 0, [...]
{'fraction_non_alphanumeric': 0.11535347509268334, 'fraction_numerical': 0.02752801806792517, 'mean_word_length': 4.8262164846077455, 'pattern_counts': {'":': 34, '<': 113, '<?xml version=': 0, '>': 112, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '30', 'dir': 'jupyter-structured-clean-dedup', 'id': '299053', 'license': 'MIT', 'n_tokens_mistral': 8220, 'n_tokens_neox': 7599, 'n_words': 2051, 'path': 'Bias Evaluation & Explainability.ipynb', 'repo_name': 'NunoEdgarGFlowHub/explainability-and-bias', 'size': '23478'}