<commit_before># ML Metadata

[ML Metadata (MLMD)](https://github.com/google/ml-metadata) is a library for
recording and retrieving metadata associated with ML developer and data
scientist workflows. MLMD is an integral part of
[TensorFlow Extended (TFX)](https://www.tensorflow.org/tfx), but is designed so
that it can be used independently.

Every run of a production ML pipeline generates metadata containing information
about the various pipeline components, their executions (e.g. training runs),
and resulting artifacts(e.g. trained models). In the event of unexpected
pipeline behavior or errors, this metadata can be leveraged to analyze the
lineage of pipeline components and debug issues.Think of this metadata as the
equivalent of logging in software development.

MLMD helps you understand and analyze all the interconnected parts of your ML
pipeline instead of analyzing them in isolation and can help you answer
questions about your ML pipeline such as:

*   Which dataset did the model train on?
*   What were the hyperparameters used to train the model?
*   Which pipeline run created the model?
*   Which training run led to this model?
*   Which version of TensorFlow created this model?
*   When was the failed model pushed?

## Metadata store

MLMD registers the following types of metadata in a database called the
**Metadata Store**.

1.  Metadata about the artifacts generated through the components/steps of your
    ML pipelines
1.  Metadata about the executions of these components/steps
1.  Metadata about pipelines and associated lineage information

The Metadata Store provides APIs to record and retrieve metadata to and from the
storage backend. The storage backend is pluggable and can be extended. MLMD
provides reference implementations for SQLite (which supports in-memory and
disk) and MySQL out of the box.

This graphic shows a high-level overview of the various components that are part
of MLMD.

![ML Metadata Overview](images/mlmd_overview.png)

### Metadata storage backends and store connection configuration

The `MetadataStore` object receives a connection configuration that corresponds
to the storage backend used.

*   **Fake Database** provides an in-memory DB (using SQLite) for fast
    experimentation and local runs. The database is deleted when the store
    object is destroyed.

```python
import ml_metadata as mlmd
from ml_metadata.proto import metadata_store_pb2

connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.fake_database.SetInParent() # Sets an empty fake database proto.
store = metadata_store.MetadataStore(connection_config)
```

*   **SQLite** reads and writes files from disk.

```python
connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.sqlite.filename_uri = '...'
connection_config.sqlite.connection_mode = 3 # READWRITE_OPENCREATE
store = metadata_store.MetadataStore(connection_config)
```

*   **MySQL** connects to a MySQL server.

```python
connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.mysql.host = '...'
connection_config.mysql.port = '...'
connection_config.mysql.database = '...'
connection_config.mysql.user = '...'
connection_config.mysql.password = '...'
store = metadata_store.MetadataStore(connection_config)
```
<commit_msg>Add missing import statement in Guide<commit_after># ML Metadata

[ML Metadata (MLMD)](https://github.com/google/ml-metadata) is a library for
recording and retrieving metadata associated with ML developer and data
scientist workflows. MLMD is an integral part of
[TensorFlow Extended (TFX)](https://www.tensorflow.org/tfx), but is designed so
that it can be used independently.

Every run of a production ML pipeline generates metadata containing information
about the various pipeline components, their executions (e.g. training runs),
and resulting artifacts(e.g. trained models). In the event of unexpected
pipeline behavior or errors, this metadata can be leveraged to analyze the
lineage of pipeline components and debug issues.Think of this metadata as the
equivalent of logging in software development.

MLMD helps you understand and analyze all the interconnected parts of your ML
pipeline instead of analyzing them in isolation and can help you answer
questions about your ML pipeline such as:

*   Which dataset did the model train on?
*   What were the hyperparameters used to train the model?
*   Which pipeline run created the model?
*   Which training run led to this model?
*   Which version of TensorFlow created this model?
*   When was the failed model pushed?

## Metadata store

MLMD registers the following types of metadata in a database called the
**Metadata Store**.

1.  Metadata about the artifacts generated through the components/steps of your
    ML pipelines
1.  Metadata about the executions of these components/steps
1.  Metadata about pipelines and associated lineage information

The Metadata Store provides APIs to record and retrieve metadata to and from the
storage backend. The storage backend is pluggable and can be extended. MLMD
provides reference implementations for SQLite (which supports in-memory and
disk) and MySQL out of the box.

This graphic shows a high-level overview of the various components that are part
of MLMD.

![ML Metadata Overview](images/mlmd_overview.png)

### Metadata storage backends and store connection configuration

The `MetadataStore` object receives a connection configuration that corresponds
to the storage backend used.

*   **Fake Database** provides an in-memory DB (using SQLite) for fast
    experimentation and local runs. The database is deleted when the store
    object is destroyed.

```python
import ml_metadata as mlmd
from ml_metadata.metadata_store import metadata_store
from ml_metadata.proto import metadata_store_pb2

connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.fake_database.SetInParent() # Sets an empty fake database proto.
store = metadata_store.MetadataStore(connection_config)
```

*   **SQLite** reads and writes files from disk.

```python
connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.sqlite.filename_uri = '...'
connection_config.sqlite.connection_mode = 3 # READWRITE_OPENCREATE
store = metadata_store.MetadataStore(connection_config)
```

*   **MySQL** connects to a MySQL server.

```python
connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.mysql.host = '...'
connection_config.mysql.port = '...'
connection_config.mysql.database = '...'
connection_config.mysql.user = '...'
connection_config.mysql.password = '...'
store = metadata_store.MetadataStore(connection_config)
```

{'fraction_non_alphanumeric': 0.06467587034813925, 'fraction_numerical': 0.0024009603841536613, 'mean_word_length': 5.031674208144796, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6673896', 'n_tokens_mistral': 1691, 'n_tokens_neox': 1630, 'n_words': 765}