<commit_before>[![ci](https://github.com/google/exposure-notifications-private-analytics-ingestion/actions/workflows/ci.yaml/badge.svg)](https://github.com/google/exposure-notifications-private-analytics-ingestion/actions/workflows/ci.yaml)

# Exposure Notification Private Analytics Ingestion

This repository contains implementations for [Apache Beam](https://beam.apache.org/)
batch pipelines to process private data shares stored in Firestore according
to the Exposure Notification Private Analytics protocol. It assumes private data
shares uploaded as Firestore documents, as done in the
[Exposure Notification Express template app](https://github.com/google/exposure-notifications-android/blob/4b7b461282b2ede6fb2a93488c6d628440052c8d/app/src/main/java/com/google/android/apps/exposurenotification/privateanalytics/PrivateAnalyticsFirestoreRepository.java#L42).
These documents contain encrypted packets using the [Prio](https://crypto.stanford.edu/prio/)
protocol. The pipeline implementation converts them into the format that
downstream Prio data processing servers expect, defined in the Avro schema
[here](https://github.com/abetterinternet/prio-server/tree/master/avro-schema).

This implementation make use of Firestore as a scalable NoSQL db for subsequent
batching and aggregation. Alternative implementations might operate a custom
backend endpoint to accumulate the packets, or use a pubsub mechanism. Since the
packets are encrypted on device, the channel over which the packets travel need
not be trusted.

This repository also contains the Firebase configuration to secure Firestore
with [security rules](config/firebase/firestore.rules) as well as
[Terraform scripts](terraform/main.tf) to bring up required infrastructure.

## Setup

### Multiple Maven modules

The project is structured into multiple maven modules to allow incorporation of
outside implementations of attestation. Implementations need only depend on the
DataShare model module, and a profile can be added to get it included in the
pipeline module build. The pipeline pulls available implementations dynamically.

Since there aren't too many individual classes that make up each module, and
since they are only meant to be packaged and executed together, we use a single
source tree for all modules.

Follow the
[Getting started with Google Cloud Dataflow](https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/dataflow/README.md)
page. You will need the following:

1. Set up a
    [Google Cloud project](https://console.cloud.google.com/projectcreate) or use an existing one.
    Then [import the Google Cloud project into Firebase](https://cloud.google.com/firestore/docs/client/get-firebase).

1. [Enable APIs](https://console.cloud.google.com/flows/enableapi?apiid=containerregistry.googleapis.com,cloudbuild.googleapis.com):
    Container Registry, Cloud Build, Cloud Datastore and Dataflow.

1. [Create an asymmetric key](https://cloud.google.com/kms/docs/creating-asymmetric-keys)

1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets)
    for your outputs.

1. Create a service account with permissions for [Firestore](https://cloud.google.com/datastore/docs/access/iam#iam_roles),
    [reading the KMS key](https://cloud.google.com/kms/docs/reference/permissions-and-roles),
    [Dataflow](https://cloud.google.com/dataflow/docs/concepts/access-control#roles),
    and [Cloud Storage](https://cloud.google.com/storage/docs/access-control/iam).

1. Download the a key for your service account and store as `credentials.json`.
    Keep those credentials save!

### Useful Environment Variables
<commit_msg>docs: update readme to mention the need to update submodules<commit_after>[![ci](https://github.com/google/exposure-notifications-private-analytics-ingestion/actions/workflows/ci.yaml/badge.svg)](https://github.com/google/exposure-notifications-private-analytics-ingestion/actions/workflows/ci.yaml)

# Exposure Notification Private Analytics Ingestion

This repository contains implementations for [Apache Beam](https://beam.apache.org/)
batch pipelines to process private data shares stored in Firestore according
to the Exposure Notification Private Analytics protocol. It assumes private data
shares uploaded as Firestore documents, as done in the
[Exposure Notification Express template app](https://github.com/google/exposure-notifications-android/blob/4b7b461282b2ede6fb2a93488c6d628440052c8d/app/src/main/java/com/google/android/apps/exposurenotification/privateanalytics/PrivateAnalyticsFirestoreRepository.java#L42).
These documents contain encrypted packets using the [Prio](https://crypto.stanford.edu/prio/)
protocol. The pipeline implementation converts them into the format that
downstream Prio data processing servers expect, defined in the Avro schema
[here](https://github.com/abetterinternet/prio-server/tree/master/avro-schema).

This implementation make use of Firestore as a scalable NoSQL db for subsequent
batching and aggregation. Alternative implementations might operate a custom
backend endpoint to accumulate the packets, or use a pubsub mechanism. Since the
packets are encrypted on device, the channel over which the packets travel need
not be trusted.

This repository also contains the Firebase configuration to secure Firestore
with [security rules](config/firebase/firestore.rules) as well as
[Terraform scripts](terraform/main.tf) to bring up required infrastructure.

## Setup

### Multiple Maven modules

The project is structured into multiple maven modules to allow incorporation of
outside implementations of attestation. Implementations need only depend on the
DataShare model module, and a profile can be added to get it included in the
pipeline module build. The pipeline pulls available implementations dynamically.

Since there aren't too many individual classes that make up each module, and
since they are only meant to be packaged and executed together, we use a single
source tree for all modules.

Follow the
[Getting started with Google Cloud Dataflow](https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/dataflow/README.md)
page. You will need the following:

1. Set up a
    [Google Cloud project](https://console.cloud.google.com/projectcreate) or use an existing one.
    Then [import the Google Cloud project into Firebase](https://cloud.google.com/firestore/docs/client/get-firebase).

1. [Enable APIs](https://console.cloud.google.com/flows/enableapi?apiid=containerregistry.googleapis.com,cloudbuild.googleapis.com):
    Container Registry, Cloud Build, Cloud Datastore and Dataflow.

1. [Create an asymmetric signing key](https://cloud.google.com/kms/docs/creating-asymmetric-keys#create_an_asymmetric_signing_key)

1. [Create a storage bucket](https://cloud.google.com/storage/docs/creating-buckets)
    for your outputs.

1. Create a service account with permissions for [Firestore](https://cloud.google.com/datastore/docs/access/iam#iam_roles),
    [reading the KMS key](https://cloud.google.com/kms/docs/reference/permissions-and-roles),
    [Dataflow](https://cloud.google.com/dataflow/docs/concepts/access-control#roles),
    and [Cloud Storage](https://cloud.google.com/storage/docs/access-control/iam).

1. Download the a key for your service account and store as `credentials.json`.
    Keep those credentials save!
                  
### Clone Submodules
A submodule is need to access the avro definitions of the Prio classes.

```sh
git submodule update --init
```

### Useful Environment Variables

{'fraction_non_alphanumeric': 0.08597585246119145, 'fraction_numerical': 0.009287514926363274, 'mean_word_length': 5.250414593698176, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 32, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '691211', 'n_tokens_mistral': 2095, 'n_tokens_neox': 1972, 'n_words': 692}