<commit_before># Four Keys Terraform

This directory contains modules and examples for deploying Four Keys with Terraform. The primary module `modules/fourkeys` uses the other sub-modules to deploy resources to a provided Google Cloud Project.  

## Usage
See `examples` directory for examples. Simple usage:

```hcl
module "fourkeys" {
  source    = "github.com/GoogleCloudPlatform/fourkeys//terraform/modules/fourkeys"
  project_id = "your-google-cloud-project-id"
  parsers   = ['github']
}
```
The example above will deploy Four Keys with a Github parser for Github events.

## Deploying with Terraform


## Generating mock data
<commit_msg>docs: deploying with tf and mock data<commit_after># Four Keys Terraform

This directory contains modules and examples for deploying Four Keys with Terraform. The primary module `modules/fourkeys` uses the other sub-modules to deploy resources to a provided Google Cloud Project.  

## Usage
Simple usage:

```hcl
module "fourkeys" {
  source    = "github.com/GoogleCloudPlatform/fourkeys//terraform/modules/fourkeys"
  project_id = "your-google-cloud-project-id"
  parsers   = ['github']
}
```
The example above will deploy Four Keys with a Github parser for Github events. See the `terraform/example` directory for full example and options. 

## Deploying with Terraform

To deploy Four Keys with Terraform, you will first need:
* A Google Cloud project
* Terraform installed on your local machine. We recommend deploying from [Cloud Shell](https://shell.cloud.google.com/?show=ide%2Cterminal) on your Google Cloud project.

1. Clone the fourkeys git repository, or copy the files in the `terraform/example` directory to your working directory

1. Rename `terraform.tfvars.example` to `terraform.tfvars`

1. Edit `terraform.tfvars` with values for the required variables. See `variables.tf` for a list of the variables, along with their descriptions and default values. To accept the default value of a variable indicated in `variables.tf`, exclude it from `terraform.tfvars`

1. Run the following commands from the `example` directory, or your working directory:

    `terraform init` to inialize Terraform and download the module

    `terraform plan` to preview changes.

    `terraform apply` to deploy the resources.

## Generating mock data

To test your Four Keys deployment, you can generate mock data that simulates events from a Github repository.  

1. Export your event handler URL an environment variable. This the webhook URL that will receive events:

    ```sh
    export WEBHOOK=`gcloud run services list | grep event-handler | awk '{print $4}'`
    ```

1. Export your event handler secret to an environment variable. This is the secret used to authenticate events sent to the webhook:

    ```sh
    export SECRET=`gcloud secrets versions access 1 --secret=event-handler`
    ```

1. From the root of the fourkeys project run:

    ```sh
    python3 data_generator/generate_data.py --vc_system=github
    ``` 
    You can see these events being run through the pipeline:
    *  The event handler logs show successful requests
    *  The Pub/Sub topic show messages posted
    *  The BigQuery GitHub parser show successful requests

1. View the generated data in the `events_raw` table in with bq: 

    ```sh
    bq query 'SELECT * FROM four_keys.events_raw WHERE source = "githubmock";'
    ```
    Or query the table directly in [BigQuery](https://console.cloud.google.com/bigquery):
    ```sql
    SELECT * FROM four_keys.events_raw WHERE source = 'githubmock';
    ```
{'fraction_non_alphanumeric': 0.07603165630299605, 'fraction_numerical': 0.0033917467495760316, 'mean_word_length': 4.048502139800285, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1160386', 'n_tokens_mistral': 979, 'n_tokens_neox': 945, 'n_words': 430}