<commit_before># Getting Started

## From source:

### Pre-Requisites
- Python 3.6+
- Must have already run `./gradlew build` in the datahub root directory.
- On MacOS: `brew install librdkafka`
- On Debian/Ubuntu: `sudo apt install librdkafka-dev python3-dev python3-venv`

### Set up python environment
```sh
python3 -m venv venv
source venv/bin/activate
pip install -e .
./scripts/codegen.sh
```

### Usage - from source
```sh
gometa-ingest -c examples/recipes/file_to_file.yml
```

## Using Docker:

### Build the image
```sh
source docker/docker_build.sh
```

### Usage - with Docker
We have a simple script provided that supports mounting a local directory for input recipes and an output directory for output data:
```sh
source docker/docker_run.sh examples/recipes/file_to_file.yml
```

# Recipes

A recipe is a configuration that tells our ingestion scripts where to pull data from (source) and where to put it (sink).
Here's a simple example that pulls metadata from MSSQL and puts it into datahub.<commit_msg>Modifying README to bring in old content<commit_after># Metadata Ingestion

This module hosts an extensible Python-based metadata ingestion system for DataHub.
This supports sending data to DataHub using Kafka or through the REST api.
The driver script is called gometa-ingest.
Read on to figure out how to use it!


## Pre-Requisites

Before running any metadata ingestion job, you should make sure that DataHub backend services are all running. If you are trying this out locally, the easiest way to do that is through Docker images.
You can run these jobs by building from source or by running docker images.
## Building from source:

### Pre-Requisites
1. Python 3.6+ must be installed in your host environment.
2. You also need to build the `mxe-schemas` module as below.
   ```
   ./gradlew :metadata-events:mxe-schemas:build
   ```
   This is needed to generate `MetadataChangeEvent.avsc` which is the schema for the `MetadataChangeEvent_v4` Kafka topic.
3. On MacOS: `brew install librdkafka`
4. On Debian/Ubuntu: `sudo apt install librdkafka-dev python3-dev python3-venv`

### Set up your Python environment
```sh
python3 -m venv venv
source venv/bin/activate
pip install -e .
./scripts/codegen.sh
```

### Usage - from source
```sh
gometa-ingest -c examples/recipes/file_to_file.yml
```

## Running Ingestion using Docker:

### Build the image
```sh
source docker/docker_build.sh
```

### Usage - with Docker
We have a simple script provided that supports mounting a local directory for input recipes and an output directory for output data:
```sh
source docker/docker_run.sh examples/recipes/file_to_file.yml
```

# Recipes

A recipe is a configuration that tells our ingestion scripts where to pull data from (source) and where to put it (sink).
Here's a simple example that pulls metadata from MSSQL and puts it into datahub.
{'fraction_non_alphanumeric': 0.08362491252624213, 'fraction_numerical': 0.005248425472358292, 'mean_word_length': 4.284658040665434, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1978383', 'n_tokens_mistral': 883, 'n_tokens_neox': 808, 'n_words': 380}