<commit_before># Kafka Connect Mongodb
The connector is used to load data both from Kafka to Mongodb
and from Mongodb to Kafka.

# Building
You can build the connector with Maven using the standard lifecycle phases:
```
mvn clean
mvn package
```

# Sample Configuration
## Source Connector
```ini
name=mongodb-source-connector
connector.class=org.apache.kafka.connect.mongodb.MongodbSourceConnector
tasks.max=1
host=127.0.0.1
port=27017
batch.size=100
schema.name=mongodbschema
topic.prefix=optionalprefix
databases=mydb.test1,mydb.test2,mydb.test3
```

## Sink Connector<commit_msg>something on the readme<commit_after># Kafka Connect Mongodb
The connector is used to load data both from Kafka to Mongodb
and from Mongodb to Kafka.

# Building
You can build the connector with Maven using the standard lifecycle phases:
```
mvn clean
mvn package
```

# Sample Configuration
## Source Connector
When the connector is run as a Source Connector, it reads data from [Mongodb oplog](https://docs.mongodb.org/manual/core/replica-set-oplog/)
and publishes it on Kafka.
3 different types of messages are read from the oplog:
* Insert
* Update
* Delete
For every message, a SourceRecord is created, having the following schema:
```json
{
  "type": "record",
  "name": "schemaname",
  "fields": [
    {
      "name": "timestamp",
      "type": [
        "null",
        "int"
      ]
    },
    {
      "name": "order",
      "type": [
        "null",
        "int"
      ]
    },
    {
      "name": "operation",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "database",
      "type": [
        "null",
        "string"
      ]
    },
    {
      "name": "object",
      "type": [
        "null",
        "string"
      ]
    }
  ],
  "connect.name": "stillmongotesting"
}
```
* **timestamp**: timestamp in seconds when the event happened
* **order**: order of the event between events with the same timestamp
* **operation**: type of operation the message represent. i: insert, u: update, d: delete
* **database**: database in which the operation took place
* **object**: inserted/updated/deleted object

```ini
name=mongodb-source-connector
connector.class=org.apache.kafka.connect.mongodb.MongodbSourceConnector
tasks.max=1
host=127.0.0.1
port=27017
batch.size=100
schema.name=mongodbschema
topic.prefix=optionalprefix
databases=mydb.test1,mydb.test2,mydb.test3
```

## Sink Connector
{'fraction_non_alphanumeric': 0.12163839470417874, 'fraction_numerical': 0.015308233347124535, 'mean_word_length': 3.341113105924596, 'pattern_counts': {'":': 14, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '580111', 'n_tokens_mistral': 807, 'n_tokens_neox': 761, 'n_words': 230}