<commit_before>[![build status badge](https://img.shields.io/travis/zalando-nakadi/nakadi-producer-spring-boot-starter.svg?label=Travis%20Build)](https://travis-ci.org/zalando-nakadi/nakadi-producer-spring-boot-starter/branches)
[![last Github release badge](https://img.shields.io/github/release/zalando-nakadi/nakadi-producer-spring-boot-starter.svg?label=Last%20Release)](https://github.com/zalando-nakadi/nakadi-producer-spring-boot-starter/releases)
![last maven central release badge](https://img.shields.io/maven-central/v/org.zalando/nakadi-producer-spring-boot-starter.svg)
[![MIT license](https://img.shields.io/github/license/zalando-nakadi/nakadi-producer-spring-boot-starter.svg)](LICENSE)

# nakadi-producer-spring-boot-starter

[Nakadi](https://github.com/zalando/nakadi) is a distributed event bus that implements a RESTful API abstraction instead of Kafka-like queues.

The goal of this Spring Boot starter is to simplify the reliable integration between event producer and Nakadi. When we send events from a transactional application, a few recurring challenges appear:
- we have to make sure that events from a transaction get sent, when the transaction has been committed,
- we have to make sure that events from a transaction do not get sent, when the transaction has been rolled back,
- we have to make sure that events get sent, even if an error occurred while sending the event,
- we want to give the event consumer a way to infer the order in which the events occurred and
- it is very comfortable for initial data loads and error recovery to be able to generate snapshots of the current db state as events.

There are already [multiple clients for the Nakadi REST API](https://zalando.github.io/nakadi/manual.html#using_clients), but none of them solves the mentioned issues. 

We solved them by persisting new events in a log table as part of the producing JDBC transaction. They will then be sent asynchonously to Nakadi after the transaction completed. If the transaction is rolled back, the events will vanish too. As a result, events will always be sent if and only if the transaction succeeded.

The Transmitter generates a strictly monotonically increasing event id that can be used for ordering the events during retrieval. It is not guaranteed, that events will be sent to Nakadi in the order they have been produced. If an event could not be sent to Nakadi, the library will periodically retry the transmission.

This project is mature, used in production in some services at Zalando, and in active development.

Be aware that this library **does neither guarantee that events are sent exactly once, nor that they are sent in the order they have been persisted**. This is not a bug but a design decision that allows us to skip and retry sending events later in case of temporary failures. So make sure that your events are designed to be processed out of order.  To help you in this matter, the library generates a *strictly monotonically increasing event id* (field `metadata/eid` in Nakadi's event object) that can be used to reconstruct the message order.

## Versioning

This library follows the [semantic versioning](http://semver.org/) schema. A major version change means either an
incompatible change in the API, or some incompatible behavioral change (e.g. the database usage), minor versions
mean new features without breaking compatibility, and patch versions are backwards compatible bug fixes.

Please also be aware that, when udating between major releases of this lib, you must not jump over a major
release (1.0 → 3.0). Please always deploy the intermediate major releases at least once – otherwise you might
lose events. You will find migration instructions between major release in
[the release notes](https://github.com/zalando-incubator/nakadi-producer-spring-boot-starter/releases).

You may of course always setup a fresh system with the newest version.


## Prerequisites

This library was tested with Spring Boot 1.5.3.RELEASE and relies on an existing configured PostgreSQL DataSource.

This library also uses:

* flyway-core
* Spring JDBC
* [fahrschein](https://github.com/zalando-nakadi/fahrschein) Nakadi client library
* jackson >= 2.7.0
* (optional) Zalando's [tracer-spring-boot-starter](https://github.com/zalando/tracer)
* (optional) Zalando's [tokens library](https://github.com/zalando/tokens) >= 0.10.0
    * Please note that [tokens-spring-boot-starter](https://github.com/zalando-stups/spring-boot-zalando-stups-tokens) 0.10.0 comes with tokens 0.9.9, which is not enough. You can manually add tokens 0.10.0 with that starter, though.


## Usage

### Setup

If you are using maven, include the library in your `pom.xml`:

```xml
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>nakadi-producer-spring-boot-starter</artifactId>
    <version>${nakadi-producer.version}</version>
</dependency>
```

The latest available version is visible in the Maven central badge at the top of this README. 

Use `@EnableNakadiProducer` annotation to activate spring boot starter auto configuration:

```java
@SpringBootApplication
@EnableNakadiProducer
public class Application {
    public static void main(final String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}
```

The library uses flyway migrations to set up its own database schema `nakadi_events`.

### Nakadi communication configuration

By default, Nakadi-producer-spring-boot starter uses the Fahrschein library to submit its events. It needs some configuration to know how to do this – we support two ways:

* Using existing Fahrschein setup
* Letting this library set things up

#### Using existing Fahrschein setup

If you are already using the [Fahrschein library](https://github.com/zalando-nakadi/fahrschein) directly (e.g. for event consumption) and have already a configured `org.zalando.fahrschein.NakadiClient` object, just make sure it is available as a Spring bean. Nakadi-Producer-Spring-Boot-Starter will pick it up and use it directly.

The configuration in the next section is then not needed at all.

#### Letting this library set things up 

If you want Nakadi-Producer-Spring-Boot-Starter to configure the connection to Nakadi, you'll need to set some properties
(and/or create beans).

You must tell the library, where it can reach your Nakadi instance:
```yaml
nakadi-producer:
  nakadi-base-uri: https://nakadi.example.org
```

Since the communication between your application and Nakadi is secured using OAuth2, you must also provide a OAuth2
token. The easiest way to do so is to include the [Zalando Tokens library](https://github.com/zalando/tokens) into your classpath:

```xml
<dependency>
    <groupId>org.zalando.stups</groupId>
    <artifactId>tokens</artifactId>
    <version>${tokens.version}</version>
</dependency>
```

This starter will detect and auto configure it. To do so, it needs to know the address of your oAuth2 server:
```yaml
nakadi-producer:
  access-token-uri: https://token.auth.example.org/oauth2/access_token
```

Since [July 2017](https://github.com/zalando/nakadi/pull/692), Nakadi (at least in the version operated at Zalando) doesn't require any scopes other than the pseudo-scope `uid` for writing events, [the authorization is instead based on event-type configuration using the service's uid](https://nakadi.io/manual.html#using_authorization).

If your Nakadi installation needs real scopes for submitting events, you can provide them via configuration, too (as a comma-separated list):

```yaml
nakadi-producer:
  access-token-uri: https://token.auth.example.org/oauth2/access_token
  access-token-scopes: my.scope.name,other.scope.name
```

If you do not use the STUPS Tokens library, you can implement token retrieval yourself by defining a Spring bean of type `org.zalando.nakadiproducer.AccessTokenProvider`. The starter will detect it and call it once for each request to retrieve the token. 


### Creating events

The typical use case for this library is to publish events like creating or updating of some objects.

In order to store events you can autowire the [`EventLogWriter`](src/main/java/org/zalando/nakadiproducer/eventlog/EventLogWriter.java) service and use its methods: `fireCreateEvent`, `fireUpdateEvent`, `fireDeleteEvent`, `fireSnapshotEvent` or `fireBusinessEvent`.

You normally don't need to call `fireSnapshotEvent` directly, see below for [snapshot creation](#event-snapshots-optional).


Example of using `fireCreateEvent`:

```java
@Service
public class SomeYourService {

    @Autowire<commit_msg>Documented how to run the library in a k8s environment. Also streamlined the text a bit, since it has become quite long and a bit unstructured already<commit_after>[![build status badge](https://img.shields.io/travis/zalando-nakadi/nakadi-producer-spring-boot-starter.svg?label=Travis%20Build)](https://travis-ci.org/zalando-nakadi/nakadi-producer-spring-boot-starter/branches)
[![last Github release badge](https://img.shields.io/github/release/zalando-nakadi/nakadi-producer-spring-boot-starter.svg?label=Last%20Release)](https://github.com/zalando-nakadi/nakadi-producer-spring-boot-starter/releases)
![last maven central release badge](https://img.shields.io/maven-central/v/org.zalando/nakadi-producer-spring-boot-starter.svg)
[![MIT license](https://img.shields.io/github/license/zalando-nakadi/nakadi-producer-spring-boot-starter.svg)](LICENSE)

# nakadi-producer-spring-boot-starter

[Nakadi](https://github.com/zalando/nakadi) is a distributed event bus that implements a RESTful API abstraction instead of Kafka-like queues.

The goal of this Spring Boot starter is to simplify the reliable integration between event producer and Nakadi. When we send events from a transactional application, a few recurring challenges appear:
- we have to make sure that events from a transaction get sent, when the transaction has been committed,
- we have to make sure that events from a transaction do not get sent, when the transaction has been rolled back,
- we have to make sure that events get sent, even if an error occurred while sending the event,
- we want to give the event consumer a way to infer the order in which the events occurred and
- it is very comfortable for initial data loads and error recovery to be able to generate snapshots of the current db state as events.

There are already [multiple clients for the Nakadi REST API](https://zalando.github.io/nakadi/manual.html#using_clients), but none of them solves the mentioned issues. 

We solved them by persisting new events in a log table as part of the producing JDBC transaction. They will then be sent asynchonously to Nakadi after the transaction completed. If the transaction is rolled back, the events will vanish too. As a result, events will always be sent if and only if the transaction succeeded.

The Transmitter generates a strictly monotonically increasing event id that can be used for ordering the events during retrieval. It is not guaranteed, that events will be sent to Nakadi in the order they have been produced. If an event could not be sent to Nakadi, the library will periodically retry the transmission.

This project is mature, used in production in some services at Zalando, and in active development.

Be aware that this library **does neither guarantee that events are sent exactly once, nor that they are sent in the order they have been persisted**. This is not a bug but a design decision that allows us to skip and retry sending events later in case of temporary failures. So make sure that your events are designed to be processed out of order.  To help you in this matter, the library generates a *strictly monotonically increasing event id* (field `metadata/eid` in Nakadi's event object) that can be used to reconstruct the message order.

## Versioning

This library follows the [semantic versioning](http://semver.org/) schema. A major version change means either an
incompatible change in the API, or some incompatible behavioral change (e.g. the database usage), minor versions
mean new features without breaking compatibility, and patch versions are backwards compatible bug fixes.

Please also be aware that, when udating between major releases of this lib, you must not jump over a major
release (1.0 → 3.0). Please always deploy the intermediate major releases at least once – otherwise you might
lose events. You will find migration instructions between major release in
[the release notes](https://github.com/zalando-incubator/nakadi-producer-spring-boot-starter/releases).

You may of course always setup a fresh system with the newest version.


## Prerequisites

This library was tested with Spring Boot 1.5.3.RELEASE and relies on an existing configured PostgreSQL DataSource.

This library also uses:

* flyway-core
* Spring JDBC
* [fahrschein](https://github.com/zalando-nakadi/fahrschein) Nakadi client library
* jackson >= 2.7.0
* (optional) Zalando's [tracer-spring-boot-starter](https://github.com/zalando/tracer)
* (optional) Zalando's [tokens library](https://github.com/zalando/tokens) >= 0.10.0
    * Please note that [tokens-spring-boot-starter](https://github.com/zalando-stups/spring-boot-zalando-stups-tokens) 0.10.0 comes with tokens 0.9.9, which is not enough. You can manually add tokens 0.10.0 with that starter, though.


## Usage

### Setup

If you are using maven, include the library in your `pom.xml`:

```xml
<dependency>
    <groupId>org.zalando</groupId>
    <artifactId>nakadi-producer-spring-boot-starter</artifactId>
    <version>${nakadi-producer.version}</version>
</dependency>
```

The latest available version is visible in the Maven central badge at the top of this README. 

Use `@EnableNakadiProducer` annotation to activate spring boot starter auto configuration:

```java
@SpringBootApplication
@EnableNakadiProducer
public class Application {
    public static void main(final String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}
```

The library uses flyway migrations to set up its own database schema `nakadi_events`.

### Nakadi communication configuration

By default, Nakadi-producer-spring-boot starter uses the Fahrschein library to submit its events. It needs some configuration to know how to do this – we support two ways:

* Using existing Fahrschein setup
* Letting this library set things up

#### Using existing Fahrschein setup

If you are already using the [Fahrschein library](https://github.com/zalando-nakadi/fahrschein) directly (e.g. for event consumption) and have already a configured `org.zalando.fahrschein.NakadiClient` object, just make sure it is available as a Spring bean. Nakadi-Producer-Spring-Boot-Starter will pick it up and use it directly.

The configuration in the next section is then not needed at all.

#### Letting this library set things up 

You must tell the library, where it can reach your Nakadi instance:
```yaml
nakadi-producer:
  nakadi-base-uri: https://nakadi.example.org
```

Since the communication between your application and Nakadi is secured using OAuth2, you must also provide an OAuth2
token. The easiest way to do so is to include the [Zalando Tokens library](https://github.com/zalando/tokens) into your classpath:

```xml
<dependency>
    <groupId>org.zalando.stups</groupId>
    <artifactId>tokens</artifactId>
    <version>${tokens.version}</version>
</dependency>
```

This starter will detect and auto configure it. To do so, it needs to know the address of your oAuth2 server:
```yaml
nakadi-producer:
  access-token-uri: https://token.auth.example.org/oauth2/access_token
```

If your application is running in Zalandos Kubernetes environment, you also have to configure the credential rotation:
```yaml
apiVersion: "zalando.org/v1"
kind: PlatformCredentialsSet
metadata:
   name: {{{APPLICATION}}}-credentials
spec:
   application: {{{KIO_NAME}}}
   token_version: v2
   tokens:
     nakadi:
       privileges: []
``` 

Since [July 2017](https://github.com/zalando/nakadi/pull/692), Nakadi (at least in the version operated at Zalando) doesn't require any scopes other than the pseudo-scope `uid` for writing events, [the authorization is instead based on event-type configuration using the service's uid](https://nakadi.io/manual.html#using_authorization).

If your Nakadi installation needs real scopes for submitting events, you can provide them via configuration, too (as a comma-separated list):

```yaml
nakadi-producer:
  access-token-uri: https://token.auth.example.org/oauth2/access_token
  access-token-scopes: my.scope.name,other.scope.name
```

If you do not use the STUPS Tokens library, you can implement token retrieval yourself by defining a Spring bean of type `org.zalando.nakadiproducer.AccessTokenProvider`. The starter will detect it and call it once for each request to retrieve the token. 


### Creating events

The typical use case for this library is to publish events like creating or updating of some objects.

In order to store events you can autowire the [`EventLogWriter`](src/main/java/org/zalando/nakadiproducer/eventlog/EventLogWriter.java) service and use its methods: `fireCreateEvent`, `fireUpdateEvent`, `fireDeleteEvent`, `fireSnapshotEvent` or `fireBusinessEvent`.

You normally don't need to call `fireSnapshotEvent` directly, see below for [snapshot creation](#event-snapshots-optional).


Example of using `fireCreateEvent`:

```java
@Service
public class SomeYourService {

    @Autowire
{'fraction_non_alphanumeric': 0.08097421203438396, 'fraction_numerical': 0.004871060171919771, 'mean_word_length': 4.786140583554377, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 39, 'https://': 40, 'lorem ipsum': 0, 'www.': 0, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '547302', 'n_tokens_mistral': 4959, 'n_tokens_neox': 4676, 'n_words': 2061}