<commit_before>== Principles

=== Based on reactive streams

`Reactivity` architecture allows the system to face an unpredictible load of data when the users, the data sources or even the resources of some parties inside the information system scale.
This is why all the core http://www.reactive-streams.org[architecture] heavily uses the reactive streams where each service acts as a consumer and/or as a producer, emphasing an event-driven model in all layers.

=== Event-driven system

The event driven system is ready to route messages from source to sink in both live and batch contexts.
This need comes from the necessity to cover messaging based communication between the users and also asynchronous streams processing to prepare the data for some views.
Since everything is message, it's mandatory to expect that a message is going to be consumed by different consumers for different purpose.
This case is typically well illustrating by `RabbitMQ` documentation when talking about `direct exchange routing`.

image:https://www.rabbitmq.com/img/tutorials/intro/exchange-direct.png[]

We can see that there is as many consumer with different purpose as many routing keys are declared.
But it's important to notice that two consumers for the same purpose can be registered, and only one of them will consume the message.

=== Stream processing capabilities

`Reactivity` architecture expects a various number of streams and transformations of those streams.
The transformation phase requires messages to be delivered in the same order they are publishing.
This means also that if a consumer fails to acknowledge a message consumption, it must be able to easily retry to process the same message before treating a new one.

Transformation can be also aggregations to satisfy some user views requirements.
This means that a stream can be composed of different streams.

There is no reason to exclude some system interractions from the asynchronous messaging model.
Even a click from a user can be considered as a stream source.

image:https://camo.githubusercontent.com/36c0a9ffd8ed22236bd6237d44a1d3eecbaec336/687474703a2f2f692e696d6775722e636f6d2f634c344d4f73532e706e67[https://github.com/staltz]

This consideration can be wrong in a few cases where technical constraints make more realistic a different technique.   

=== Streaming platform

Routing messages with guarantee of delivery, ordering preservation, acknowledgement mechanism and high performance must be addressed by a distributed streaming platform.
That streaming platform will allow all producers and consumers to register themselves.
This is that platform that will receive all messages from producers and push them to consumers.

image:streaming-platform.png[]<commit_msg>Add the microservice architecture style as principle.<commit_after>== Principles

=== Based on reactive streams

`Reactivity` architecture allows the system to face an unpredictible load of data when the users, the data sources or even the resources of some parties inside the information system scale.
This is why all the core http://www.reactive-streams.org[architecture] heavily uses the reactive streams where each service acts as a consumer and/or as a producer, emphasing an event-driven model in all layers.

=== Event-driven system

The event driven system is ready to route messages from source to sink in both live and batch contexts.
This need comes from the necessity to cover messaging based communication between the users and also asynchronous streams processing to prepare the data for some views.
Since everything is message, it's mandatory to expect that a message is going to be consumed by different consumers for different purpose.
This case is typically well illustrating by `RabbitMQ` documentation when talking about `direct exchange routing`.

image:https://www.rabbitmq.com/img/tutorials/intro/exchange-direct.png[]

We can see that there is as many consumer with different purpose as many routing keys are declared.
But it's important to notice that two consumers for the same purpose can be registered, and only one of them will consume the message.

=== Stream processing capabilities

`Reactivity` architecture expects a various number of streams and transformations of those streams.
The transformation phase requires messages to be delivered in the same order they are publishing.
This means also that if a consumer fails to acknowledge a message consumption, it must be able to easily retry to process the same message before treating a new one.

Transformation can be also aggregations to satisfy some user views requirements.
This means that a stream can be composed of different streams.

There is no reason to exclude some system interractions from the asynchronous messaging model.
Even a click from a user can be considered as a stream source.

image:https://camo.githubusercontent.com/36c0a9ffd8ed22236bd6237d44a1d3eecbaec336/687474703a2f2f692e696d6775722e636f6d2f634c344d4f73532e706e67[https://github.com/staltz]

This consideration can be wrong in a few cases where technical constraints make more realistic a different technique.   

=== Streaming platform

Routing messages with guarantee of delivery, ordering preservation, acknowledgement mechanism and high performance must be addressed by a distributed streaming platform.
That streaming platform will allow all producers and consumers to register themselves.
This is that platform that will receive all messages from producers and push them to consumers.

image:streaming-platform.png[]

=== Microservice architecture

The streaming platform will interact with producers and consumers in respect of http://www.martinfowler.com/articles/microservices.html[microservice architecture definition].
This mainly means that a consumer or a producer is considered as a component serving one purpose, exposing a clear API and easy to replace in the information system.

A microservice is autonomous as much as possible which is possible when control of data is decentralized.
Consumers and producers can take advantage of this by embedding their own database, which can lead to a duplication of data.
A consistency check mechanism should be considered to regularly make sure that a database is not incorrectly synchronized with others.
In case of unexpected desynchronization (typically a state that is expected to be synchronized regarding the messages that are delivered and those who are not), a re-initialization of that data should be possible.
Message streaming should be used as much as possible to replicate a data from scratch, including transformations process.
{'fraction_non_alphanumeric': 0.03390856796851347, 'fraction_numerical': 0.020284589766878594, 'mean_word_length': 5.174766355140187, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '574168', 'n_tokens_mistral': 1570, 'n_tokens_neox': 1430, 'n_words': 908}