<commit_before>Table of Contents

  - [Long Running Actions](#long-running-actions)
    - [Introduction](#introduction)
    - [Motivation](#motivation)
    - [Proposed solution](#proposed-solution)
      - [The Model](#the-model)
      - [CDI Annotations for LRAs](#cdi-annotations-for-lras)
        - [Java Annotations](#java-annotations)
          - [Controlling the lifecycle of an LRA](#controlling-the-lifecycle-of-an-lra)
          - [Compensating Activities](#compensating-activities)
          - [Nesting LRAs](#nesting-lras)
          - [Timing out LRAs and Compensators](#timing-out-lras-and-compensators)
          - [Leaving an LRA](#leaving-an-lra)
          - [Reporting the status of a participant](#reporting-the-status-of-a-participant)
          - [Forgetting an LRA](#forgetting-an-lra)
        - [Injecting an LRA Client API](#injecting-an-lra-client-api)
        - [Java based participant registration API](#java-based-participant-registration-api)
      - [Interoperability with other languages](#interoperability-with-other-languages)
        - [Protocol URLs](#protocol-urls)
          - [LRA Coordinator URL](#lra-coordinator-url)
          - [The LRA URL](#the-lra-url)
          - [Participants](#participants)
  - [Short Running Actions](#short-running-actions)
    - [Architecture](#architecture)
    - [State Transitions](#state-transitions)
    - [Transaction Context Propagation](#transaction-context-propagation)
    - [CDI Annotations for SRAs](#cdi-annotations-for-sras)
      - [SRA Lifecycle Support](#sra-lifecycle-support)
      - [SRA Participant Support](#sra-participant-support)
        - [Rollback](#rollback)
        - [Commit](#commit)
        - [OnePhaseCommit](#onephasecommit)
        - [Prepare](#prepare)
        - [Status](#status)
      - [Timing out SRAs and Participants](#timing-out-sras-and-participants)
        - [TimeLimit](#timelimit)
      - [Java based participant registration API](#java-based-participant-registration-api)
  - [Relationship Between SRAs and LRAs](#relationship-between-sras-and-lras)
  - [Appendix 1](#appendix-1)
    - [Typical Recovery Scenarios](#typical-recovery-scenarios)

# Consistency Models for Microservices
# Long Running Actions

* Proposal: [MP-0009](0009-LRA.md)
* Authors: [<NAME>](https://github.com/mmusgrov/)
* Status: **Sandbox**
* Decision Notes: [Discussion thread topic covering the
Rationale](https://groups.google.com/forum/#!searchin/microprofile/transactions%7Csort:relevance/microprofile/CJirjFkM9Do/TrApz-fBDQAJ)

## Introduction

The proposal introduces APIs for services to coordinate activities.

The main thrust of the proposal introduces an API for loosely coupled
services to coordinate long running activities in such a way as to guarantee
a globally consistent outcome without the need to take locks on data.

A secondary aim of the proposal is to introduce an API for services to
coordinate short running activities in such a way as to guarantee
a globally consistent outcome but this time with the possibility that 
it may be necessary to lock data. The goal of this part of the specification
is to facilitate the developer in writing the individual steps involved in a
long running activity.

The APIs are framed in terms of Java CDI annotations that should be applied
to JAX-RS resources. However, the proposal is also described in terms of
REST resources which means that services written in other languages can
also coordinate with CDI annotated JAX-RS resources. And for services that
do not communicate via HTTP we also define a pure java based registration API.

It is expected that different implementations of this specification will need
to interoperate and implementers should pay special attention
[to the section on interoperability](Interoperability with other languages)
to achieve this goal.

## Motivation

In a loosely coupled service based environment there is sometimes a need for
different services to provide consistency guarantees. Typical examples include:

* order processing involving three services (take order, bill customer,
  ship product). If the shipping service finds that it is out of stock then
  the customer will have been billed with no prospect of receiving his item;
* an airline overbooks a flight which means that the booking count and the
  flight capacity are inconsistent.

There are various ways that systems overcome such inconsistency but it would be
advantageous to provide a generic solution which handles failure conditions,
maintains state for those flows that span long periods of time and ensures
that correcting activities are called correctly.

Traditional techniques for guaranteeing consistency in distributed environments
has focused on XA transactions where locks may be held for long periods thereby
introducing strong coupling between services and decreasing concurrency to
unacceptable levels. Additionally, if such a transaction aborts then valuable
work which may be valid will be rolled back. In view of these issues an
alternative approach is desirable.

Goals

- support long running actions 
- no strong coupling between services
- allow actions to finish early 
- allow compensating actions if a business activity is cancelled
- support short running actions that may need to be started to
  facilitate the execution of a step in a long running action

## Proposed solution

We propose a compensation based approach in which participants 
make changes visible but register a compensatory action
which is performed if something goes wrong. We call the model LRA
(short for Long Running Action) and is based on work done within
the [OASIS Web Services Composite Application Framework Technical
Committee](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=ws-caf):
[Long Running Action transaction
model](https://www.oasis-open.org/committees/document.php?document_id=12794)
but updated to be more suited for use in microservice based architectures.

Within the LRA model, an activity reflects business interactions:
all work performed within the scope of an activity is required to be
compensatable. Therefore, an activity’s work is either performed successfully
or undone. How services perform their work and ensure it can be undone if
compensation is required are implementation choices and is not exposed to the
LRA model which simply defines the triggers for compensation actions and the
conditions under which those triggers are executed. In other words, a LRA
coordinator is concerned only with ensuring participants obey the protocol
necessary to make an activity compensatable (and the semantics of the business
interactions are not part of the model). Issues such as isolation of services
between potentially conflicting activities and durability of service work
are assumed to be implementation decisions. The coordination protocol used to
ensure an activity is completed successfully or compensated is not two-phase
and is intended to better model interactions between microservices. Although
this may result in non-atomic behaviour for the overall business activity,
other activities may be started by the service to attempt to compensate in
some other manner.

In the model, an LRA is tied to the scope of an activity so that when the
activity terminates the LRA coordination protocol will be automatically
performed either to accept or to compensate the work. For example, when a
user reserves a seat on a flight, the airline reservation centre may take an
optimistic approach and actually book the seat and debit the user's account,
relying on the fact that most of their customers who reserve seats later
book them; the compensation action for this activity would be to un-book
the seat and credit the user’s account.

As in any business interaction, service activities may or may not be
compensatable. Even the ability to compensate may be a transient capability
of a service. A Compensator (or simply LRA participant) is the LRA participant
that operates on behalf of a service to undo the work it performs within the
scope of a LRA or to compensate for the fact that the original work could not
be completed.

### The Model

The model concerns participants and a coordinator. A client starts a new
LRA via a call to an LRA coordination service. This call creates a new LRA coordinator.
When a business service does work that may have to be later compensated
for within the scope of the LRA, it enlists a participant with the LRA
coordinator. Subsequently the client closes or cancels the LRA via the
coordinator which in turn tells all enlisted participants to either
complete or compensate:

![LRA Protocol Sequence](./lra.png)

The lifecycle of a LRA

The LRA participant will be invoked in the following way by the LRA
coordinator when the activity terminates:
 
* Success: the activity has completed successfully. If the activity is
  nested then participants may propagate themselves to the enclosing LRA.
  Otherwise the participants are informed that the activity has terminated
  and they can perform any necessary cleanups.
* Fail: the activity has completed unsuccessfully. All participants that
  are registered with the LRA will be invoked to perform compensation in
  the reverse order. The coordinator forgets about all participants that
  indicated they operated correctly. Otherwise, compensation may be
  attempted again (possibly after a period of time) or alternatively a
  compensation violation has occurred and must be logged. Each service is
  required to log sufficient information in order to ensure (with best
  effort) that compensation is possible. Each participant or subordinate
  coordinator (in the case of nested LRAs) is responsible for ensuring
  that sufficient data is made durable in order to undo the LRA in the
  event of failures. Interposition and check pointing of state allow the
  system to drive a consistent view of the outcome and recovery actions taken,
  but allowing always the possibility that recovery isn’t possible and must
  be logged or flagged for the administrator. In a large scale environment
  or in the presence of long term failures recovery may not be automatic and
  manual intervention may be necessary to restore an application’s consistency.
  Note that calling participants in reverse order does not guarantee that
  the compensation actions will be performed in strict sequential
  order since participants are allowed to indicate that the compensation
  is in progress and will complete at some future time. Furthermore a
  participant can indicate that it failed to compensate in which case
  it will be periodically retried (out of order).

Participants follow a state model:
<a name="participant-state-model"></a>

   - `Compensating`: a participant is currently compensating for the work
      it did during the LRA;
   - `Compensated`: a participant has successfully compensated for the LRA.
   - `FailedToCompensate`: the participant was not able to compensate for
      the LRA. It must maintain information about the work it was to
      compensate until the coordinator sends it a forget message.
   - `Completing`: the participant is tidying up after being told to complete.
   - `Completed`: the participant has confirmed that it has finished tidying up.
   - `FailedToComplete`: the participant was unable to tidy-up.
 
The LRA follows a similar state model:
<a name="LRA-state-model"></a>

   - `Compensating`: the LRA is currently being cancelled
   - `Compensated`: the LRA has successfully cancelled
   - `FailedToCompensate`: one or more participants was not able to compensate
   - `Completing`: the LRA is currently being closed
   - `Completed`: the LRA has closed
   - `FailedToComplete`: one or more participants was not able to complete
 
Different usage patterns for LRAs are possible, for example LRAs may
be used sequentially and concurrently, where the termination of one LRA
signals the start of some other unit of work within an application. However,<commit_msg>allow for unavailable participant<commit_after>Table of Contents

  - [Long Running Actions](#long-running-actions)
    - [Introduction](#introduction)
    - [Motivation](#motivation)
    - [Proposed solution](#proposed-solution)
      - [The Model](#the-model)
      - [CDI Annotations for LRAs](#cdi-annotations-for-lras)
        - [Java Annotations](#java-annotations)
          - [Controlling the lifecycle of an LRA](#controlling-the-lifecycle-of-an-lra)
          - [Compensating Activities](#compensating-activities)
          - [Nesting LRAs](#nesting-lras)
          - [Timing out LRAs and Compensators](#timing-out-lras-and-compensators)
          - [Leaving an LRA](#leaving-an-lra)
          - [Reporting the status of a participant](#reporting-the-status-of-a-participant)
          - [Forgetting an LRA](#forgetting-an-lra)
        - [Injecting an LRA Client API](#injecting-an-lra-client-api)
        - [Java based participant registration API](#java-based-participant-registration-api)
      - [Interoperability with other languages](#interoperability-with-other-languages)
        - [Protocol URLs](#protocol-urls)
          - [LRA Coordinator URL](#lra-coordinator-url)
          - [The LRA URL](#the-lra-url)
          - [Participants](#participants)
  - [Short Running Actions](#short-running-actions)
    - [Architecture](#architecture)
    - [State Transitions](#state-transitions)
    - [Transaction Context Propagation](#transaction-context-propagation)
    - [CDI Annotations for SRAs](#cdi-annotations-for-sras)
      - [SRA Lifecycle Support](#sra-lifecycle-support)
      - [SRA Participant Support](#sra-participant-support)
        - [Rollback](#rollback)
        - [Commit](#commit)
        - [OnePhaseCommit](#onephasecommit)
        - [Prepare](#prepare)
        - [Status](#status)
      - [Timing out SRAs and Participants](#timing-out-sras-and-participants)
        - [TimeLimit](#timelimit)
      - [Java based participant registration API](#java-based-participant-registration-api)
  - [Relationship Between SRAs and LRAs](#relationship-between-sras-and-lras)
  - [Appendix 1](#appendix-1)
    - [Typical Recovery Scenarios](#typical-recovery-scenarios)

# Consistency Models for Microservices
# Long Running Actions

* Proposal: [MP-0009](0009-LRA.md)
* Authors: [<NAME>](https://github.com/mmusgrov/)
* Status: **Sandbox**
* Decision Notes: [Discussion thread topic covering the
Rationale](https://groups.google.com/forum/#!searchin/microprofile/transactions%7Csort:relevance/microprofile/CJirjFkM9Do/TrApz-fBDQAJ)

## Introduction

The proposal introduces APIs for services to coordinate activities.

The main thrust of the proposal introduces an API for loosely coupled
services to coordinate long running activities in such a way as to guarantee
a globally consistent outcome without the need to take locks on data.

A secondary aim of the proposal is to introduce an API for services to
coordinate short running activities in such a way as to guarantee
a globally consistent outcome but this time with the possibility that 
it may be necessary to lock data. The goal of this part of the specification
is to facilitate the developer in writing the individual steps involved in a
long running activity.

The APIs are framed in terms of Java CDI annotations that should be applied
to JAX-RS resources. However, the proposal is also described in terms of
REST resources which means that services written in other languages can
also coordinate with CDI annotated JAX-RS resources. And for services that
do not communicate via HTTP we also define a pure java based registration API.

It is expected that different implementations of this specification will need
to interoperate and implementers should pay special attention
[to the section on interoperability](Interoperability with other languages)
to achieve this goal.

## Motivation

In a loosely coupled service based environment there is sometimes a need for
different services to provide consistency guarantees. Typical examples include:

* order processing involving three services (take order, bill customer,
  ship product). If the shipping service finds that it is out of stock then
  the customer will have been billed with no prospect of receiving his item;
* an airline overbooks a flight which means that the booking count and the
  flight capacity are inconsistent.

There are various ways that systems overcome such inconsistency but it would be
advantageous to provide a generic solution which handles failure conditions,
maintains state for those flows that span long periods of time and ensures
that correcting activities are called correctly.

Traditional techniques for guaranteeing consistency in distributed environments
has focused on XA transactions where locks may be held for long periods thereby
introducing strong coupling between services and decreasing concurrency to
unacceptable levels. Additionally, if such a transaction aborts then valuable
work which may be valid will be rolled back. In view of these issues an
alternative approach is desirable.

Goals

- support long running actions 
- no strong coupling between services
- allow actions to finish early 
- allow compensating actions if a business activity is cancelled
- support short running actions that may need to be started to
  facilitate the execution of a step in a long running action

## Proposed solution

We propose a compensation based approach in which participants 
make changes visible but register a compensatory action
which is performed if something goes wrong. We call the model LRA
(short for Long Running Action) and is based on work done within
the [OASIS Web Services Composite Application Framework Technical
Committee](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=ws-caf):
[Long Running Action transaction
model](https://www.oasis-open.org/committees/document.php?document_id=12794)
but updated to be more suited for use in microservice based architectures.

Within the LRA model, an activity reflects business interactions:
all work performed within the scope of an activity is required to be
compensatable. Therefore, an activity’s work is either performed successfully
or undone. How services perform their work and ensure it can be undone if
compensation is required are implementation choices and is not exposed to the
LRA model which simply defines the triggers for compensation actions and the
conditions under which those triggers are executed. In other words, a LRA
coordinator is concerned only with ensuring participants obey the protocol
necessary to make an activity compensatable (and the semantics of the business
interactions are not part of the model). Issues such as isolation of services
between potentially conflicting activities and durability of service work
are assumed to be implementation decisions. The coordination protocol used to
ensure an activity is completed successfully or compensated is not two-phase
and is intended to better model interactions between microservices. Although
this may result in non-atomic behaviour for the overall business activity,
other activities may be started by the service to attempt to compensate in
some other manner.

In the model, an LRA is tied to the scope of an activity so that when the
activity terminates the LRA coordination protocol will be automatically
performed either to accept or to compensate the work. For example, when a
user reserves a seat on a flight, the airline reservation centre may take an
optimistic approach and actually book the seat and debit the user's account,
relying on the fact that most of their customers who reserve seats later
book them; the compensation action for this activity would be to un-book
the seat and credit the user’s account.

As in any business interaction, service activities may or may not be
compensatable. Even the ability to compensate may be a transient capability
of a service. A Compensator (or simply LRA participant) is the LRA participant
that operates on behalf of a service to undo the work it performs within the
scope of a LRA or to compensate for the fact that the original work could not
be completed.

### The Model

The model concerns participants and a coordinator. A client starts a new
LRA via a call to an LRA coordination service. This call creates a new LRA coordinator.
When a business service does work that may have to be later compensated
for within the scope of the LRA, it enlists a participant with the LRA
coordinator. Subsequently the client closes or cancels the LRA via the
coordinator which in turn tells all enlisted participants to either
complete or compensate:

![LRA Protocol Sequence](./lra.png)

The lifecycle of a LRA

The LRA participant will be invoked in the following way by the LRA
coordinator when the activity terminates:
 
* Success: the activity has completed successfully. If the activity is
  nested then participants may propagate themselves to the enclosing LRA.
  Otherwise the participants are informed that the activity has terminated
  and they can perform any necessary cleanups.
* Fail: the activity has completed unsuccessfully. All participants that
  are registered with the LRA will be invoked to perform compensation in
  the reverse order. The coordinator forgets about all participants that
  indicated they operated correctly. Otherwise, compensation may be
  attempted again (possibly after a period of time) or alternatively a
  compensation violation has occurred and must be logged. Each service is
  required to log sufficient information in order to ensure (with best
  effort) that compensation is possible. Each participant or subordinate
  coordinator (in the case of nested LRAs) is responsible for ensuring
  that sufficient data is made durable in order to undo the LRA in the
  event of failures. Interposition and check pointing of state allow the
  system to drive a consistent view of the outcome and recovery actions taken,
  but allowing always the possibility that recovery isn’t possible and must
  be logged or flagged for the administrator. In a large scale environment
  or in the presence of long term failures recovery may not be automatic and
  manual intervention may be necessary to restore an application’s consistency.
  Note that calling participants in reverse order does not guarantee that
  the compensation actions will be performed in strict sequential
  order since participants are allowed to indicate that the compensation
  is in progress and will complete at some future time. Furthermore a
  participant can indicate that it failed to compensate, or could be unavailable
  in which case it will be periodically retried (out of order).

Participants follow a state model:
<a name="participant-state-model"></a>

   - `Compensating`: a participant is currently compensating for the work
      it did during the LRA;
   - `Compensated`: a participant has successfully compensated for the LRA.
   - `FailedToCompensate`: the participant was not able to compensate for
      the LRA. It must maintain information about the work it was to
      compensate until the coordinator sends it a forget message.
   - `Completing`: the participant is tidying up after being told to complete.
   - `Completed`: the participant has confirmed that it has finished tidying up.
   - `FailedToComplete`: the participant was unable to tidy-up.
 
The LRA follows a similar state model:
<a name="LRA-state-model"></a>

   - `Compensating`: the LRA is currently being cancelled
   - `Compensated`: the LRA has successfully cancelled
   - `FailedToCompensate`: one or more participants was not able to compensate
   - `Completing`: the LRA is currently being closed
   - `Completed`: the LRA has closed
   - `FailedToComplete`: one or more participants was not able to complete
 
Different usage patterns for LRAs are possible, for example LRAs may
be used sequentially and concurrently, where the termination of one LRA
signals the start of some other unit of work within an application. However,
{'fraction_non_alphanumeric': 0.05028119141845449, 'fraction_numerical': 0.001416371589252239, 'mean_word_length': 4.3110619469026545, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 8, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5261904', 'n_tokens_mistral': 6017, 'n_tokens_neox': 5604, 'n_words': 3094}