<commit_before># Siddhi Architecture 

WSO2 Siddhi is a software that can run as server by it own or within WSO2 Stream Processor, embalmed in to Java based 
systems and Android applications to provide analytical operators, orchestrate data flow, calculate analytics, and detect
patterns on event data from multiple, disparate live data sources to allow developers to build applications that sense, 
think, and act in real time.

This section illustrate the architecture of Siddhi and take you through its key functionality; We hope that the article 
will help developers to better understand the code, and will help them to better used siddhi, fix bugs and also improve Siddhi. 

## Main Design Decisions

- Siddhi is implemented specifically for event by event processing of real-time streaming data. 
- It provides an intuitive way to expressing stream processing logic and Complex 
Event Processing constructs like Patterns and Sequences with its Siddhi Streaming SQL.  
- It processes all events in-memory and when necessary it also keeps its processing state in-memory. 
- It enforces strict schema for event streams such that it can precompile the queries to support efficient event data retrieval, 
manipulation, optimise performance and to optimise memory consumption.  
- All processing is optimise for memory consumption by only having the absolute necessary information in memory and 
dropping the rest as soon as possible. 
- Have multiple extension points to support diverse set of functionality such as supporting multiple sources, sinks, functions, 
aggregation operations, windows and many more.

## High Level Architecture

![Simple Siddhi Overview](../images/architecture/siddhi-overview-simple.png?raw=true "Simple Siddhi Overview")
 
In very high level Siddhi consumes events from various events source processed them according to the defined Siddhi application 
 and produce results to subscribed event sinks. 
 Siddhi stores its processing events in in-memory table or when configured on external data stores such as `RDBMS`, `MangoDB`, 
 `Hazelcast` in-memory grid, etc. Siddhi also allows applications and users to query Siddhi via its Store Query API to interactively 
 retrieve data from its in-memory and other stores.
 
### Main Modules of Siddhi

Siddhi comprises four main components, they are: 

- **Siddhi Query API** : This let you define execution logic via the Siddhi application and its queries and definitions using POJO classes. 
Internally Siddhi uses these object to understand what user expects Siddhi to do. 

- **Siddhi Query Compiler** : This let you use the Siddhi Streaming SQL to define the Siddhi application and its queries and definitions, 
 and it converts the Streaming SQL to Siddhi Query Objects such that Siddhi to execute them. 
 
- **Siddhi Core** : This is responsible for building the execution runtime and processing events as and when they arrive. 
 
- **Siddhi Annotation** : This is a helping component that lets all extensions to be annotated such that they can be 
pricked by Siddhi Core for processing and help you generate appropriate documentation. 

### Siddhi Component Architecture 

The following diagram gives more detail information on major components of Siddhi and how they are related together. 

![Siddhi Component Architecture](../images/architecture/siddhi-architecture-highlevel.png "Siddhi Component Architecture")
 
Here Siddhi Core module is responsible for maintaining the execution logic, and interacting with the external environment and systems 
 to consuming and publishing events. It uses multiple components to achieve its tasks such as:  
 
- Siddhi Manager : This is one of the critical component inside Siddhi Core that is responsible for managing Siddhi App Runtimes 
  and facilitate their functionality via Siddhi Context with periodic state persistence, statistics reporting and extension loading. 
   
- Siddhi App Runtime : There will be one Siddhi App runtime generated for each Siddhi App deployed by the user. Siddhi Apps runtime
provide an isolated execution environment for all of the queries and execution logic defined in the corresponding Siddhi App.
 These Siddhi App Runtimes based on the logic defied consumes and publishes events from various external system, Java and/or Python programmes. 
  
- Siddhi Context : This is a shared object for all the Siddhi App runtimes within the a Siddhi manager, and it contains references 
to the persistence store for periodic persistence, statistics manager to report performance statistics of Siddhi App Runtimes 
and for loading Siddhi extensions. 

### Siddhi App Execution Flow

Following diagram depicts the execution flow within a Siddhi App Runtime. 

![Execution Flow in Siddhi App](../images/architecture/siddhi-event-flow.png "Execution Flow in Siddhi App")
 
The path events take within the system is coloured in blue. 

Flowing are the components get involve in handling the events. 

- Stream Junction
    
    Responsible for routing events to various components within the Siddhi App Runtime, there will a Stream Junction 
    generated for for each stream defined or inferred in the Siddhi App. This by default uses the incoming event's thread
    for processing subscribed components, but it can also be configured via `@Async` annotation to buffer the events and 
    use a different thread for subsequent execution.
    
- Input Handler 
    
    There will an instance of Input Handler for each Stream Junction, and this is used for pushing `Event` and
     `Event[]`s objects into Stream Junctions from Sources and from Java and Python programmes 
    
- Stream Callback 

    This is responsible for receiving `Event[]`s from Stream Junction and pass them to Sinks to publish to external endpoints 
    or pass them subscribed Java and Python programmes for further processing. 
    
- Queries & Partitions 

    They are the components responsible for processing the events by filtering, transforming, joining, patten matching, 
    etc. They consume events from one or more Stream Junctions, process them and publishes the newly produced events 
     into Stream Junctions based on the defined query or partition. 
     
- Source 
    
    This is responsible for consuming events from external sources on various data formats convert the events into Siddhi event
    and then pass them to corresponding Stream Junction via it's Input Handler. There will be one Source generated 
    for each `@Source` annotation defined on top of the Streams. 
    
- Source Mapper 

    There will be a Source Mapper for each source, responsible for converting the incoming event format into Siddhi Events. 
    Source Mapper type can be configured using the `@Map` annotation used within the `@Source` annotation. When `@Map` annotation 
    is not defined Siddhi uses PassThroughMapper, assuming that the incoming message is already in Siddhi Event format and 
    it does not need any changes.
    
- Sink 

    This is responsible for publish the events to external endpoints by converting the Siddhi Events to various data formats. 
    There will be one Sink generated for each `@Sink` annotation defined on top of the Streams to consume and publish the events arriving 
    on that stream. 
    
- Sink Mapper 

    There will be one Sink Mapper for each Sink inorder to map the Siddhi events to various data formats such that they 
    can be published via Sink. Sink Mapper type can be configured using the `@Map` annotation used within the `@Sink`
    annotation. When `@Map` annotation is not defined Siddhi uses PassThroughMapper, by passing the events as it is 
    any to the Sink.
    
- Table
    
    Table is responsible for storing events, by default Siddhi uses In-Memory Table implementation, and when `@Store` annotation
    is used it load the associated Table implantation based on the defined `store` type. Most table implementations are 
    extended from the AbstractRecordTable abstract class for the easy of development.
    
- Window
    
    Window is responsible for storing and expiring the events based on the given window constrain. Multiple version of windows 
    can be implemented by extending the WindowProcessor abstract class. 
    
- Incremental Aggregation 

    Manges incremental aggregation by letting you obtain aggregates in an incremental manner for a specified set of time periods.
    When defining aggregates incremental aggregation functions can be implemented by extending IncrementalAttributeAggregator. 
     
- Trigger
 
    Trigger triggers events on a given interval to the stream junction with has the same name as of the Trigger.
    
- Query Callback 

    This can be used to get events generated from Queries, this notifies the event occurrence `timestamp`, and classifies 
    the events into `currentEvents`, and `expiredEvents`. 
    
## Siddhi Query Execution 

Siddhi queries can be categorised in to three main types such as single input queries which comprises query types filter and windows,
 two input queries comprising joins, and multi input queries comprising pattern and sequences. 

Following section explains the internal of each query type. 

### Single Input Query Runtime (Filter & Windows)

![Single Input Query](../images/architecture/siddhi-query-single.png "Single Input Query (Filter & Window)")
 
Single input query runtime is generated for filter and window queries, they consumes events from a Stream Junction 
and convert the events according to the expected output stream format at the ProcessStreamReceiver dropping all the unrelated 
incoming stream attributes. 


- Partition
    - inner stream
<commit_msg>adding content to Single input Stream query<commit_after># Siddhi Architecture 

WSO2 Siddhi is a software that can run as server by it own or within WSO2 Stream Processor, embalmed in to Java based 
systems and Android applications to provide analytical operators, orchestrate data flow, calculate analytics, and detect
patterns on event data from multiple, disparate live data sources to allow developers to build applications that sense, 
think, and act in real time.

This section illustrate the architecture of Siddhi and take you through its key functionality; We hope that the article 
will help developers to better understand the code, and will help them to better used siddhi, fix bugs and also improve Siddhi. 

## Main Design Decisions

- Siddhi is implemented specifically for event by event processing of real-time streaming data. 
- It provides an intuitive way to expressing stream processing logic and Complex 
Event Processing constructs like Patterns and Sequences with its Siddhi Streaming SQL.  
- It processes all events in-memory and when necessary it also keeps its processing state in-memory. 
- It enforces strict schema for event streams such that it can precompile the queries to support efficient event data retrieval, 
manipulation, optimise performance and to optimise memory consumption.  
- All processing is optimise for memory consumption by only having the absolute necessary information in memory and 
dropping the rest as soon as possible. 
- Have multiple extension points to support diverse set of functionality such as supporting multiple sources, sinks, functions, 
aggregation operations, windows and many more.

## High Level Architecture

![Simple Siddhi Overview](../images/architecture/siddhi-overview-simple.png?raw=true "Simple Siddhi Overview")
 
In very high level Siddhi consumes events from various events source processed them according to the defined Siddhi application 
 and produce results to subscribed event sinks. 
 Siddhi stores its processing events in in-memory table or when configured on external data stores such as `RDBMS`, `MangoDB`, 
 `Hazelcast` in-memory grid, etc. Siddhi also allows applications and users to query Siddhi via its Store Query API to interactively 
 retrieve data from its in-memory and other stores.
 
### Main Modules of Siddhi

Siddhi comprises four main components, they are: 

- **Siddhi Query API** : This let you define execution logic via the Siddhi application and its queries and definitions using POJO classes. 
Internally Siddhi uses these object to understand what user expects Siddhi to do. 

- **Siddhi Query Compiler** : This let you use the Siddhi Streaming SQL to define the Siddhi application and its queries and definitions, 
 and it converts the Streaming SQL to Siddhi Query Objects such that Siddhi to execute them. 
 
- **Siddhi Core** : This is responsible for building the execution runtime and processing events as and when they arrive. 
 
- **Siddhi Annotation** : This is a helping component that lets all extensions to be annotated such that they can be 
pricked by Siddhi Core for processing and help you generate appropriate documentation. 

### Siddhi Component Architecture 

The following diagram gives more detail information on major components of Siddhi and how they are related together. 

![Siddhi Component Architecture](../images/architecture/siddhi-architecture-highlevel.png "Siddhi Component Architecture")
 
Here Siddhi Core module is responsible for maintaining the execution logic, and interacting with the external environment and systems 
 to consuming and publishing events. It uses multiple components to achieve its tasks such as:  
 
- Siddhi Manager : This is one of the critical component inside Siddhi Core that is responsible for managing Siddhi App Runtimes 
  and facilitate their functionality via Siddhi Context with periodic state persistence, statistics reporting and extension loading. 
   
- Siddhi App Runtime : There will be one Siddhi App runtime generated for each Siddhi App deployed by the user. Siddhi Apps runtime
provide an isolated execution environment for all of the queries and execution logic defined in the corresponding Siddhi App.
 These Siddhi App Runtimes based on the logic defied consumes and publishes events from various external system, Java and/or Python programmes. 
  
- Siddhi Context : This is a shared object for all the Siddhi App runtimes within the a Siddhi manager, and it contains references 
to the persistence store for periodic persistence, statistics manager to report performance statistics of Siddhi App Runtimes 
and for loading Siddhi extensions. 

### Siddhi App Execution Flow

Following diagram depicts the execution flow within a Siddhi App Runtime. 

![Execution Flow in Siddhi App](../images/architecture/siddhi-event-flow.png "Execution Flow in Siddhi App")
 
The path events take within the system is coloured in blue. 

Flowing are the components get involve in handling the events. 

- Stream Junction
    
    Responsible for routing events to various components within the Siddhi App Runtime, there will a Stream Junction 
    generated for for each stream defined or inferred in the Siddhi App. This by default uses the incoming event's thread
    for processing subscribed components, but it can also be configured via `@Async` annotation to buffer the events and 
    use a different thread for subsequent execution.
    
- Input Handler 
    
    There will an instance of Input Handler for each Stream Junction, and this is used for pushing `Event` and
     `Event[]`s objects into Stream Junctions from Sources and from Java and Python programmes 
    
- Stream Callback 

    This is responsible for receiving `Event[]`s from Stream Junction and pass them to Sinks to publish to external endpoints 
    or pass them subscribed Java and Python programmes for further processing. 
    
- Queries & Partitions 

    They are the components responsible for processing the events by filtering, transforming, joining, patten matching, 
    etc. They consume events from one or more Stream Junctions, process them and publishes the newly produced events 
     into Stream Junctions based on the defined query or partition. 
     
- Source 
    
    This is responsible for consuming events from external sources on various data formats convert the events into Siddhi event
    and then pass them to corresponding Stream Junction via it's Input Handler. There will be one Source generated 
    for each `@Source` annotation defined on top of the Streams. 
    
- Source Mapper 

    There will be a Source Mapper for each source, responsible for converting the incoming event format into Siddhi Events. 
    Source Mapper type can be configured using the `@Map` annotation used within the `@Source` annotation. When `@Map` annotation 
    is not defined Siddhi uses PassThroughMapper, assuming that the incoming message is already in Siddhi Event format and 
    it does not need any changes.
    
- Sink 

    This is responsible for publish the events to external endpoints by converting the Siddhi Events to various data formats. 
    There will be one Sink generated for each `@Sink` annotation defined on top of the Streams to consume and publish the events arriving 
    on that stream. 
    
- Sink Mapper 

    There will be one Sink Mapper for each Sink inorder to map the Siddhi events to various data formats such that they 
    can be published via Sink. Sink Mapper type can be configured using the `@Map` annotation used within the `@Sink`
    annotation. When `@Map` annotation is not defined Siddhi uses PassThroughMapper, by passing the events as it is 
    any to the Sink.
    
- Table
    
    Table is responsible for storing events, by default Siddhi uses In-Memory Table implementation, and when `@Store` annotation
    is used it load the associated Table implantation based on the defined `store` type. Most table implementations are 
    extended from the AbstractRecordTable abstract class for the easy of development.
    
- Window
    
    Window is responsible for storing and expiring the events based on the given window constrain. Multiple version of windows 
    can be implemented by extending the WindowProcessor abstract class. 
    
- Incremental Aggregation 

    Manges incremental aggregation by letting you obtain aggregates in an incremental manner for a specified set of time periods.
    When defining aggregates incremental aggregation functions can be implemented by extending IncrementalAttributeAggregator. 
     
- Trigger
 
    Trigger triggers events on a given interval to the stream junction with has the same name as of the Trigger.
    
- Query Callback 

    This can be used to get events generated from Queries, this notifies the event occurrence `timestamp`, and classifies 
    the events into `currentEvents`, and `expiredEvents`. 
    

## Siddhi Query Execution 

Siddhi queries can be categorised in to three main types such as single input stream queries which comprises query types filter and windows,
 join input stream queries comprising joins, and state input stream queries comprising pattern and sequences. 

Following section explains the internal of each query type. 

### Single Input Stream Query Runtime (Filter & Windows)

![Single Input Query](../images/architecture/siddhi-query-single.png "Single Input Query (Filter & Window)")
 
Single input stream query runtime is generated for filter and window queries, they consumes events from a Stream Junction 
and convert the events according to the expected output stream format at the ProcessStreamReceiver and by dropping 
all the unrelated incoming stream attributes. Query runtime can also consume events from a Window.

Then the converted events are passed through few Processors such as Filter, StreamProcessor, StreamFunctionProcessor, 
WindowProcessor and QuerySelector. Here StreamProcessor, StreamFunctionProcessor, and WindowProcessor can be extended with 
various stream processing capabilities. The chain of processors are always ended with a QuerySelector and the chain can 
only have at most one WindowProcessor. When query runtime consumes events from a Window its chain of processors cannot 
contain a WindowProcessor.

When it comes to the Filter processor, its implemented with an Expression that returns boolean value. Expressions have a tree structure, and 
they are processed with the Depth First Search Algorithm. To achieve high performance, currently, Siddhi depends on the user
to formulate the lease success case in the leftmost side of the condition, thereby increasing the chances of early false detection.

The condition expression `price >= 100 and ( Symbol == 'IBM' or Symbol == 'MSFT' )` will be represented as below.

![Siddhi Expression](../images/architecture/siddhi-expressions.png "Single Input Query (Filter & Window)")

These Expressions also supports executing user defined functions (UDFs), and they can be implemented by extending the FunctionExecutor class. 

Events after getting processed by each processor they will reach to the QuerySelector for transformation. At the QuerySelector
Events are transformed based on the `select` clause of the query. If there is a `Group By` defined then the GroupByKeyGenerator identifies the 
group by key and then each AttributeProcessor is executed based on the group by key. AttributeProcessors can contain any Expression including 
constant values, variables and user defined functions, at the same time they can also contain AttributeAggregators for 
processing aggregation operations such as `sum`, `count`, etc. Here for each group by key a AttributeAggregator will be generated to 
keep track of the aggregation state and when it becomes obsolete it will be destroyed. Through this operation the event 
will be transformed to the expected output format. 
 
After the event got transformed to the output format it will evaluated against the HavingConditionExecutor if a `having` clause is 
provided, and only the succeeding events will be pushed to OutputRateLimiter.   

At OutputRateLimiter the events output is controlled before sending the events to the stream junction or to the query callback. 
When `output` clause is not defined PassThroughOutputRateLimiter is used by passing all the events without any rate limiting. 

### Join Input Stream Query Runtime (Join)
 
WIP

 
## Siddhi Event Formats

Siddhi has three event formats. 

- Event 

    This is the format exposed to end users when they send events via InputHandler and Consume events via StreamCallback or QueryCallback.
    This comprises an `Object[]` containing all the values in accordance to the corresponding Streams. 
     
- StreamEvent (Subtype of ComplexEvent)

    This is used within queries. This contains three `Object[]`s named as beforeWindowData, onAfterWindowData and outputData. Here the 
    outputData contains the values in accordance to the output Stream of the query, and beforeWindowData contains values that are only used 
    in Processors that are executed before the WindowProcessor, the onAfterWindowData will be contains values that are only used by WindowProcessor
    and other processors after that and that are not sent out as output. Here the content on the beforeWindowData will be cleared before the 
    event enter into the WindowProcessor to optimize the amount to data that will be stored in-memory at windows. StreamEvents can also be chained 
    by linking each other via the `next` property in them.  
    
- StateEvent (Subtype of ComplexEvent)

    This is used in Joins, Patterns and Sequences to when we need to associate with multiple different type of Streams, Tables, Windows and Aggregations.
    This has a collection StreamEvents representing different streams used in the query and outputData for containing the values that need to be outputted 
    according to the output Stream of the query. StreamEvents can also be chained by linking each other with the `next` property in them. 
    
**Event Chunks**

Event Chunks provide an easier way of manipulating the chain of StreamEvents and StateEvents, such that they are be easily iterated, removed 
and letting new events to be inserted. 



- Partition
    - inner stream

{'fraction_non_alphanumeric': 0.030956260182980325, 'fraction_numerical': 0.0002924343067218114, 'mean_word_length': 4.16573154941735, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '400064', 'n_tokens_mistral': 5685, 'n_tokens_neox': 5292, 'n_words': 3370}