TECHNICAL FIELD

One or more implementations relate to the field of event consumption; and more specifically, to event consumer management in an aggregated event platform.

BACKGROUND ART

Web applications that serve and manage millions of Internet users, such as Facebook™, Instagram™, Twitter™, banking websites, as well as online retail shops, such as Amazon.com™ or eBay™ are faced with the challenge of ingesting high volumes of data as fast as possible so that the end users can be provided with a real-time experience.

The “Internet of Things” (IoT) is another major contributor to big data, supplying huge volumes of data. IoT has become a pervasive presence in the environment, with a variety of things/objects that communicate via wireless and wired connections to interact with each other and cooperate with other things/objects to create new applications/services. These applications/services exist in smart cities (regions), smart cars and mobility, smart homes and assisted living, smart industries, public safety, energy and environmental protection, agriculture and tourism. A massive quantity of data gets persisted from the millions of IoT devices and web applications.

In current event recordation systems, a high volume of events is stored and shared across features, tenants, and/or topics. These events are stored in an event recordation system based on an aggregate topic, where each aggregate topic is used for a subset of tenants from multiple tenants in a multi-tenant environment or for a subset of topics from multiple topics available for each tenant (e.g., the events are stored across a set of Kafka partitions, where each Kafka partition is used for a subset of tenants from a multi-tenant system or alternatively each Kafka topic is used for multiple initial topics with which the events were received in the system).

While the events are persisted based on an aggregate topic, these events are generated and consumed based on initial topics that are different from the aggregate topics. Since many initial topics and/or associated initial partitions share a given physical storage channel identified with an aggregate topic, if an instance of an event consumer is to request the events based on the aggregate topic, when receiving the events, the event consumer potentially needs to skip a large number of events that are associated with other initial topics and/or partitions in order to get data for a desired initial topic. The management of the instances of event consumers is quite challenging in an aggregated event platform. For example, if instances of event consumers are regularly put to sleep (i.e., in an idle state) each time an instance of an event consumer is woken up from the idle state, there's a large number of events to process and filter before relevant events are obtained. Alternatively, if the instances of the event consumers are never put to sleep (i.e., they are always running), this puts more load on the system, even if events aren't being consumed for certain initial topics. In another example, if event consumers are woken up each time an event is published, overhead for all event published is added.

