CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No. 17/145,971, entitled “Application Code Management Using an Event Stream,” filed on Jan. 11, 2021, and incorporated by reference herein in its entirety.

TECHNICAL FIELD

This disclosure relates to managing event streams in a shared computer environment, and more specifically to techniques for updating schemas for data communicated on the event stream and application logic for applications using the event stream.

BACKGROUND

A shared event stream provides a useful medium for globally ordering messages from different sources at high speed. This can support high throughput and low latency when processing transactions or other information within a local computing environment, a distributed computing environment, or some combination of these. However, users of the event stream may periodically want or need to change the format in which they share data, or the manner in which they process data. There remains a need for techniques by which a user of the event stream can update how data is shared or processed in a time-based manner synchronized with the event stream so that other users can reliably encode and decode event stream messages.

SUMMARY

In a message processing system, an application connected to an event stream publishes a schema for the structure of data payloads contained in messages from the application. Other readers and writers within the system can receive the schema and apply the schema to subsequent inbound and/or outbound messages as appropriate. The application may from time to time dynamically update the schema and publish a revised schema for use by the other applications connected to the event stream for subsequent messages. In another aspect, the application may receive code update instructions or identify code versions using the event stream.

In an aspect, a method disclosed herein includes: reading sequenced messages from an event stream, the sequenced messages having globally unique sequence numbers within the event stream; processing the sequenced messages with an application to provide a first number of unsequenced messages responsive to the sequenced messages, where the application generates messages using a first schema identified by a first schema identifier; writing the first number of unsequenced messages to the event stream without globally unique sequence numbers using a first data structure specified by the first schema; revising the first data structure to provide a second data structure specified by a second schema; writing a schema change message to the event stream with a second schema identifier that identifies the second schema; and, after writing the schema change message to the event stream and receiving a sequenced version of the schema change message on the event stream, writing a second number of unsequenced messages to the event stream using the second data structure specified by the second schema.

Implementations may include one or more of the following features. The globally unique sequence numbers may be monotonically increasing numbers assigned by a sequencer for the event stream according to a time received by the sequencer. The method may further include adding a time stamp to each of the sequenced messages with the sequencer. The method may further include, in response to receiving a second schema change message with one of the globally unique sequence numbers on the event stream from a second application, applying a corresponding schema to one or more sequenced messages from the second application that have reference sequence numbers greater than or equal to the one of the globally unique sequence numbers for the second schema change message. The method may further comprise including a corresponding one of the first schema identifier or the second schema identifier in each of the unsequenced messages from the application. The method may further include executing a plurality of instances of the application at a plurality of venues coupled in a communicating relationship with the event stream. The method may further include digitally signing each of the unsequenced messages before writing to the event stream. The method may further include filtering the sequenced messages to remove one or more of the sequenced messages from a local queue for the application. The application may process the sequenced messages with an order of execution determined by the globally unique sequence numbers. The method may further include processing one of the sequenced messages from the application sourcing the schema change message with a second application, where the second application uses the first schema when a reference sequence number for the one of the sequenced messages is less than the globally unique sequence number for the schema change message, and where the second application uses the second schema when the reference sequence number for the one of the sequenced messages is equal to or greater than the globally unique sequence number for the schema change message. The method may further include sequencing messages on the event stream with a sequencer by: reading unsequenced messages from the event stream, applying a time stamp to each of the unsequenced messages corresponding to a time received at the sequencer, arranging the unsequenced messages in a chronological order according to the time received at the sequencer, assigning monotonically increasing sequence numbers according to the chronological order to the unsequenced messages as the globally unique sequence numbers, and writing the unsequenced messages with the monotonically increasing sequence numbers to the event stream as sequenced messages. The method may further include processing an inbound message to the application using an updated schema from a source application based on all applicable schema update messages having globally unique sequence identifiers up to and including a reference sequence number for the inbound message. The method may further include writing an outbound message from the application using an updated schema for the application based on all applicable schema update messages having globally unique sequence identifiers up to and including a reference sequence number for the outbound message.

In an aspect, a computer program product disclosed herein includes computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, updates schemas used in messages among users of an event stream by performing the steps of: reading sequenced messages from the event stream, the sequenced messages including sequence numbers; processing the sequenced messages with an application to provide a first number of unsequenced messages responsive to the sequenced messages, where the application generates messages using a first schema identified by a first schema identifier; writing the first number of unsequenced messages to the event stream without sequence numbers using a first data structure specified by the first schema, each unsequenced message including a digital signature for authenticating the application; revising the first data structure to provide a second data structure specified by a second schema; writing a schema change message to the event stream with a second schema identifier that identifies the second schema; and, after writing the schema change message to the event stream and receiving a sequenced version of the schema change message on the event stream, writing a second number of unsequenced messages to the event stream using the second data structure specified by the second schema.

Implementations may include one or more of the following features. The computer program product may further include code that sequences messages on the event stream by performing the steps of: reading unsequenced messages from the event stream, applying a time stamp to each of the unsequenced messages corresponding to a time received at a sequencer, arranging the unsequenced messages in a chronological order according to the time received at the sequencer, assigning monotonically increasing sequence numbers to the unsequenced messages as the sequence numbers according to the chronological order, and writing the unsequenced messages with the monotonically increasing sequence numbers to the event stream as sequenced messages. The sequence numbers may be monotonically increasing numbers assigned by the sequencer for the event stream according to a time received by the sequencer. The computer program product may further include code that performs the step of adding a time stamp to each of the sequenced messages with the sequencer. Each of the sequence numbers may be a globally unique sequence number within the event stream. The computer program product may further include code that performs the step of, in response to receiving a second schema change message with one of the sequence numbers on the event stream from a second application, applying a corresponding schema to one or more sequenced messages from the second application having a reference sequence number greater than or equal to the one of the sequence numbers for the second schema change message. The computer program product may further include code that performs the step of processing an inbound message to the application using an updated schema from a source application based on all applicable schema update messages having globally unique sequence identifiers up to and including a reference sequence number for the inbound message. The computer program product may further include code that performs the step of writing an outbound message from the application using an updated schema for messages from the application based on all applicable schema update messages having globally unique sequence identifiers up to and including a reference sequence number for the outbound message.

In an aspect, a computing system disclosed herein includes: an event stream; a sequencer that receives unsequenced messages from users of the event stream, arranges the unsequenced messages into sequenced messages having a global sequence in the event stream, and publishes the sequenced messages to the event stream; and an application coupled in a communicating relationship with the event stream and the sequencer, the application including a writer configured to publish one or more of the unsequenced messages, a reader to read the sequenced messages from the event stream, and a processing engine that sends a first message to the event stream containing a description of a schema that specifies a structure for data in one or more other messages published by the writer of the application.

Implementations may include one or more of the following features. The application may receive one or more of the sequenced messages from a second application coupled to the event stream, determine a second schema used by the second application based on schema information for the second application in the event stream, and process the sequenced messages from the second application according to the second schema. The application may be configured to change the schema to a second schema and to publish a schema update message to the event stream notifying other users of the event stream of the second schema.

In an aspect, a method disclosed herein includes: generating a first number of unsequenced messages with an application using a first schema identified by a first schema identifier; writing the first number of unsequenced messages to an event stream without globally unique sequence numbers for the event stream using a first data structure specified by the first schema; revising the first data structure to provide a second data structure specified by a second schema; writing a schema change message to the event stream with a second schema identifier that identifies the second schema; and, after receiving a sequenced version of the schema change message from a sequencer for the event stream, writing a second number of unsequenced messages to the event stream using the second data structure specified by the second schema.

Implementations may include one or more of the following features. Revising the first data structure may include revising the first data structure in response to an update in application logic for the application. Revising the first data structure may include revising the first data structure in response to a schema update received from schema management application coupled to the event stream. One of the first number of unsequenced messages may be generated by the application independently from messages received on the event stream. One of the first number of unsequenced messages may be generated by the application in response to a message received on the event stream, where the method further includes adding a reference sequence number to the one of the first number of unsequenced messages based on a globally unique sequence identifier for the message received on the event stream. The method may further include reading sequenced messages from the event stream, the sequenced messages having globally unique sequence numbers within the event stream, and processing the sequenced messages with the application.

In a processing system, an identifier for a current version of code for an application is published in a message to a shared event stream. Any instances of the application within the processing system may update to the current version before processing other messages having sequence numbers after the message containing the identifier. In this manner, multiple instances of the application may be periodically updated to maintain global consistency of the application and the shared event stream.

In an aspect, a method disclosed herein includes: reading sequenced messages from an event stream, the sequenced messages including globally unique sequence identifiers; writing unsequenced messages to the event stream from an application without globally unique sequence identifiers; in response to identifying one of the sequenced messages containing a second version number for the application different than a first version number corresponding to a current version of the application, retrieving a different version of the application corresponding to the second version number and replacing the current version of the application with the different version of the application; and processing one or more of the sequenced messages having a reference sequence number greater than or equal to the one of the globally unique sequence identifiers for the one of the sequenced messages containing the second version number using the different version of the application.

Implementations may include one or more of the following features. The globally unique sequence identifiers may be monotonically increasing numbers assigned by a sequencer for the event stream according to a time received by the sequencer. The method may further include adding a time stamp to each of the sequenced messages with the sequencer. The globally unique sequence identifiers may include monotonically increasing sequence numbers assigned according to an arrival at a sequencer. The method may further include, in response to the one of the sequenced messages containing the second version number, retrieving the different version of the application from a code repository. The method may further include, in response to the one of the sequenced messages containing the second version number, retrieving the different version of the application from the event stream. The method may further comprise including a corresponding one of the first version number or the second version number in each of the unsequenced messages from the application indicating a source application version associated with each of the unsequenced messages. The method may further include executing a plurality of instances of the application at a plurality of venues coupled in a communicating relationship with the event stream. The method may further include digitally signing each of the unsequenced messages before writing to the event stream. The method may further include filtering the sequenced messages to remove one or more of the sequenced messages from a local queue for the application. The application may process the sequenced messages with an order of execution determined by the globally unique sequence identifiers. The method may further include processing one of the sequenced messages having a reference identifier preceding a first globally unique sequence identifier of the sequenced messages containing the second version number with a version of the application corresponding to the first version number. The method may further include sequencing messages on the event stream by: reading unsequenced messages from the event stream, applying a time stamp to each of the unsequenced messages corresponding to a time received at a sequencer, arranging the unsequenced messages in a chronological order according to the time received at the sequencer, assigning monotonically increasing sequence numbers to the unsequenced messages according to the chronological order, and writing the unsequenced messages to the even stream using the monotonically increasing sequence numbers as the globally unique sequence identifiers. The method may further include processing an inbound message received by the application using an updated version of the application based on all revisions to the application up to and including any revisions corresponding to a reference sequence number for the inbound message. The method may further include processing an outbound message written from the application using an updated version of the application based on all revisions to the application up to and including any revisions corresponding to a reference sequence number for the outbound message.

In an aspect, a computer program product disclosed herein includes computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, updates application logic in a processing system by performing the steps of: reading sequenced messages from an event stream, the sequenced messages including sequence numbers; processing the sequenced messages with an application identified by a first version number; writing unsequenced messages from the application to the event stream without sequence numbers; in response to identifying one of the sequenced messages containing a second version number for the application, retrieving a different version of the application corresponding to the second version number and replacing the application with the different version of the application; and processing one or more of the sequenced messages having a reference sequence number greater than or equal to the one of the sequenced messages containing the second version number according to the different version of the application.

Implementations may include one or more of the following features. The computer program product may further include code that sequences messages on the event stream by performing the steps of: reading unsequenced messages from the event stream, applying a time stamp to each of the unsequenced messages corresponding to a time received at a sequencer, arranging the unsequenced messages in a chronological order according to the time received at the sequencer, assigning monotonically increasing sequence numbers to the unsequenced messages according to the chronological order, and writing the unsequenced messages to the event stream as sequenced messages using the monotonically increasing sequence numbers as the sequence numbers. The sequence numbers may be monotonically increasing numbers assigned by a sequencer for the event stream according to a time received by the sequencer, and each of the sequence numbers may be a globally unique sequence number within the event stream. The computer program product may further include code that performs the step of processing an inbound message received by the application using an updated version of the application based on all revisions to the application up to and including any revisions corresponding to a reference sequence number for the inbound message. The computer program product may further include code that performs the step of processing an outbound message written from the application using an updated version of the application based on all revisions to the application up to and including any revisions corresponding to a reference sequence number for the outbound message.

In an aspect, a computing system disclosed herein includes: an event stream; a sequencer that receives unsequenced messages from users of the event stream, arranges the unsequenced messages into sequenced messages having a global sequence in the event stream, and publishes the sequenced messages to the event stream; and an application coupled in a communicating relationship with the event stream and the sequencer, the application including a writer configured to publish one or more of the unsequenced messages, a reader to read one or more of the sequenced messages from the event stream, and a processing engine configured to process messages by at least one of processing one or more of the sequenced messages received on the event stream and creating one or more of the unsequenced messages for writing to the event stream, the application responsive to an update in the version by sending a second message to the event stream containing an updated version number for the application corresponding to an updated version of the processing engine.

Implementations may include one or more of the following features. The application may respond to the updated version number for the application received on the event stream by updating the processing engine of the application to the updated version. The application may be updated out-of-band from the event stream.

In an aspect, a computing system disclosed herein includes: an event stream; a sequencer that receives unsequenced messages from users of the event stream, arranges the unsequenced messages into sequenced messages having a global sequence in the event stream, and publishes the sequenced messages to the event stream; and an application coupled in a communicating relationship with the event stream and the sequencer, the application including a writer configured to publish one or more of the unsequenced messages, a reader to read one or more of the sequenced messages from the event stream, and a processing engine configured to process messages by at least one of processing one or more of the sequenced messages received on the event stream and creating one or more of the unsequenced messages for writing to the event stream, the application responsive to an update message received on the event stream and containing an updated version number for the processing engine by updating the processing engine to the updated version.

