Embodiments relate generally to computer database systems and computer networks, and more particularly, to methods, systems and computer readable media for remote data object publishing and/or subscribing using a multicast key-value protocol.

A need may exist for remote systems utilizing data to have access to updated data. Some conventional database systems may not offer a means to provide updated data to tables (or other data objects) at a remote system in a manner that is sufficiently timely, robust, and/or ordered for certain computing system applications.

Further, a need may exist to bring data sources from outside a computer data system into the computer data system and make the data available to a query engine. For example, data may be received from an external system such as a securities trading data source, an embedded system (e.g., an Internet of Things or IoT device, etc.). There may be a desire to seamlessly integrate data from one or more of these external data sources (which may be changing in real time, e.g. the price of a security) with native data in the query engine.

Embodiments were conceived in light of the above mentioned needs, problems and/or limitations, among other things.

Some implementations can include a system comprising a data source computing device including a first nontransitory memory containing instructions for a publishing process using a key-value protocol. The system can also include a first electronic communications channel including a first communications protocol, and a second electronic communications channel including a second communications protocol. The system can further include a query computing device coupled to the data source computing device via the first electronic communications channel and the second electronic communications channel, the query computing device including a second nontransitory memory containing instructions for a subscriber process using the key-value protocol.

The instructions for a publishing process stored in the first nontransitory memory can include instructions that, when executed by the data source computing device, cause the data source computing device to perform operations. The operations can include receiving, at the data source computing device, an indication of new data available, and updating, at the data source computing device, a sequence identifier value. The operations can also include packaging, at the data source computing device, at least a portion of the new data into a publisher message including a key-value pair and the updated sequence identifier value, and broadcasting, from the data source computing device, the publisher message via the first electronic communication channel.

The instructions for a subscriber process stored in the second nontransitory memory can include instructions that, when executed by the query computing device, cause the query computing device to perform operations. The operations can include receiving, at the query computing device, a request for a group of keys from a remote query processor, and transmitting, from the query computing device via the second electronic communications channel, a request for current key-value data from the data source computing device. The operations can also include receiving a request from a remote query processor to subscribe to a group of keys, and adding, at the query computing device, the current values to a key-value map. The operations can further include receiving, at the query computing device, a key-value protocol broadcast message, and for each key in the received keys that are in the key-value map, evaluating, at the query computing device, a sequence identifier (ID) to determine most recent data and updating the value for that key in the key-value map with the most recent data. The operations can also include generating an updated object based on the received key-value data, and propagating changes based on the updated data object to any dependent objects according to an update propagation graph of data object dependencies.

The first communications protocol can include an Internet protocol (IP) multicast protocol. The second communications protocol can include transmission control protocol (TCP).

The data source computing device can further include a data source process coupled to the multicast publishing process, one or more key log files generated by the data source process, and a log tailer process configured to access the one or more key log files.

The data source computing device can further include one or more data log files, wherein the log tailer process is configured to access the one or more data log files.

The query computing device further includes one or more remote query processors, and a shared memory accessible by the multicast subscriber process and at least one of the one or more remote query processors.

Some implementations can include a method for publishing that can include receiving, at a data source computing device, an indication of new data available, and updating, at the data source computing device, a sequence identifier value. The method can also include packaging, at the data source computing device, at least a portion of the new data into a publisher message including a key-value pair and the updated sequence identifier value, and broadcasting, from the data source computing device, the publisher message via a first electronic communication channel in a first communication protocol.

The first communication protocol can include an Internet protocol (IP) multicast protocol. The data source computing device can include a data source process coupled to the publishing process, one or more key log files generated by the data source process, and a log tailer process configured to access the one or more key log files. The data source computing device can further include one or more data log files, wherein the log tailer process is configured to access the one or more data log files.

Some implementations can include a method for subscription. The method can include receiving, at a query computing device, a request for a group of keys from a remote query processor, and transmitting, from the query computing device via a second electronic communications channel in a first communication protocol, a request for current key-value data to a data source computing device. The method can also include receiving a request from a remote query processor to subscribe to a group of keys, and adding, at the query computing device, the current values to a key-value map. The method can further include receiving, at the query computing device, a key-value protocol broadcast message, and for each key in the received keys that are in the key-value map, evaluating, at the query computing device, a sequence identifier (ID) to determine most recent data and updating the value for that key in the key-value map with the most recent data. The method can also include generating an updated object based on the received key-value data, and propagating changes based on the updated data object to any dependent objects according to an update propagation graph of data object dependencies.

The first communication protocol includes transmission control protocol (TCP). The query computing device can further include one or more remote query processors, and a shared memory accessible by the subscriber process and at least one of the one or more remote query processors.

