<story><title>KSQL: Open Source Streaming SQL for Apache Kafka</title><url>https://www.confluent.io/blog/ksql-open-source-streaming-sql-for-apache-kafka/</url></story><parent_chain><item><author>daddykotex</author><text>I&amp;#x27;ve had the question for a while so I&amp;#x27;ll ask it here, maybe someone can help me.&lt;p&gt;Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1.&lt;p&gt;Over time, things have changed and you have evolved your events to include some fields and deprecate others. You could do this without any downtime whatsoever by changing your events in a way that is backward compatible way.&lt;p&gt;What is the good approach to what I&amp;#x27;d call a `replay`?&lt;p&gt;When you want to replay all events, the version of your apps that will consume the events may not know about the fields that were in the event for day one.</text></item></parent_chain><comment><author>stingraycharles</author><text>As always in these types of the scenarios, the answer is: it depends. It depends on the amount of data you have. It depends upon how big the diversion from the original schema is. Etcetera.&lt;p&gt;My personal philosophy is to always leave event data at rest alone: data is immutable, you don&amp;#x27;t convert it, and you treat it like a historical artifact. You version each event, but never convert it into a new version in the actual event store. Any version upgrades that should be applied are done when the event is read; this requires automated procedures to convery any event version N to another version N + 1, but having these kind of procedures in place is good practice anyway. Some might argue that doing this every time an event is read is a waste of CPU cycles, but in my experience this far outweights possible downsides of losing the actual event stored at that time in the past, and this type of data is accessed far less frequently than new event data.</text></comment>
<story><title>KSQL: Open Source Streaming SQL for Apache Kafka</title><url>https://www.confluent.io/blog/ksql-open-source-streaming-sql-for-apache-kafka/</url></story><parent_chain><item><author>daddykotex</author><text>I&amp;#x27;ve had the question for a while so I&amp;#x27;ll ask it here, maybe someone can help me.&lt;p&gt;Suppose you modeled your domain with events and your stack is build on top of it. As stuff happens in your application, events are generated and appended to the stream. The stream is consumed by any number of consumers and awesome stuff is produced with it. The stream is persisted and you have all events starting from day 1.&lt;p&gt;Over time, things have changed and you have evolved your events to include some fields and deprecate others. You could do this without any downtime whatsoever by changing your events in a way that is backward compatible way.&lt;p&gt;What is the good approach to what I&amp;#x27;d call a `replay`?&lt;p&gt;When you want to replay all events, the version of your apps that will consume the events may not know about the fields that were in the event for day one.</text></item></parent_chain><comment><author>dm3</author><text>Here&amp;#x27;s a pretty good, even if a bit too verbose, explanation of various issues and solutions related to event versioning: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;leanpub.com&amp;#x2F;esversioning&amp;#x2F;read&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;leanpub.com&amp;#x2F;esversioning&amp;#x2F;read&lt;/a&gt;.&lt;p&gt;The text is written by Greg Young - the lead on the EventStore [1] project.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;geteventstore.com&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;geteventstore.com&amp;#x2F;&lt;/a&gt;</text></comment>