<commit_before>SnappyData’s streaming functionality builds on top of Spark Streaming and primarily is aimed at making it simpler to build streaming applications and integration with the built-in store. 
Here is a brief overview of [Spark streaming](Spark streaming) from the Spark Streaming guide. 

## Spark Streaming Overview
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ, Kinesis, or TCP sockets, and can be processed using complex
algorithms expressed with high-level functions like `map`, `reduce`, `join` and `window`.

Finally, processed data can be pushed out to filesystems, databases,
and live dashboards. In fact, you can apply Spark's [machine learning](mllib-guide.html) and [graph processing](graphx-programming-guide.html) algorithms on data streams.



<p style="text-align: center;">
  <img
    src="http://spark.apache.org/docs/latest/img/streaming-arch.png"
    title="Spark Streaming architecture"
    alt="Spark Streaming"
    width="70%"
  />
</p>

Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.

<p style="text-align: center;">
 <img src="http://spark.apache.org/docs/latest/img/streaming-flow.png"
 title="Spark Streaming data flow"
 alt="Spark Streaming"
 width="70%" />
 </p>
 
 Spark Streaming provides a high-level abstraction called *discretized stream* or *DStream*,
which represents a continuous stream of data. DStreams can be created either from input data
streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level
operations on other DStreams. Internally, a DStream is represented as a sequence of
[RDDs](api/scala/index.html#org.apache.spark.rdd.RDD).


Additional details on the Spark Streaming concepts and programming is covered [here](Spark Streaming).

## SnappyData Streaming extensions over Spark
We offer the following enhancements over Spark Streaming : 

1. __Manage Streams declaratively__: Similar to SQL Tables, Streams can be defined declaratively from any SQL client and managed as Tables in the persistent system catalog of SnappyStore. The declarative language follows the SQL language and provides access to the any of the Spark Streaming streaming adapters such as Kafka or file input streams. Raw tuples arriving can be transformed into a proper structure through pluggable transformers providing the desired flexibility for custom filtering or type conversions. 

2. __SQL based stream processing__: With streams visible as Tables they can be joined with other streams or resident tables (reference data, history, etc). Essentially, the entire SQL language can be used to analyze distributed streams. 

3. __Continuous queries and time windows__: Similar to popular stream processing products, applications can register “continuous” queries on streams. By default, spark streaming emits batches once every second and any registered queries would be executed each time a batch is emitted. To support arbitrary time ranges, we extend the standard SQL to be able to specify the time window for the query. 

4. __OLAP optimizations__: By integrating and collocating stream processing with our hybrid in-memory storage engine, we leverage our optimizer and column store for expensive scans and aggregations, while providing fast key-based operations with our row store.

5. __Reduced shuffling through co-partitioning__: With SnappyData, the partitioning key used by the input queue (e.g., for Kafka sources), the stream processor and the underlying store can all be the same. This dramatically reduces the need to shuffle records.

6. __Approximate stream analytics__: When the volumes are too high, a stream can be summarized using various forms of samples and sketches to enable fast time series analytics. This is particularly useful when applications are interested in trending patterns, for instance, rendering a set of trend lines in real time on user displays.


## Working with stream tables
SnappyData supports creation of stream tables from Twitter, Kafka, Files, Sockets sources. For example to create a stream table using kafka source : 
<commit_msg>streaming docs changes<commit_after>SnappyData’s streaming functionality builds on top of Spark Streaming and primarily is aimed at making it simpler to build streaming applications and integration with the built-in store. 
Here is a brief overview of [Spark streaming](Spark streaming) from the Spark Streaming guide. 

## Spark Streaming Overview
Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ, Kinesis, or TCP sockets, and can be processed using complex
algorithms expressed with high-level functions like `map`, `reduce`, `join` and `window`.

Finally, processed data can be pushed out to filesystems, databases,
and live dashboards. In fact, you can apply Spark's [machine learning](mllib-guide.html) and [graph processing](graphx-programming-guide.html) algorithms on data streams.



<p style="text-align: center;">
  <img
    src="http://spark.apache.org/docs/latest/img/streaming-arch.png"
    title="Spark Streaming architecture"
    alt="Spark Streaming"
    width="70%"
  />
</p>

Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.

<p style="text-align: center;">
 <img src="http://spark.apache.org/docs/latest/img/streaming-flow.png"
 title="Spark Streaming data flow"
 alt="Spark Streaming"
 width="70%" />
 </p>
 
 Spark Streaming provides a high-level abstraction called *discretized stream* or *DStream*,
which represents a continuous stream of data. DStreams can be created either from input data
streams from sources such as Kafka, Flume, and Kinesis, or by applying high-level
operations on other DStreams. Internally, a DStream is represented as a sequence of
[RDDs](api/scala/index.html#org.apache.spark.rdd.RDD).


Additional details on the Spark Streaming concepts and programming is covered [here](Spark Streaming).

## SnappyData Streaming extensions over Spark
We offer the following enhancements over Spark Streaming : 

1. __Manage Streams declaratively__: Similar to SQL Tables, Streams can be defined declaratively from any SQL client and managed as Tables in the persistent system catalog of SnappyStore. The declarative language follows the SQL language and provides access to the any of the Spark Streaming streaming adapters such as Kafka or file input streams. Raw tuples arriving can be transformed into a proper structure through pluggable transformers providing the desired flexibility for custom filtering or type conversions. 

2. __SQL based stream processing__: With streams visible as Tables they can be joined with other streams or resident tables (reference data, history, etc). Essentially, the entire SQL language can be used to analyze distributed streams. 

3. __Continuous queries and time windows__: Similar to popular stream processing products, applications can register “continuous” queries on streams. By default, spark streaming emits batches once every second and any registered queries would be executed each time a batch is emitted. To support arbitrary time ranges, we extend the standard SQL to be able to specify the time window for the query. 

4. __OLAP optimizations__: By integrating and collocating stream processing with our hybrid in-memory storage engine, we leverage our optimizer and column store for expensive scans and aggregations, while providing fast key-based operations with our row store.

5. __Reduced shuffling through co-partitioning__: With SnappyData, the partitioning key used by the input queue (e.g., for Kafka sources), the stream processor and the underlying store can all be the same. This dramatically reduces the need to shuffle records.

6. __Approximate stream analytics__: When the volumes are too high, a stream can be summarized using various forms of samples and sketches to enable fast time series analytics. This is particularly useful when applications are interested in trending patterns, for instance, rendering a set of trend lines in real time on user displays.


## Working with stream tables
SnappyData supports creation of stream tables from Twitter, Kafka, Files, Sockets sources.

    // DDL for creating a stream table
    CREATE STREAM TABLE [IF NOT EXISTS] table_name
    (COLUMN_DEFININTION)
    USING 'kafka_stream | file_stream | twitter_stream | socket_stream | directkafka_stream'
    OPTIONS (
     // multiple stream source specific options
    storagelevel '', 
    rowConverter '', 
    zkQuorum '',
    groupId '',
    topics '', 
    kafkaParams '',
    consumerKey '',
    consumerSecret '',
    accessToken '',
    accessTokenSecret '',
    hostname '',
    port '',
    directory ''
    )

    // DDL for dropping a stream table
    DROP TABLE [IF EXISTS] table_name

    // Initialize StreamingContext
    STREAMING INIT <batchIntervalSeconds> 

    // Start streaming
    STREAMING START

    // Stop streaming
    STREAMING STOP

For example to create a stream table using kafka source : 

{'fraction_non_alphanumeric': 0.05469977994341402, 'fraction_numerical': 0.002095777009326208, 'mean_word_length': 4.627358490566038, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1844178', 'n_tokens_mistral': 2404, 'n_tokens_neox': 2239, 'n_words': 1253}