<jupyter_start><jupyter_text>Continuous Analytics and Machine Learning over Streaming DataStreaming technologies provide you with the tools to collect, process, and analyze data streams in real time. AWS offers a wide range of streaming technology options including Amazon Managed Streaming for Apache Kafka (Amazon MSK), and the [Amazon Kinesis](https://aws.amazon.com/kinesis/) family of services. With Kinesis Data Firehose, you can prepare and load the data continuously to a destination of your choice. With Kinesis Data Analytics, you can process and analyze the data as it arrives. And with Kinesis Data Streams, you can manage the ingest of data streams for custom applications. In this section, we move from our customer reviews training dataset into a real-world scenario. Customer feedback about products appear in all of a company's social media channels, on partner websites, in customer support messages etc. We need to capture this valuable customer sentiment about our products as quickly as possible to spot trends and react fast.We will focus on analyzing a continuous stream of product review messages that we collect from all available online channels.   In a first step, we analyze the sentiment of the customer, so we can identify which customers  might need high-priority attention. Next, we run continuous streaming analytics over the incoming review messages to capture the average sentiment per product category. We visualize the continuous average sentiment in a metrics dashboard for the line of business owners. The line of business owners can now detect sentiment trends quickly,  and take action. We also calculate an anomaly score of the incoming messages to detect anomalies in the data schema or data values. In case of a rising anomaly score, we can alert the application developers in charge to investigate the root cause. As a last metric, we also calculate a continuous approximate count of the received messages. This number of online messages could be used by the digital marketing team to measure effectiveness of social media campaigns.  _Kinesis Data Firehose vs. Kinesis Data Streams_ Kinesis Data Firehose* Amazon Kinesis Data Firehose is the easiest way to load streaming data into data stores and analytics tools. * It can capture, transform, and load streaming data into Amazon S3, Amazon Redshift, Amazon Elasticsearch Service, and Splunk, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. * It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security. Kinesis Data Streams* Amazon Kinesis Data Streams enables you to build custom applications that process or analyze streaming data for specialized needs. * You can continuously add various types of data such as clickstreams, application logs, and social media to an Amazon Kinesis data stream from hundreds of thousands of sources. * Within seconds, the data will be available for your Amazon Kinesis Applications to read and process from the stream.  Ingest Streaming Data Using Kinesis Data Firehose  _Transform Data in Kinesis Data Firehose delivery stream_   _Preprocess streaming data in Kinesis Data Analytics_   Analyze Streaming Data with Kinesis Data Analytics _Calculating AVG Star Rating_   _Detect Anomalies of Streaming Data_   _Calculate Approxmimate Counts of Streaming Data_   Release Resources<jupyter_code>%%html

<p><b>Shutting down your kernel for this notebook to release resources.</b></p>
<button class="sm-command-button" data-commandlinker-command="kernelmenu:shutdown" style="display:none;">Shutdown Kernel</button>
        
<script>
try {
    els = document.getElementsByClassName("sm-command-button");
    els[0].click();
}
catch(err) {
    // NoOp
}    
</script>
%%javascript

try {
    Jupyter.notebook.save_checkpoint();
    Jupyter.notebook.session.delete();
}
catch(err) {
    // NoOp
}<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04079169683804007, 'fraction_numerical': 0.0004827419744146754, 'mean_word_length': 4.962589928057554, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '578399', 'license': 'Apache-2.0', 'n_tokens_mistral': 954, 'n_tokens_neox': 926, 'n_words': 569, 'path': '11_stream/00_Overview.ipynb', 'repo_name': 'MarcusFra/workshop', 'size': '4143'}