<commit_before># coding=utf-8
# Licensed Materials - Property of IBM
# Copyright IBM Corp. 2015,2017

"""
Python API to allow creation of streaming applications for
IBM Streams & Streaming Analytics service on Bluemix.

Overview
########

IBM® Streams is an advanced analytic platform that allows user-developed applications to quickly ingest,
analyze and correlate information as it arrives from thousands of real-time sources.
Streams can handle very high data throughput rates, millions of events or messages per second.
With this API Python developers can build streaming applications that can be executed using IBM Streams,
including the processing being distributed across multiple computing resources (hosts or machines) for scalability.
"""

from __future__ import unicode_literals
from __future__ import print_function
from __future__ import division
from __future__ import absolute_import
from builtins import super
from builtins import range
try:
    from future import standard_library
    standard_library.install_aliases()
except (ImportError,NameError):<commit_msg>Add more doc<commit_after># coding=utf-8
# Licensed Materials - Property of IBM
# Copyright IBM Corp. 2015,2017

"""
Python API to allow creation of streaming applications for
IBM Streams & Streaming Analytics service on Bluemix.

Overview
########

IBM® Streams is an advanced analytic platform that allows user-developed
applications to quickly ingest, analyze and correlate information as it
arrives from thousands of real-time sources.
Streams can handle very high data throughput rates, millions of events
or messages per second.

With this API Python developers can build streaming applications
that can be executed using IBM Streams, including the processing
being distributed across multiple computing resources
(hosts or machines) for scalability.

Topology
########

A Topology declare a graph of *streams* and *operations* against
tuples (data items) on those streams.

After being declared, a Topology is submitted to be compiled into
a Streams application bundle (sab file) and then executed.
The sab file is a self contained bundle that can be executed
in a distributed Streams instance either using the Streaming
Analytics service on IBM Bluemix cloud platform or an on-premise
IBM Streams installation.

The compilation step invokes the Streams compiler to produce a bundle.
This effectively, from a Python point of view, produces a runnable
version of the Python topology that includes generated application
specific Python C extensions to optimize performance.

The Streams runtime distributes the application's operations
across the resources available in the instance.

Stream
######

A stream is a DEFINITION.

A stream has a schema that defines the type of each tuple on the stream.
The schema for a Python Topology is either:

* Untyped - A tuple may be any Python object. This is the default.
* String - Each tuple is a Unicode string.
* Binary - Each tuple is a blob.
* Json - Each tuple is a Python dict that can be expressed as a JSON object.
* Structured - A stream that is an ordered list of attributes, with each attribute having a fixed type (e.g. float64 or int32) and a name.

Stream operations
#################

A stream is processed to produce zero or more transformed streams,
such as filtering a stream to drop unwanted tuples, producing a stream
that only contains the required tuples.

An operation on a stream can be:


* A Python lambda function. The lambda is called for each tuple on a stream, passing the tuple.
* A Python function. The function is called for each tuple on a stream, passing the tuple.
* An instance of a Python callable class. The instance is called for each tuple on the stream, passing the tuple.  Use of an instance allows the operation to be stateful by maintaining state in instance attributes across invocations.
* An invocation of an SPL operator.
"""

from __future__ import unicode_literals
from __future__ import print_function
from __future__ import division
from __future__ import absolute_import
from builtins import super
from builtins import range
try:
    from future import standard_library
    standard_library.install_aliases()
except (ImportError,NameError):
{'fraction_non_alphanumeric': 0.04382847666429756, 'fraction_numerical': 0.005212035062781332, 'mean_word_length': 4.896648044692737, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6722768', 'n_tokens_mistral': 1033, 'n_tokens_neox': 972, 'n_words': 594}