<commit_before>.. _topology:

Problem statement & topology
============================

.. todo:: Convert the original problem statement into a doc.

.. todo:: Visualize birding topology.

Next, goto one of:

* :ref:`solo`
* :ref:`tour`
<commit_msg>Convert original problem statement into doc.<commit_after>.. _topology:

Problem statement & topology
============================

Problem Statement
-----------------

Take as input a sequence of terms and timestamps and produce a "filtered
firehose" of twitter_ activity using only `twitter's public APIs`_, without
requiring special API access to twitter or any third party.


Specifics
---------

* Input is in the format of (term, timestamp), where `term` is any string and
  `timestamp` is a date/time value in an ISO 8601 format,
  e.g. ``2015-06-25T08:00Z``.
* The motivating use-case:

  * provides this input as a Kafka_ topic
  * prefers output be sent to a Kafka topic & include full twitter API results
  * prefers the solution be implemented in Python_


Observations
------------

Twitter provides `GET search/tweets`_ to get relevant Tweets_ (status updates)
matching a specified query. Any detail not provided in the search results can
be accessed with `GET statuses/lookup`_, looking up multiple status updates in
a batched request.

The problem has potentially unbounded streams of data, which makes Storm_ a
relevant technology for the solution. Given that the motivating use-case
prefers Python with Kafka I/O, streamparse_ and pykafka_ are relavant.

Topology
--------

.. todo:: Visualize birding topology.


Other Goals
-----------

The solution should:

* Encode best practices about how to use Storm_/streamparse_ and
  Kafka_/pykafka_.
* Be fully public & open source to serve as an example project, so it should
  not depend on anything specific to a specific company/organization. Depending
  on the publicly scrutable Twitter API is, of course, okay.
* Include basic command-line tools for testing the topology with data and ways
  to configure things like Twitter authentication credentials.

Next, goto one of:

* :ref:`solo`
* :ref:`tour`

.. _twitter: https://twitter.com
.. _`twitter's public APIs`: https://api.twitter.com
.. _Kafka: http://kafka.apache.org
.. _Python: http://python.org
.. _`GET search/tweets`:
   https://dev.twitter.com/rest/reference/get/search/tweets
.. _`Tweets`: https://dev.twitter.com/overview/api/tweets
.. _`GET statuses/lookup`:
   https://dev.twitter.com/rest/reference/get/statuses/lookup
.. _Storm: http://storm.apache.org
.. _streamparse: https://github.com/Parsely/streamparse
.. _pykafka: https://github.com/Parsely/pykafka

{'fraction_non_alphanumeric': 0.12816635160680528, 'fraction_numerical': 0.0060491493383742915, 'mean_word_length': 3.7675675675675677, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '461467', 'n_tokens_mistral': 791, 'n_tokens_neox': 737, 'n_words': 299}