<commit_before># dnflow

An early experiment in automating a series of actions with Twitter
data for docnow.

Uses [Luigi](http://luigi.readthedocs.org/) for workflow automation.


## running it for yourself

First create your dnflow configuration file, and add your Twitter application
keys to it:

    cp dnflow.cfg.template dnflow.cfg

The current `summarize.py` is set up to collect a handful of tweets
based on a search, then execute a series of counts against it.  This
will result in one data file (the source tweets) and several count
files (with the same name under `data/` but with extensions like
`-urls`, `-hashtags` added on.

Assuming you either have an activated virtualenv or similar sandbox,
install the requirements first:
```
% pip install -r requirements
```

Start the `luigid` central scheduler, best done in another terminal:
```
% luigid
```

To test the workflow, run the following to kick it off (substituting a
search term of interest):
```
% python -m luigi --module summarize RunFlow --term lahoreblast
```<commit_msg>added note about ```HOSTNAME``` to ```README.md```, refs #16, #20<commit_after># dnflow

An early experiment in automating a series of actions with Twitter
data for docnow.

Uses [Luigi](http://luigi.readthedocs.org/) for workflow automation.


## running it for yourself

First create your dnflow configuration file, and add your Twitter application
keys to it:

    cp dnflow.cfg.template dnflow.cfg

If you are running on a non-standard HTTP port, such as the flask default,
```localhost:5000```, be sure to include the port number in the value of
```HOSTNAME```, e.g.:

    HOSTNAME = 'localhost:5000'

The current `summarize.py` is set up to collect a handful of tweets
based on a search, then execute a series of counts against it.  This
will result in one data file (the source tweets) and several count
files (with the same name under `data/` but with extensions like
`-urls`, `-hashtags` added on.

Assuming you either have an activated virtualenv or similar sandbox,
install the requirements first:
```
% pip install -r requirements
```

Start the `luigid` central scheduler, best done in another terminal:
```
% luigid
```

To test the workflow, run the following to kick it off (substituting a
search term of interest):
```
% python -m luigi --module summarize RunFlow --term lahoreblast
```
{'fraction_non_alphanumeric': 0.0851063829787234, 'fraction_numerical': 0.005106382978723404, 'mean_word_length': 4.367579908675799, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6657544', 'n_tokens_mistral': 706, 'n_tokens_neox': 657, 'n_words': 334}