<commit_before># TweetSets
[![DOI](https://zenodo.org/badge/91485768.svg)](https://zenodo.org/badge/latestdoi/91485768)

Twitter datasets for research and archiving.

* Create your own Twitter dataset from existing datasets.
* Conforms with Twitter policies.

TweetSets allows users to (1) select from existing datasets; (2) limit the dataset by querying on keywords, hashtags, 
and other parameters; (3) generate and download dataset derivatives such as the list of tweet ids and mention nodes/edges.

## Modes
TweetSets can be run in different modes. The modes determine which datasets are available and what type of dataset
derivates can be generated.

* public mode: Source datasets that are marked as local only are excluded. Dataset derivates that include the text of the
tweet cannot be generated.
* local mode: All source datasets are included, including those that are marked as local only. All dataset derivatives 
can be generated, including those that include the text of the tweet.
* both mode: For configured network IP ranges, the user is placed in local mode. Otherwise, the user is placed in public
mode.

These modes allow conforming with the Twitter policy that prohibits sharing complete tweets with 3rd parties.

Modes are configured in the `.env` file as described below.

## Installing
### Prerequisites
* Docker
* Docker-compose
* Set `vm_max_map_count` as described in the [ElasticSearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html).

### Installation
1. Create data directories on a volume with adequate storage:

        mkdir -p /tweetset_data/redis
        mkdir -p /tweetset_data/datasets
        mkdir -p /tweetset_data/elasticsearch/esdata1
        mkdir -p /tweetset_data/elasticsearch/esdata2
        chown -R 1000:1000 /tweetset_data/elasticsearch

Note:
* Create an `esdata<number>` directory for each ElasticSearch container.
* On OS X, the `redis` and `esdata<number>` directories must be `ugo+rwx`.
2. Clone or download this repository:

        git clone https://github.com/justinlittman/TweetSets.git
        
3. Change to the `docker` directory:
   
        cd docker
4. Copy the example docker files:

        cp example.docker-compose.yml docker-compose.yml
        cp example.env .env

5. Edit `.env`. This file is annotated to help you select appropriate values.
6. Create `dataset_list_msg.txt`. The contents of this file will be displayed on the dataset list page. It can 
be used to list other datasets that are available, but not yet loaded. If leaving the file empty then:

        touch dataset_list_msg.txt

7. Bring up the containers:

        docker-compose up -d

For HTTPS support, uncomment and configure the nginx-proxy container in `docker-compose.yml`.

### Cluster installation
Clusters must have at least a primary node and two additional nodes.

#### Primary node
1. Create data directories on a volume with adequate storage:

        mkdir -p /tweetset_data/redis
        mkdir -p /tweetset_data/datasets
        mkdir -p /tweetset_data/elasticsearch
        chown -R 1000:1000 /tweetset_data/elasticsearch

2. Clone or download this repository:

        git clone https://github.com/justinlittman/TweetSets.git
        
3. Change to the `docker` directory:
   
        cd docker
4. Copy the example docker files:

        cp example.cluster-primary.docker-compose.yml docker-compose.yml
        cp example.env .env

5. Edit `.env`. This file is annotated to help you select appropriate values.
6. Create `dataset_list_msg.txt`. The contents of this file will be displayed on the dataset list page. It can 
be used to list other datasets that are available, but not yet loaded. If leaving the file empty then:

        touch dataset_list_msg.txt

7. Bring up the containers:

        docker-compose up -d

For HTTPS support, uncomment and configure the nginx-proxy container in `docker-compose.yml`.

#### Cluster node
1. Create data directories on a volume with adequate storage:

        mkdir -p /tweetset_data/elasticsearch
        chown -R 1000:1000 /tweetset_data/elasticsearch

2. Clone or download this repository:

        git clone https://github.com/justinlittman/TweetSets.git
        
3. Change to the `docker` directory:
   
        cd docker
4. Copy the example docker files:

        cp example.cluster-node.docker-compose.yml docker-compose.yml
        cp example.cluster-node.env .env

5. Edit `.env`. This file is annotated to help you select appropriate values. Note that 2 cluster nodes
must have `MASTER` set to `true`.
6. Bring up the containers:

        docker-compose up -d


## Loading a source dataset
### Prepping the source dataset
1. Create a dataset directory within the dataset filepath configured in your `.env`.
2. Place tweet files in the directory. The tweet files can be line-oriented JSON (.json) or gzip compressed
line-oriented JSON (.json.gz).
3. Create a dataset description file in the directory named `dataset.json`. See `example.dataset.json` for
the format of the file.

### Loading
1. Start and connect to a loader container:

        docker-compose run --rm loader /bin/bash
2. Invoke the loader:

        python tweetset_loader.py create /dataset/path/to
        
To see other loader commands:

        python tweetset_loader.py

Note that tweets are never added to an existing index. When using the `reload` command, a new index is created
for a dataset that replaces the existing index. The new index replaces the old index only after the new index
has been created, so user's are not effected by reloading.

### Loading with Apache Spark
When using the Spark loader, the dataset files must be located at the dataset filepath
on all nodes (e.g., by having separate copies or using a network share such as [NFS](https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-16-04)).

In general, using Spark withing Docker is tricky because the Spark driver, Spark master, and Spark
nodes all need to be able to communicate and the ports are dynamically selected. (Some of the ports
can be fixed, but supporting multiple simultaneous loaders requires leaving some dynamic.) This
doesn't play well with Docker's port mapping, since the hostnames and ports that Spark advertises internally
must match what is available through Docker. Further complicating this is that host networking (which is
used to support the dynamic ports) does not work correctly on Mac.

#### Cluster mode
1. Start and connect to a loader container:

        docker-compose -f loader.docker-compose.yml run --rm loader /bin/bash
2. Invoke the loader:

        spark-submit \
        --jars elasticsearch-hadoop.jar \
        --master spark://$SPARK_MASTER_HOST:7101 \
        --py-files dist/TweetSets-0.1-py3.6.egg,dependencies.zip \
        --conf spark.driver.bindAddress=0.0.0.0 \
        --conf spark.driver.host=$SPARK_DRIVER_HOST \
        tweetset_loader.py spark-create /dataset/path/to

## Kibana
Elastic's [Kibana](https://www.elastic.co/products/kibana) is a general-purpose framework for exploring, 
analyzing, and visualizing data. Since the tweets are already indexed in ElasticSearch, they are ready
to be used from Kibana.
<commit_msg>updating install instructions<commit_after># TweetSets
[![DOI](https://zenodo.org/badge/91485768.svg)](https://zenodo.org/badge/latestdoi/91485768)

Twitter datasets for research and archiving.

* Create your own Twitter dataset from existing datasets.
* Conforms with Twitter policies.

TweetSets allows users to (1) select from existing datasets; (2) limit the dataset by querying on keywords, hashtags, 
and other parameters; (3) generate and download dataset derivatives such as the list of tweet ids and mention nodes/edges.

## Modes
TweetSets can be run in different modes. The modes determine which datasets are available and what type of dataset
derivates can be generated.

* public mode: Source datasets that are marked as local only are excluded. Dataset derivates that include the text of the
tweet cannot be generated.
* local mode: All source datasets are included, including those that are marked as local only. All dataset derivatives 
can be generated, including those that include the text of the tweet.
* both mode: For configured network IP ranges, the user is placed in local mode. Otherwise, the user is placed in public
mode.

These modes allow conforming with the Twitter policy that prohibits sharing complete tweets with 3rd parties.

Modes are configured in the `.env` file as described below.

## Installing
### Prerequisites
* Docker
* Docker-compose
* Set `vm_max_map_count` as described in the [ElasticSearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html).

### Installation
1. Create data directories on a volume with adequate storage:

        mkdir -p /tweetset_data/redis
        mkdir -p /tweetset_data/datasets
        mkdir -p /tweetset_data/elasticsearch/esdata1
        mkdir -p /tweetset_data/elasticsearch/esdata2
        chown -R 1000:1000 /tweetset_data/elasticsearch

Note:
* Create an `esdata<number>` directory for each ElasticSearch container.
* On OS X, the `redis` and `esdata<number>` directories must be `ugo+rwx`.
2. Clone or download this repository:

        git clone https://github.com/justinlittman/TweetSets.git
        
3. Change to the `docker` directory:
   
        cd docker
4. Copy the example docker files:

        cp example.docker-compose.yml docker-compose.yml
        cp example.env .env

5. Edit `.env`. This file is annotated to help you select appropriate values.
6. Create `dataset_list_msg.txt`. The contents of this file will be displayed on the dataset list page. It can 
be used to list other datasets that are available, but not yet loaded. If leaving the file empty then:

        touch dataset_list_msg.txt

7. Bring up the containers:

        docker-compose up -d

For HTTPS support, uncomment and configure the nginx-proxy container in `docker-compose.yml`.

### Cluster installation
Clusters must have at least a primary node and two additional nodes.

#### Primary node
1. Create data directories on a volume with adequate storage:

        mkdir -p /tweetset_data/redis
        mkdir -p /tweetset_data/datasets
        mkdir -p /tweetset_data/elasticsearch
        chown -R 1000:1000 /tweetset_data/elasticsearch

2. Clone or download this repository:

        git clone https://github.com/justinlittman/TweetSets.git
        
3. Change to the `docker` directory:
   
        cd docker
4. Copy the example docker files:

        cp example.cluster-primary.docker-compose.yml docker-compose.yml
        cp example.env .env

5. Edit `.env`. This file is annotated to help you select appropriate values.
6. Create `dataset_list_msg.txt`. The contents of this file will be displayed on the dataset list page. It can 
be used to list other datasets that are available, but not yet loaded. If leaving the file empty then:

        touch dataset_list_msg.txt

7. Bring up the containers:

        docker-compose up -d

For HTTPS support, uncomment and configure the nginx-proxy container in `docker-compose.yml`.

#### Cluster node
1. Create data directories on a volume with adequate storage:

        mkdir -p /tweetset_data/elasticsearch
        chown -R 1000:1000 /tweetset_data/elasticsearch

2. Clone or download this repository:

        git clone https://github.com/justinlittman/TweetSets.git
        
3. Change to the `docker` directory:
   
        cd docker
4. Copy the example docker files:

        cp example.cluster-node.docker-compose.yml docker-compose.yml
        cp example.cluster-node.env .env

5. Edit `.env`. This file is annotated to help you select appropriate values. Note that 2 cluster nodes
must have `MASTER` set to `true`.
6. Bring up the containers:

        docker-compose up -d


## Loading a source dataset
### Prepping the source dataset
1. Create a dataset directory within the dataset filepath configured in your `.env`.
2. Place tweet files in the directory. The tweet files can be line-oriented JSON (.json) or gzip compressed
line-oriented JSON (.json.gz).
3. Create a dataset description file in the directory named `dataset.json`. See `example.dataset.json` for
the format of the file.

### Loading
Use this method when Elasticsearch is on the same machine as TweetSets (non-cluster option), or for otherwise loading without using Spark.
1. Start and connect to a loader container:

        docker-compose run --rm loader /bin/bash
2. Invoke the loader:

        python tweetset_loader.py create /dataset/path/to
        
To see other loader commands:

        python tweetset_loader.py

Note that tweets are never added to an existing index. When using the `reload` command, a new index is created
for a dataset that replaces the existing index. The new index replaces the old index only after the new index
has been created, so users are not affected by reloading.

### Loading with Apache Spark
When using the Spark loader, the dataset files must be located at the dataset filepath on all nodes 
(e.g., by having separate copies or using a network share such as [NFS](https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-16-04)).

In general, using Spark withing Docker is tricky because the Spark driver, Spark master, and Spark
nodes all need to be able to communicate and the ports are dynamically selected. (Some of the ports
can be fixed, but supporting multiple simultaneous loaders requires leaving some dynamic.) This
doesn't play well with Docker's port mapping, since the hostnames and ports that Spark advertises internally
must match what is available through Docker. Further complicating this is that host networking (which is
used to support the dynamic ports) does not work correctly on Mac. Use the regular loader rather than the Spark
loader Elasticsearch is on the same machine as TweetSets (e.g., in a small development environment, not a cluster). 

#### Cluster mode
1. Start and connect to a loader container:

        docker-compose -f loader.docker-compose.yml run --rm loader /bin/bash
2. Invoke the loader:

        spark-submit \
        --jars elasticsearch-hadoop.jar \
        --master spark://$SPARK_MASTER_HOST:7101 \
        --py-files dist/TweetSets-0.1-py3.6.egg,dependencies.zip \
        --conf spark.driver.bindAddress=0.0.0.0 \
        --conf spark.driver.host=$SPARK_DRIVER_HOST \
        tweetset_loader.py spark-create /dataset/path/to

## Kibana
Elastic's [Kibana](https://www.elastic.co/products/kibana) is a general-purpose framework for exploring, 
analyzing, and visualizing data. Since the tweets are already indexed in ElasticSearch, they are ready
to be used from Kibana.

{'fraction_non_alphanumeric': 0.06939827691472764, 'fraction_numerical': 0.012210840512855302, 'mean_word_length': 3.4443774495025625, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 16, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1131100', 'n_tokens_mistral': 4269, 'n_tokens_neox': 3935, 'n_words': 1835}