<commit_before># PokeData

[![Build Status](https://travis-ci.org/PokemonGoers/PokeData.svg?branch=develop)](https://travis-ci.org/PokemonGoers/PokeData)[![Join the chat at https://gitter.im/pokemongoers/PokeData](https://badges.gitter.im/pokemongoers/PokeData.svg)](https://gitter.im/pokemongoers/PokeData?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

## API Documentation
 A detailed documentation of the API is available [here](http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/doc/).
 
## Data Sources
 Data accessible through the API has been extracted from these sources:
  - [Twitter](https://twitter.com/)
  - [PokeSniper](https://pokesnipers.com)
  - [PokeRadar](https://www.pokeradar.io)
  - [SkipLagged](https://skiplagged.com/catch-that)
  - [PokeCrew](https://www.pokecrew.com)

# Build pokemon database and listen for pokemon sightings

# To run locally

- To run locally, mongodb is required, use 'mongod' to do so
- change the script part of package.json to the following
```
"scripts": {
    "start": "node app.js",
    "listen": "node scripts/listen.js",
    "build": "node scripts/build.js",
    "test": "node test/main.js"
  }
```
# Build pokemon database
npm run build -collection=pokemon

## Listen for PokemonSightings

- Liste for pokemon sightings

```
npm run listen -collection=<data-source-name>
```
- data-source-name - rarePokemon, pokeRadar, skiplagged, pokecrew, fastpokemap, pokezz, pokedexs, pokemap

- Listen for Twitter pokemon sightings:

```
CONSUMER_KEY=<CONSUMER_KEY> CONSUMER_SECRET=<CONSUMER_SECRET> ACCESS_TOKEN=<ACCESS_TOKEN> ACCESS_TOKEN_SECRET=<ACCESS_TOKEN_SECRET> NODE_ENV=<NODE_ENV> npm run listen -collection=twitter
```

- CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET - keys provided by twitter (https://apps.twitter.com/)

# To run in production

## Docker Build

 To build the project using Docker, follow these steps.
 
 1. Execute the following commands in the terminal.

  
  ```bash
  #!/bin/bash
  
  # Stop all containers
  docker stop $(docker ps -a -q)
  
  # Delete all containers
  docker rm $(docker ps -a -q)
  
  # Delete all images
  docker rmi $(docker images -q)
  <commit_msg>added commands to build pokeomndb and pokemong sightings in production<commit_after># PokeData

[![Build Status](https://travis-ci.org/PokemonGoers/PokeData.svg?branch=develop)](https://travis-ci.org/PokemonGoers/PokeData)[![Join the chat at https://gitter.im/pokemongoers/PokeData](https://badges.gitter.im/pokemongoers/PokeData.svg)](https://gitter.im/pokemongoers/PokeData?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

## API Documentation
 A detailed documentation of the API is available [here](http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/doc/).
 
## Data Sources
 Data accessible through the API has been extracted from these sources:
  - [Twitter](https://twitter.com/)
  - [PokeSniper](https://pokesnipers.com)
  - [PokeRadar](https://www.pokeradar.io)
  - [SkipLagged](https://skiplagged.com/catch-that)
  - [PokeCrew](https://www.pokecrew.com)

# Build pokemon database and listen for pokemon sightings

# To run locally

- To run locally, mongodb is required, use 'mongod' to do so
- change the script part of package.json to the following
```
"scripts": {
    "start" : "node app.js",
    "listen": "node scripts/listen.js",
    "build" : "node scripts/build.js",
    "test"  : "node test/main.js"
  }
```
# Build pokemon database
```
npm run build -collection=pokemon
```

## Listen for PokemonSightings

- Listen for pokemon sightings

```
// data-source-name - rarePokemon, pokeRadar, skiplagged, pokecrew, fastpokemap, pokezz, pokedexs, pokemap
npm run listen -collection=<data-source-name>
```

- Listen for Twitter pokemon sightings:

```
CONSUMER_KEY=<CONSUMER_KEY> CONSUMER_SECRET=<CONSUMER_SECRET> ACCESS_TOKEN=<ACCESS_TOKEN> ACCESS_TOKEN_SECRET=<ACCESS_TOKEN_SECRET> NODE_ENV=<NODE_ENV> npm run listen -collection=twitter
```

- CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET - keys provided by twitter (https://apps.twitter.com/)

# To run in production
- To use mlab as a database, get the username, password, uri and the collection name.

# Build pokemon database

```
MLAB_USERNAME=<MLAB_USERNAME> MLAB_PASSWORD=<<PASSWORD>> MLAB_URI=<MLAB_URI> MLAB_COLLECTION=<MLAB_COLLECTION> npm run build -collection=pokemon
```
## Listen for PokemonSightings

- Listen for pokemon sightings

```
// data-source-name - rarePokemon, pokeRadar, skiplagged, pokecrew, fastpokemap, pokezz, pokedexs, pokemap
MLAB_USERNAME=<MLAB_USERNAME> MLAB_PASSWORD=<<PASSWORD>> MLAB_URI=<MLAB_URI> MLAB_COLLECTION=<MLAB_COLLECTION> npm run listen -collection=<data-source-name>
```

- Listen for Twitter pokemon sightings:

```
MLAB_USERNAME=<MLAB_USERNAME> MLAB_PASSWORD=<<PASSWORD>> MLAB_URI=<MLAB_URI> MLAB_COLLECTION=<MLAB_COLLECTION> CONSUMER_KEY=<CONSUMER_KEY> CONSUMER_SECRET=<CONSUMER_SECRET> ACCESS_TOKEN=<ACCESS_TOKEN> ACCESS_TOKEN_SECRET=<ACCESS_TOKEN_SECRET> NODE_ENV=<NODE_ENV> npm run listen -collection=twitter
```

- CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET - keys provided by twitter (https://apps.twitter.com/)

## Docker Build

 To build the project using Docker, follow these steps.
 
 1. Execute the following commands in the terminal.

  
  ```bash
  #!/bin/bash
  
  # Stop all containers
  docker stop $(docker ps -a -q)
  
  # Delete all containers
  docker rm $(docker ps -a -q)
  
  # Delete all images
  docker rmi $(docker images -q)
  
{'fraction_non_alphanumeric': 0.1265365148228489, 'fraction_numerical': 0.0036153289949385392, 'mean_word_length': 4.930332261521972, 'pattern_counts': {'":': 7, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 23, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7400741', 'n_tokens_mistral': 2091, 'n_tokens_neox': 2001, 'n_words': 450}