<commit_before># Deploying Scumblr 2.0 as a Security Tool

This repo contains everything you need to deploy an instance of Scumblr https://github.com/Netflix/Scumblr to monitor for potentially malicious cyber activity. Scumblr is a Netflix open source project that allows performing periodic searches and storing / taking actions on the identified results.  I have made a few changes and added some useful Search Providers. More on that further down.  The original setup documentation can be found here: https://github.com/Netflix/Scumblr/wiki/Setting-up-Scumblr-2.0-(New-install)

# Table of Contents
- Deploying Scumblr 2.0 as a Security Tool
	- Search Providers
- SETUP
	- Requirements
	- Follow Scumblr 2.0 Docs to build
	- Follow Sketchy Docs to build
	- Configure Scumblr to use Sketchy
	- Creating Services
        - Scumblr Service
		- Sketchy Service
	- Running Scumblr
	- Running Sketchy
- Automatic Syncing
- Configuring Search Providers
- Add keys and uncomment ones in use
	- Google Custom Search Providers
	- Facebook Search Provider
	- Twitter Search Provider
	- Pastebin Custom Search Provider
	- 4chan and 8ch Custom Search Providers
	- YouTube Search Provider
- Starting it All

## Search Providers

- 4chan
- 8ch
- Onion (Tor based) Sites
- Pastebin
- Reddit


# SETUP

## Requirements

- Ubuntu Server 16.04
- install Openssh-server if not already installed
	- $ sudo apt-get install openssh-server
- Harden your Server if you have not already done so. Good instructions here:
http://plusbryan.com/my-first-5-minutes-on-a-server-or-essential-security-for-linux-servers
	- $ sudo apt-get update
	- $ sudo apt-get -y install git libxslt-dev libxml2-dev build-essential bison openssl zlib1g libxslt1.1 libssl-dev libxslt1-dev libxml2 libffi-dev libxslt-dev libpq-dev autoconf libc6-dev libreadline6-dev zlib1g-dev libtool libsqlite3-dev libcurl3 libmagickcore-dev ruby-build libmagickwand-dev imagemagick bundler

## Follow Scumblr 2.0 Docs to build

https://github.com/Netflix/Scumblr/wiki/Setting-up-Scumblr-2.0-(New-install)

## Follow Sketchy Docs to build

https://github.com/Netflix/Sketchy/wiki

#### Configure Scumblr to use Sketchy:

The :host option can also use an IP address and/or include the port if non-standard (i.e. "192.168.10.101:3000")
```
$ vi Scumblr/config/environments/development.rb
Rails.application.routes.default_url_options[:host] = "localhost:3000"
```

```
$ vi Scumblr/config/initializers/scumblr.rb
config.sketchy_url = "http://127.0.0.1:8000/api/v1.0/capture"
config.sketchy_use_ssl = "sketchy_use_ssl" == "false" ? false : true # Does sketchy use ssl?
config.sketchy_verify_ssl = "false" ? false : true # Should scumblr verify sketchy's cert
#config.sketchy_tag_status_code = "sketchy_tag_status_code" # Add a tag indicating last status code sketchy received
#config.sketchy_access_token = "sketchy_access_token"
```

## Creating Sketchy and Scumblr Services
you can create a service for scumblr and sketchy.

### Scumblr service

$ vi controller/start_scumblr.sh
```
#!/bin/bash
# start scumblr
cd /home/johnsnow/Scumblr
nohup redis-server &>/dev/null & ../.rbenv/shims/bundle exec sidekiq -d -l log/sidekiq.log & nohup ../.rbenv/shims/bundle exec rails s -b 0.0.0.0 &>/dev/null &
```


$ vi controller/stop_scumblr.sh
```
#!/bin/bash
# Grabs and kill a process from the pidlist that has the word 'sidekiq 4.2.3 Scumblr'
ps aux | grep 'sidekiq 4.2.3 Scumblr' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'rails master -b'
ps aux | grep 'rails master -b' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'rails worker'
ps aux | grep 'rails worker' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'redis-server'
ps aux | grep 'redis-server' | awk '{print $2}' | xargs kill -9
```

$ vi controller/scumblr
```
#!/bin/bash
# Scumblr Control /etc/init.d/ script
#
# Copy this file into /etc/init.d/ then chmod +x (add execution options) it and 'update-rc.d scumblr defaults'
#

case $1 in
        start)
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/start_scumblr.sh
        ;;
        stop)
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/stop_scumblr.sh
        ;;
        restart)
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/stop_scumblr.sh
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/start_scumblr.sh
        ;;
esac
exit 0
```

Set permissions and move service
```
$ chmod a+x controller/start_scumblr.sh
$ chmod a+x controller/stop_scumblr.sh
$ chmod a+x controller/scumblr
$ sudo cp controller/scumblr /etc/init.d/

update init.d service
$ update-rc.d scumblr defaults
```

### Sketchy Service
Create sketchy service

$ vi sketchy/controller/stop_sketchy.sh
```
#!/bin/bash
# stop nginx
service nginx stop
# Grabs and kill a process from the pidlist that has the word 'sidekiq 4.2.3 Scumblr'
ps aux | grep 'supervisord -c' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'rails master -b'
ps aux | grep 'celery worker' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'rails worker'
ps aux | grep 'gunicorn sketchy:app' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'redis-server'
ps aux | grep 'celery worker' | awk '{print $2}' | xargs kill -9
```

$ vi sketchy/controller/start_sketchy.sh
```
#!/bin/bash
service nginx start
cd /home/johnsnow/sketchy
source sketchenv/bin/activate
supervisord -c supervisor/supervisord.ini
exit
```


$ vi sketchy/controller/sketchy
```
#!/bin/bash
# Sketchy Control /etc/init.d/ script
#
# Copy this file into /etc/init.d/ then chmod +x (add execution options) it and 'update-rc.d sketchy defaults'
#

case $1 in
        start)
                /bin/bash /home/johnsnow/sketchy/controller/start_sketchy.sh
        ;;<commit_msg>sketchy config update and service update<commit_after># Deploying Scumblr 2.0 as a Security Tool

This repo contains everything you need to deploy an instance of Scumblr https://github.com/Netflix/Scumblr to monitor for potentially malicious cyber activity. Scumblr is a Netflix open source project that allows performing periodic searches and storing / taking actions on the identified results.  I have made a few changes and added some useful Search Providers. More on that further down.  The original setup documentation can be found here: https://github.com/Netflix/Scumblr/wiki/Setting-up-Scumblr-2.0-(New-install)

# Table of Contents
- Deploying Scumblr 2.0 as a Security Tool
	- Search Providers
- SETUP
	- Requirements
	- Follow Scumblr 2.0 Docs to build
	- Follow Sketchy Docs to build
	- Configure Scumblr to use Sketchy
	- Creating Services
        - Scumblr Service
		- Sketchy Service
	- Running Scumblr
	- Running Sketchy
- Automatic Syncing
- Configuring Search Providers
- Add keys and uncomment ones in use
	- Google Custom Search Providers
	- Facebook Search Provider
	- Twitter Search Provider
	- Pastebin Custom Search Provider
	- 4chan and 8ch Custom Search Providers
	- YouTube Search Provider
- Starting it All

## Search Providers

- 4chan
- 8ch
- Onion (Tor based) Sites
- Pastebin
- Reddit


# SETUP

## Requirements

- Ubuntu Server 16.04
- install Openssh-server if not already installed
	- $ sudo apt-get install openssh-server
- Harden your Server if you have not already done so. Good instructions here:
http://plusbryan.com/my-first-5-minutes-on-a-server-or-essential-security-for-linux-servers
	- $ sudo apt-get update
	- $ sudo apt-get -y install git libxslt-dev libxml2-dev build-essential bison openssl zlib1g libxslt1.1 libssl-dev libxslt1-dev libxml2 libffi-dev libxslt-dev libpq-dev autoconf libc6-dev libreadline6-dev zlib1g-dev libtool libsqlite3-dev libcurl3 libmagickcore-dev ruby-build libmagickwand-dev imagemagick bundler

## Follow Scumblr 2.0 Docs to build

https://github.com/Netflix/Scumblr/wiki/Setting-up-Scumblr-2.0-(New-install)

## Follow Sketchy Docs to build

https://github.com/Netflix/Sketchy/wiki

#### Configure Scumblr to use Sketchy:

The :host option can also use an IP address and/or include the port if non-standard (i.e. "192.168.10.101:3000")
```
$ vi Scumblr/config/environments/development.rb
Rails.application.routes.default_url_options[:host] = "localhost:3000"
```

```
$ vi Scumblr/config/initializers/scumblr.rb
config.sketchy_url = "https://127.0.0.1/api/v1.0/capture"
config.sketchy_use_ssl = "sketchy_use_ssl" == "false" ? false : true # Does sketchy use ssl?
config.sketchy_verify_ssl = "false" ? false : true # Should scumblr verify sketchy's cert
#config.sketchy_tag_status_code = "sketchy_tag_status_code" # Add a tag indicating last status code sketchy received
#config.sketchy_access_token = "sketchy_access_token"
```

## Creating Sketchy and Scumblr Services
you can create a service for scumblr and sketchy.

### Scumblr service

$ vi controller/start_scumblr.sh
```
#!/bin/bash
# start scumblr
cd /home/johnsnow/Scumblr
nohup redis-server &>/dev/null & ../.rbenv/shims/bundle exec sidekiq -d -l log/sidekiq.log & nohup ../.rbenv/shims/bundle exec rails s -b 0.0.0.0 &>/dev/null &
```


$ vi controller/stop_scumblr.sh
```
#!/bin/bash
# Grabs and kill a process from the pidlist that has the word 'sidekiq 4.2.10 Scumblr'
ps aux | grep 'sidekiq 4.2.10 Scumblr' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'rails master -b'
ps aux | grep 'rails master -b' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'rails worker'
ps aux | grep 'rails worker' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'redis-server'
ps aux | grep 'redis-server' | awk '{print $2}' | xargs kill -9
```

$ vi controller/scumblr
```
#!/bin/bash
# Scumblr Control /etc/init.d/ script
#
# Copy this file into /etc/init.d/ then chmod +x (add execution options) it and 'update-rc.d scumblr defaults'
#

case $1 in
        start)
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/start_scumblr.sh
        ;;
        stop)
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/stop_scumblr.sh
        ;;
        restart)
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/stop_scumblr.sh
                sudo -u johnsnow /bin/bash /home/johnsnow/Scumblr/controller/start_scumblr.sh
        ;;
esac
exit 0
```

Set permissions and move service
```
$ chmod a+x controller/start_scumblr.sh
$ chmod a+x controller/stop_scumblr.sh
$ chmod a+x controller/scumblr
$ sudo cp controller/scumblr /etc/init.d/

update init.d service
$ update-rc.d scumblr defaults
```

### Sketchy Service
Create sketchy service

$ vi sketchy/controller/stop_sketchy.sh
```
#!/bin/bash
# stop nginx
service nginx stop
# Grabs and kill a process from the pidlist that has the word 'supervisord -c'
ps aux | grep 'supervisord -c' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'celery worker'
ps aux | grep 'celery worker' | awk '{print $2}' | xargs kill -9
# Grabs and kill a process from the pidlist that has the word 'gunicorn sketchy:app'
ps aux | grep 'gunicorn sketchy:app' | awk '{print $2}' | xargs kill -9
```

$ vi sketchy/controller/start_sketchy.sh
```
#!/bin/bash
service nginx start
cd /home/johnsnow/sketchy
source sketchenv/bin/activate
supervisord -c supervisor/supervisord.ini
exit
```


$ vi sketchy/controller/sketchy
```
#!/bin/bash
# Sketchy Control /etc/init.d/ script
#
# Copy this file into /etc/init.d/ then chmod +x (add execution options) it and 'update-rc.d sketchy defaults'
#

case $1 in
        start)
                /bin/bash /home/johnsnow/sketchy/controller/start_sketchy.sh
        ;;
{'fraction_non_alphanumeric': 0.10287481196724052, 'fraction_numerical': 0.015293331104796925, 'mean_word_length': 3.550190114068441, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 7, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 4}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3328845', 'n_tokens_mistral': 4395, 'n_tokens_neox': 4015, 'n_words': 1331}