<commit_before>SnapSearch-Client-Node-Heroku-Demo
==================================

This repository shows a simple Express application deployed with SnapSearch-Client-Node as an intercetor middleware.

Heroku like other PAAS systems are behind a reverse proxy, which means certain flags needs to be turned true for SnapSearch to work. These flags allow SnapSearch-Client-Node to acquire the original hostnames, ports and protocol that was used between the client and the reverse proxy.

See the `index.js` for the example source code.

The below shell snippet shows how to deploy this on your own Heroku. Note the usage of environment variables to pass the SnapSearch API credentials.

```sh
# clone the project
git clone https://github.com/SnapSearch/SnapSearch-Client-Node-Heroku-Demo.git

# get into the project's directory
cd SnapSearch-Client-Node-Heroku-Demo

# optional, run it locally
SNAPEMAIL=XXXX SNAPKEY=XXXX node index.js

# get heroku CLI and login
heroku login

# create a unique snapsearch demo application, where X is your unique random string
heroku create snapsearch-demo-X

# set your SnapSearch API Email and Key (you need to sign up on heroku) for this app
heroku config:set SNAPEMAIL=XXXX SNAPKEY=XXXX

# deploy on heroku 
git push heroku master

# visit the app with a non-robot-like useragent, you will not see the fully evaluated JS content
curl -A "Meh" https://snapsearch-demo-X.herokuapp.com/

# visit the app with a robot-like useragent, triggering the scrape, you'll see the fully evaluated JS content
curl -A "Googlebot" https://snapsearch-demo-X.herokuapp.com/

# compare the difference between the output!

# check the logs
heroku logs
```<commit_msg>added note about the special features of this demo<commit_after>SnapSearch-Client-Node-Heroku-Demo
==================================

This repository shows a simple Express application deployed with SnapSearch-Client-Node as an intercetor middleware.

Heroku like other PAAS systems are behind a reverse proxy, which means certain flags needs to be turned true for SnapSearch to work. These flags allow SnapSearch-Client-Node to acquire the original hostnames, ports and protocol that was used between the client and the reverse proxy.

See the `index.js` for the example source code.

The below shell snippet shows how to deploy this on your own Heroku. Note the usage of environment variables to pass the SnapSearch API credentials.

```sh
# clone the project
git clone https://github.com/SnapSearch/SnapSearch-Client-Node-Heroku-Demo.git

# get into the project's directory
cd SnapSearch-Client-Node-Heroku-Demo

# optional, run it locally
SNAPEMAIL=XXXX SNAPKEY=XXXX node index.js

# get heroku CLI and login
heroku login

# create a unique snapsearch demo application, where X is your unique random string
heroku create snapsearch-demo-X

# set your SnapSearch API Email and Key (you need to sign up on heroku) for this app
heroku config:set SNAPEMAIL=XXXX SNAPKEY=XXXX

# deploy on heroku 
git push heroku master

# visit the app with a non-robot-like useragent, you will not see the fully evaluated JS content
curl -A "Meh" https://snapsearch-demo-X.herokuapp.com/

# visit the app with a robot-like useragent, triggering the scrape, you'll see the fully evaluated JS content
curl -A "Googlebot" https://snapsearch-demo-X.herokuapp.com/

# compare the difference between the output!

# check the logs
heroku logs
```

Special Features of this Demo:

* Shows how to make the compression middleware work with SnapSearch!
* Shows how to deal with non-html resources such as sitemaps!
* Shows how to make it work in Heroku!
{'fraction_non_alphanumeric': 0.07470946319867183, 'fraction_numerical': 0.0, 'mean_word_length': 4.630841121495327, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '271365', 'n_tokens_mistral': 1013, 'n_tokens_neox': 977, 'n_words': 477}