<commit_before>### Welcome!
Seo Server is a command line tool that runs a server that allows GoogleBot (and any other crawlers) to crawl your heavily Javascript built websites. The tool works with very little changes to your server or client side code.


### Getting started
* Add your configuration into `src/seoserver.coffee` (domain, memcached, logentries)
* Install npm dependencies <br/>
<code>npm install</code>
* Install PhantomJS <br/>
<code>npm install -g phantomjs</code>
* Start the main process on port 10300 and with default memcached conf:<br/>
<code>bin/seoserver start -p 10300</code>


### Internals
The crawler has three parts:

**lib/phantom-server.js** A small PhantomJS script for fetching the page and returning the response along with the response headers in serialized form. It can be executed via:

<code>phantomjs lib/phantom-server.js http://moviepilot.com/stories</code>

**lib/seoserver.js** A node express app responsible for accepting the requests from Googlebot, checking if there is a cached version on memcached, otherwise fetching the page via `phantom-server.js`. 

You can start it locally with:

<code>node lib/seoserver.js start</code>

And test its output with:

<code>curl -v http://localhost:10300</code>

**bin/seoserver** Forever-monitor script, for launching and monitoring the node main process.

<code>bin/seoserver start -p 10300</code>

### Credits

This code is based on a [tutorial by <NAME>](http://backbonetutorials.com/seo-for-single-page-apps/) and on https://github.com/apiengine/seoserver<commit_msg>configuration examples for nginx and varnish<commit_after>### Welcome!
Seo Server is a command line tool that runs a server that allows GoogleBot (and any other crawlers) to crawl your heavily Javascript built websites. The tool works with very little changes to your server or client side code.


### Getting started
* Add your configuration into `src/seoserver.coffee` (domain, memcached, logentries)
* Install npm dependencies <br/>
<code>npm install</code>
* Install PhantomJS <br/>
<code>npm install -g phantomjs</code>
* Start the main process on port 10300 and with default memcached conf:<br/>
<code>bin/seoserver start -p 10300</code>


### Internals
The crawler has three parts:

**lib/phantom-server.js** A small PhantomJS script for fetching the page and returning the response along with the response headers in serialized form. It can be executed via:

<code>phantomjs lib/phantom-server.js http://moviepilot.com/stories</code>

**lib/seoserver.js** A node express app responsible for accepting the requests from Googlebot, checking if there is a cached version on memcached, otherwise fetching the page via `phantom-server.js`. 

You can start it locally with:

<code>node lib/seoserver.js start</code>

And test its output with:

<code>curl -v http://localhost:10300</code>

**bin/seoserver** Forever-monitor script, for launching and monitoring the node main process.

<code>bin/seoserver start -p 10300</code>

### Nginx and Varnish configuration examples

Your webserver has to detect incoming search engine requests in to route
them to the seoserver. A quite lazy way in doing so is looking for the
string "bot" in the User-Agent-Header. In Nginx you can check the
variable $http_user_agent and set the backend similar to this:

<code>location / {
  proxy_pass  http://defaultbackend;
  if ($http_user_agent ~* bot)  {
    proxy_pass  http://seoserver;
}</code>

If you deliver a cached version of your website with a reverse proxy 
in front, you can do a similar check. A vcl example for Varnish: 

<code>if (req.http.User-Agent ~ "bot") { 
set req.http.UA-Type = "crawler"; 
} else { 
set req.http.UA-Type = "regular"; 
}</code>

### Credits

This code is based on a [tutorial by <NAME>](http://backbonetutorials.com/seo-for-single-page-apps/) and on https://github.com/apiengine/seoserver
{'fraction_non_alphanumeric': 0.09591520165460186, 'fraction_numerical': 0.010341261633919338, 'mean_word_length': 4.607246376811594, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6730114', 'n_tokens_mistral': 1204, 'n_tokens_neox': 1117, 'n_words': 476}