<commit_before># Spider Authors

### Skills

What skills do I need to author spiders?
In general, the ideal spider author is someone that's automated
website testing using [Selenium 2.0](http://www.seleniumhq.org/projects/webdriver/). More specifically:

1. "get" how HTML websites are built and how to test them using automated tools
1. basic object oriented Python 2.7.x
1. solid understanding of [Selenium 2.0](http://www.seleniumhq.org/projects/webdriver/)
1. good understanding of [XPath](http://en.wikipedia.org/wiki/XPath)

### Getting Started

Let's start with an overview of the spider development process.
In this documentation we'll use [these spiders](https://github.com/simonsdave/gaming-spiders) as examples.
Embedded in this documentation is a healthy dose of best practice
guidance as well as required practice.
Best efforts will be made to note when something is best practice.

* create a private or public repo on [github](https://github.com)
* setup the repo to produce a Python distribution with a distribution name
ending in ```-spiders``` (this naming convention is important because
it's relied upon by the ```spiders.py``` utility during spider discovery
* connect the repo to Travis and configure Travis to run a build on at least a nightly basis (so you have daily feedback on if your spiders are broken) using a [Travis Cron Job](https://docs.travis-ci.com/user/cron-jobs/) - each Travis build should:
  * run pycodestyle/flake8 on the spiders
  * create a pip installable Python package containing all spiders
  * create a docker image with all the spiders
  * test the spiders by running them inside a docker container created from the newly created docker image
  * push the newly created docker image to a docker registry
* if you know [Python](https://www.python.org), [JSON](http://www.json.org)
and [Selenium](http://www.seleniumhq.org) writing spiders is going to feel
very straightforward
  * best practice recommends creating one spider per ```.py``` file
  * each spider is a Python class derived from ```cloudfeaster.spider.Spider```
  * spiders define metadata which describes the website to be scraped and
the crawl arguments required; arguments are referred to as factors because
they are typically identifying and authenticating factors used to login
to a website on behalf of a user; metadata is expressed in a JSON document; the
JSON document is validated by [this](../cloudfeaster/jsonschemas/spider_metadata.json) [jsonschema](http://json-schema.org/)
  * spiders also supply a single ```crawl()``` method which is a Selenium script

* ```lots more to fill in here```

## Collaboration

The following outlines the recommended best practice for
how a (small) group of spiders authors can collaborate on authoring
a collection of spiders

* the group can use either the [fork & pull](https://help.github.com/articles/types-of-collaborative-development-models/#fork--pull)
or the [shared repo](https://help.github.com/articles/types-of-collaborative-development-models/#shared-repository-model)
model of collaboration although
the [shared repo model](https://help.github.com/articles/types-of-collaborative-development-models/#shared-repository-model)
has been found to be effective
* with the [shared repo model](https://help.github.com/articles/types-of-collaborative-development-models/#shared-repository-model)
each spider author works in their own topic/feature branch of the repo
* commits to the topic/feature branch will cause a Travis build to be kicked off & each spider author is expected
to monitor those builds and, as required, fix errors
* once the spider author is happy with their commits they
create a [pull request](https://help.github.com/articles/using-pull-requests/)
into the master branch from the topic/feature branch
* once the pull request has been merged, the master branch is built
including building docker images

## Continuous Spider Delivery Pipeline

We've previously reviewed the best practice recommendations
about connecting your spider github repo to
Travis, Travis building your spider docker image and
Travis pushing the image to DockerHub. If you've done this
then you've achieved your goal of creating a continuous spider
delivery pipeline. How? Before running a spider, Cloudfeaster
Services confirms it has the latest copy of your docker image
and, if required, downloads the latest/updated docker image.

## Overview of a Spider's Structure

First a spider with no factors.

```python
```

Now a spider with factors.

```python
```

## Metadata

* ...

### TTL

* Cloudfeaster Services will cache the results of running a spider
for the number of seconds defined by the ```ttl_in_seconds``` property of a
spider's metadata.
* 60 seconds is the default value for ```ttl_in_seconds```
* ```ttl_in_seconds``` must be at least 60 (screen scraping web sites with info
that often takes a bit to refresh and thus the rational for the
minimum ```ttl_in_seconds``` value of 60)

```python
class MySpider(spider.Spider):

    @classmethod
    def get_metadata(self):
        return {
            'url': 'https://example.com',
            'ttl_in_seconds': 120,
```

### Paranoia Level

* some web site owners do not like spiders crawling their web sites 
and put in place mechanisms to defend against crawling
* Cloudfeaster employees various approaches for circumventing these defenses
and it is expected that over time these approaches will evolve
* spider authors can optionally add a ```paranoia_level``` property
to a spider's metadata to describe how serious a web site owner is
about defending against crawling
* ```low``` is the default value for ```paranoia_level``` with ```medium```
and ```high``` being the other permissible values
* based on the ```paranoia_level```, Cloudfeater will select appropriate
circumventing approach with ```low``` meaning Cloudfeaster does nothing
* one word of caution - spider authors should expect that setting ```paranoia_level```
to ```high``` will cause a spider to run slower and cost more to run that setting ```paranoia_level```
to  ```low```

```python
class MySpider(spider.Spider):

    @classmethod
    def get_metadata(self):
        return {
            'url': 'https://example.com',
            'paranoia_level': 'low',
```

### Maximum Crawl Time

* by setting the ```max_crawl_time_in_seconds``` spider authors can optionally
define the maximum number of seconds needed for a spider crawl to a website 
* the default value for ```max_crawl_time_in_seconds``` is 30
* ```max_crawl_time_in_seconds``` must be at least 5 and no more than 300 (5 mins * 60 seconds / minute)
* ```max_crawl_time_in_seconds``` is important because it allows Cloudfeaster's infrastructure
to optimize the use of cloud computing resources however spider authors need to be thoughtful
when setting this value because if ```max_crawl_time_in_seconds``` is set too low Cloudfeaster's infrastructure
will kill a spider mid-crawl and if ```max_crawl_time_in_seconds``` is too high spider
per crawl prices will be higher than necessary
* in future the Cloudfeaster infrastructure will recommend values for ```max_crawl_time_in_seconds```

```python
class MySpider(spider.Spider):

    @classmethod
    def get_metadata(self):
        return {
            'url': 'https://example.com',
            'max_crawl_time_in_seconds': '60',
```

### Factor Display Names

* [locale](https://en.wikipedia.org/wiki/Locale)
* ```LANG``` environment variable which will look something like ```en_CA.UTF-8``` - take the first 2 characters
and this will be the [ISO639-2](http://www.loc.gov/standards/iso639-2/php/code_list.php) language code

```python
'factor_display_names': {
    'email': {
        'en': 'e-mail',
        'fr': 'e-mail',
        'ja': '電子メール',
    },
    'password': {
        'en': 'Password',
        'fr': 'mot de passe',
        'ja': 'パスワード',
    },
},
```
<commit_msg>fix minor errors in spider_authors.md<commit_after># Spider Authors

### Skills

What skills do I need to author spiders?
In general, the ideal spider author is someone that's automated
website testing using [Selenium 2.0](http://www.seleniumhq.org/projects/webdriver/). More specifically:

1. "get" how HTML websites are built and how to test them using automated tools
1. basic object oriented Python 2.7.x
1. solid understanding of [Selenium 2.0](http://www.seleniumhq.org/projects/webdriver/)
1. good understanding of [XPath](http://en.wikipedia.org/wiki/XPath)

### Getting Started

Let's start with an overview of the spider development process.
In this documentation we'll use [these spiders](https://github.com/simonsdave/gaming-spiders) as examples.
Embedded in this documentation is a healthy dose of best practice
guidance as well as required practice.
Best efforts will be made to note when something is best practice.

* create a private or public repo on [github](https://github.com)
* setup the repo to produce a Python distribution with a distribution name
ending in ```-spiders``` (this naming convention is important because
it's relied upon by the ```spiders.py``` utility during spider discovery
* connect the repo to Travis and configure Travis to run a build on at least a nightly basis (so you have daily feedback on if your spiders are broken) using a [Travis Cron Job](https://docs.travis-ci.com/user/cron-jobs/) - each Travis build should:
  * run pycodestyle/flake8 on the spiders
  * create a pip installable Python package containing all spiders
  * create a docker image with all the spiders
  * test the spiders by running them inside a docker container created from the newly created docker image
  * push the newly created docker image to a docker registry
* if you know [Python](https://www.python.org), [JSON](http://www.json.org)
and [Selenium](http://www.seleniumhq.org) writing spiders is going to feel
very straightforward
  * best practice recommends creating one spider per ```.py``` file
  * each spider is a Python class derived from ```cloudfeaster.spider.Spider```
  * spiders define metadata which describes the website to be scraped and
the crawl arguments required; arguments are referred to as factors because
they are typically identifying and authenticating factors used to login
to a website on behalf of a user; metadata is expressed in a JSON document; the
JSON document is validated by [this](../cloudfeaster/jsonschemas/spider_metadata.json) [jsonschema](http://json-schema.org/)
  * spiders also supply a single ```crawl()``` method which is a Selenium script

* ```lots more to fill in here```

## Collaboration

The following outlines the recommended best practice for
how a (small) group of spiders authors can collaborate on authoring
a collection of spiders

* the group can use either the [fork & pull](https://help.github.com/articles/types-of-collaborative-development-models/#fork--pull)
or the [shared repo](https://help.github.com/articles/types-of-collaborative-development-models/#shared-repository-model)
model of collaboration although
the [shared repo model](https://help.github.com/articles/types-of-collaborative-development-models/#shared-repository-model)
has been found to be effective
* with the [shared repo model](https://help.github.com/articles/types-of-collaborative-development-models/#shared-repository-model)
each spider author works in their own topic/feature branch of the repo
* commits to the topic/feature branch will cause a Travis build to be kicked off & each spider author is expected
to monitor those builds and, as required, fix errors
* once the spider author is happy with their commits they
create a [pull request](https://help.github.com/articles/using-pull-requests/)
into the master branch from the topic/feature branch
* once the pull request has been merged, the master branch is built
including building docker images

## Continuous Spider Delivery Pipeline

We've previously reviewed the best practice recommendations
about connecting your spider github repo to
Travis, Travis building your spider docker image and
Travis pushing the image to DockerHub. If you've done this
then you've achieved your goal of creating a continuous spider
delivery pipeline. How? Before running a spider, Cloudfeaster
Services confirms it has the latest copy of your docker image
and, if required, downloads the latest/updated docker image.

## Overview of a Spider's Structure

First a spider with no factors.

```python
```

Now a spider with factors.

```python
```

## Metadata

* ...

### TTL

* Cloudfeaster Services will cache the results of running a spider
for the number of seconds defined by the ```ttl_in_seconds``` property of a
spider's metadata.
* 60 seconds is the default value for ```ttl_in_seconds```
* ```ttl_in_seconds``` must be at least 60 (screen scraping web sites with info
that often takes a bit to refresh and thus the rational for the
minimum ```ttl_in_seconds``` value of 60)

```python
class MySpider(spider.Spider):

    @classmethod
    def get_metadata(self):
        return {
            'url': 'https://example.com',
            'ttl_in_seconds': 120,
        }
```

### Paranoia Level

* some web site owners do not like spiders crawling their web sites 
and put in place mechanisms to defend against crawling
* Cloudfeaster employees various approaches for circumventing these defenses
and it is expected that over time these approaches will evolve
* spider authors can optionally add a ```paranoia_level``` property
to a spider's metadata to describe how serious a web site owner is
about defending against crawling
* ```low``` is the default value for ```paranoia_level``` with ```medium```
and ```high``` being the other permissible values
* based on the ```paranoia_level```, Cloudfeater will select appropriate
circumventing approach with ```low``` meaning Cloudfeaster does nothing
* one word of caution - spider authors should expect that setting ```paranoia_level```
to ```high``` will cause a spider to run slower and cost more to run that setting ```paranoia_level```
to  ```low```

```python
class MySpider(spider.Spider):

    @classmethod
    def get_metadata(self):
        return {
            'url': 'https://example.com',
            'paranoia_level': 'low',
        }
```

### Maximum Crawl Time

* by setting the ```max_crawl_time_in_seconds``` spider authors can optionally
define the maximum number of seconds needed for a spider crawl to a website 
* the default value for ```max_crawl_time_in_seconds``` is 30
* ```max_crawl_time_in_seconds``` must be at least 5 and no more than 300 (5 mins * 60 seconds / minute)
* ```max_crawl_time_in_seconds``` is important because it allows Cloudfeaster's infrastructure
to optimize the use of cloud computing resources however spider authors need to be thoughtful
when setting this value because if ```max_crawl_time_in_seconds``` is set too low Cloudfeaster's infrastructure
will kill a spider mid-crawl and if ```max_crawl_time_in_seconds``` is too high spider
per crawl prices will be higher than necessary
* in future the Cloudfeaster infrastructure will recommend values for ```max_crawl_time_in_seconds```

```python
class MySpider(spider.Spider):

    @classmethod
    def get_metadata(self):
        return {
            'url': 'https://example.com',
            'max_crawl_time_in_seconds': 60,
        }
```

### Factor Display Names

* [locale](https://en.wikipedia.org/wiki/Locale)
* ```LANG``` environment variable which will look something like ```en_CA.UTF-8``` - take the first 2 characters
and this will be the [ISO639-2](http://www.loc.gov/standards/iso639-2/php/code_list.php) language code

```python
'factor_display_names': {
    'email': {
        'en': 'e-mail',
        'fr': 'e-mail',
        'ja': '電子メール',
    },
    'password': {
        'en': 'Password',
        'fr': '<PASSWORD>',
        'ja': 'パスワード',
    },
},
```

{'fraction_non_alphanumeric': 0.09211523881728582, 'fraction_numerical': 0.005180692443770533, 'mean_word_length': 4.396863279918173, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 26, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1166622', 'n_tokens_mistral': 4540, 'n_tokens_neox': 4185, 'n_words': 1904}