<commit_before>Nutch Selenium
==============

This plugin allows you to fetch javascript pages using an existing Selenium Hub/Node set-up, while relying on the rest of the awesome Nutch stack! This allows you to

A) Leverage Nutch, a world class web crawler

B) Not have to use some paid service just to perform large-scale javascript/ajax aware web crawls

C) Not have to wait another 2 years for Nutch to patch in either the [Ajax crawler hashbang workaround](https://issues.apache.org/jira/browse/NUTCH-1323) and then, not having to patch it to get the use case of ammending the original url with the hashbang-workaround's content.

The underlying code is based on the nutch-htmlunit plugin, which was in turn based on nutch-httpclient. I also have patches to send through on nutch-htmlunit which get it working with nutch 2.2.1, so stay tuned if you want to use htmlunit for some reason.

## TODO

- Set up to handle https:// as well in Nutch

## NOTE:

- Currently, the plugin is set to retrieve only the content of `<body>` - you can change this yourself in `src/plugin/lib-selenium/src/java/org/apache/nutch/protocol/selenium/HttpWebClient.java`

## Installation (tested on Ubuntu 14.0x)

Part 1: Setting up Selenium<commit_msg>Added note about docker containers for selenium<commit_after>Nutch Selenium
==============

This plugin allows you to fetch javascript pages using an existing Selenium Hub/Node set-up, while relying on the rest of the awesome Nutch stack! This allows you to

A) Leverage Nutch, a world class web crawler

B) Not have to use some paid service just to perform large-scale javascript/ajax aware web crawls

C) Not have to wait another 2 years for Nutch to patch in either the [Ajax crawler hashbang workaround](https://issues.apache.org/jira/browse/NUTCH-1323) and then, not having to patch it to get the use case of ammending the original url with the hashbang-workaround's content.

The underlying code is based on the nutch-htmlunit plugin, which was in turn based on nutch-httpclient. I also have patches to send through on nutch-htmlunit which get it working with nutch 2.2.1, so stay tuned if you want to use htmlunit for some reason.

## TODO

- Set up to handle https:// as well in Nutch

## NOTE:

- Currently, the plugin is set to retrieve only the content of `<body>` - you can change this yourself in `src/plugin/lib-selenium/src/java/org/apache/nutch/protocol/selenium/HttpWebClient.java`

## Getting Selenium Hub/Node Started (Docker)

I've created two docker containers to help you get started: Selenium-Hub and Selenium-Node if you're comfortable with Docker.

1. https://github.com/momer/docker-selenium-hub
2. https://github.com/momer/docker-selenium-node

## Installation (tested on Ubuntu 14.0x)

Part 1: Setting up Selenium
{'fraction_non_alphanumeric': 0.07456772334293948, 'fraction_numerical': 0.009365994236311239, 'mean_word_length': 4.279467680608365, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4681287', 'n_tokens_mistral': 845, 'n_tokens_neox': 778, 'n_words': 388}