<commit_before># A Reactive Web Crawler

A Recursive Web crawler built in Java 8 with reactive streams, async queues and AWS DynamoDB.

## Problem
Crawling Web sites is an intensive task, and it comes with some challenges.

* How to crawl sub-pages when the website hierarchy is unknown?
* How to crawl websites efficiently?
* How to manage long-running HTTP requests?

## Diagram v1
![Diagram](public/images/crawler_v1.png)

## Implementation
* A version the [Depth-first search (DFS)](https://en.wikipedia.org/wiki/Depth-first_search) algorithm has been implemented. The search starts from the seed URL and traverses the sub-pages until the pre-defined depth limit is reached, or there are no more sub-pages.
* Using [netty.io](https://netty.io/) client with non-blocking calls and multiple threads.
* If the Seed URL exists:
  * Reply with existing record.
* If the Seed URL doesn't exist:
  * Create a record in a NEW state and reply to the API Consumer.
  * Send the request to the queue.
  * The batch server picks up the request and updates the record to IN_PROGRESS.
  * Once the task is completed save the record to the database. If successful set status to COMPLETED, otherwise FAILED.

### Limitations
- [TODO] The API Consumer is not notified when the crawling process is completed [Version 2](#diagram-v2-todo).
- [TODO] In-Memory cache to reduce response time. [Version 2](#diagram-v2-todo).
- [TODO] API Security mechanisms: Request logging, throttling and auditing [Version 2](#diagram-v2-todo).
- [TODO] Automated CI/CD Pipelines [Version 2](#diagram-v2-todo).
- [TODO] UI [Version 2](#diagram-v2-todo).
- [TODO] Add contract, component, integration and performance tests [Version 2].
- Crawling depth must be defined.
- Only nodes prefixed with the Seed URL will be recursed.
> **Example:**<commit_msg>Updated README.<commit_after># A Reactive Web Crawler

A Recursive Web crawler built in Java 8 with reactive streams, async queues and AWS DynamoDB.

## Problem
Crawling Web sites is an intensive task, and it comes with some challenges.

* How to crawl sub-pages when the website hierarchy is unknown?
* How to crawl websites efficiently?
* How to manage long-running HTTP requests?

## Diagram v1
![Diagram](public/images/crawler_v1.png)

## Implementation
* A version the [Depth-first search (DFS)](https://en.wikipedia.org/wiki/Depth-first_search) algorithm has been implemented. The search starts from the seed URL and traverses the sub-pages until the pre-defined depth limit is reached, or there are no more sub-pages.
* Using [netty.io](https://netty.io/) client with non-blocking calls and multiple threads.
* If the Seed URL exists:
  * Reply with existing record.
* If the Seed URL doesn't exist:
  * Create a record in a NEW state and reply to the API Consumer.
  * Send the request to the queue.
  * The batch server picks up the request and updates the record to IN_PROGRESS.
  * Once the task is completed save the record to the database. If successful set status to COMPLETED, otherwise FAILED.

### Limitations
- [TODO] The API Consumer is not notified when the crawling process is completed [Version 2](#diagram-v2-todo).
- [TODO] In-Memory cache to reduce response time. [Version 2](#diagram-v2-todo).
- [TODO] API Security mechanisms: Request logging, throttling and auditing [Version 2](#diagram-v2-todo).
- [TODO] Automated CI/CD Pipelines [Version 2](#diagram-v2-todo).
- [TODO] UI [Version 2](#diagram-v2-todo).
- [TODO] Add contract, component, integration and performance tests [Version 2](#diagram-v2-todo).
- Crawling depth must be defined.
- Only nodes prefixed with the Seed URL will be recursed.
> **Example:**
{'fraction_non_alphanumeric': 0.0907103825136612, 'fraction_numerical': 0.007923497267759563, 'mean_word_length': 4.298118668596238, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4054783', 'n_tokens_mistral': 1103, 'n_tokens_neox': 1039, 'n_words': 482}