<commit_before>.. _topics-index:

================================
Frontera |version| documentation
================================

`Frontera`_ is a crawl frontier framework for a `Crawling System`_. :ref:`Crawl Frontier <crawl-frontier>` is
managing *when* and *what* to crawl next, and checking for *crawling goal* accomplishment.

Frontera also provides replication, sharding and isolation of all parts of Frontera-based crawler to scale and
distribute it.

Frontera contain components to allow creation of fully-operational web crawler with `Scrapy`_. Even though it was
originally designed for Scrapy, it can also be used with any other Crawling framework/system as the framework offers
a generic tool box.


Introduction
============

The purpose of this chapter is to introduce you to the concepts behind Frontera so that you can get an idea of
how it works and decide if it is suited to your needs.

.. toctree::
   :hidden:

   topics/overview
   topics/run-modes
   topics/quick-start-single
   topics/quick-start-distributed

:doc:`topics/overview`
    Understand what Frontera is and how it can help you.

:doc:`topics/run-modes`
    High level architecture and Frontera run modes.<commit_msg>frontera definition updated<commit_after>.. _topics-index:

================================
Frontera |version| documentation
================================

`Frontera`_ is a web crawling tool box, allowing to build crawlers of any scale and purpose.

`Frontera`_ provides :ref:`Crawl Frontier <crawl-frontier>` framework by managing *when* and *what* to crawl next,
and checking for *crawling goal* accomplishment.

Frontera also provides replication, sharding and isolation of all parts of Frontera-based crawler to scale and
distribute it.

Frontera contain components to allow creation of fully-operational web crawler with `Scrapy`_. Even though it was
originally designed for Scrapy, it can also be used with any other Crawling framework/system as the framework offers
a generic tool box.


Introduction
============

The purpose of this chapter is to introduce you to the concepts behind Frontera so that you can get an idea of
how it works and decide if it is suited to your needs.

.. toctree::
   :hidden:

   topics/overview
   topics/run-modes
   topics/quick-start-single
   topics/quick-start-distributed

:doc:`topics/overview`
    Understand what Frontera is and how it can help you.

:doc:`topics/run-modes`
    High level architecture and Frontera run modes.
{'fraction_non_alphanumeric': 0.11629778672032193, 'fraction_numerical': 0.0, 'mean_word_length': 4.63718820861678, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3788012', 'n_tokens_mistral': 690, 'n_tokens_neox': 658, 'n_words': 306}