



  

Swiftea-Crawler 1.0.2 documentation

 [Swiftea-Crawler](index.html#document-index) 

 master

 

* [1. Modules](index.html#document-doc_modules)

	+ [1.1. crawler main module](index.html#crawler-main-module)

	+ [1.2. statistiques module](index.html#module-stats)

* [2. Package](index.html#document-doc_packages)

	+ [2.1. swiftea\_bot.module module](index.html#module-swiftea_bot.module)

	+ [2.2. swiftea\_bot.data module](index.html#module-swiftea_bot.data)

	+ [2.3. swiftea\_bot.file\_manager module](index.html#module-swiftea_bot.file_manager)

	+ [2.4. crawling.web\_connection module](index.html#module-crawling.web_connection)

	+ [2.5. crawling.connection module](index.html#module-crawling.connection)

	+ [2.6. crawling.site\_informations module](index.html#module-crawling.site_informations)

	+ [2.7. crawling.searches module](index.html#module-crawling.searches)

	+ [2.8. crawling.parsers module](index.html#module-crawling.parsers)

	+ [2.9. database.database module](index.html#module-database.database)

	+ [2.10. database.database\_manager module](index.html#module-database.database_manager)

	+ [2.11. database.database\_swiftea module](index.html#module-database.database_swiftea)

	+ [2.12. index.index module](index.html#module-index.index)

	+ [2.13. index.inverted\_index module](index.html#module-index.inverted_index)

	+ [2.14. index.sftp\_manager module](index.html#index-sftp-manager-module)

	+ [2.15. index.sftp\_swiftea module](index.html#index-sftp-swiftea-module)

* [3. Tests](index.html#document-doc_tests)

	+ [3.1. tests.run\_tests](index.html#tests-run-tests)

	+ [3.2. tests.swiftea\_bot\_test](index.html#tests-swiftea-bot-test)

	+ [3.3. tests.crawling\_test](index.html#tests-crawling-test)

	+ [3.4. tests.database\_test](index.html#tests-database-test)

	+ [3.5. tests.index\_test](index.html#tests-index-test)

	+ [3.6. tests.crawler\_test](index.html#tests-crawler-test)

	+ [3.7. tests.global\_test](index.html#tests-global-test)

	+ [3.8. tests.test\_data](index.html#tests-test-data)

[Swiftea-Crawler](index.html#document-index)

* [Docs](index.html#document-index) »

* Swiftea-Crawler 1.0.2 documentation

* [Edit on GitHub](https://github.com/Swiftea/Swiftea-Crawler/blob/master/docs/index.rst)

---

Welcome to Swiftea-Crawler’s documentation![¶](#welcome-to-swiftea-crawler-s-documentation "Permalink to this headline")

========================================================================================================================

Swiftea-Crawler is an open source web crawler for Swiftea.

It can’t be run by contributors because it needs private\_data.py wich is not upload for obvious reason.

Modules[¶](#modules "Permalink to this headline")

-------------------------------------------------

Here are described the two executables of Swiftea-Crawler.

### crawler main module[¶](#crawler-main-module "Permalink to this headline")

### statistiques module[¶](#module-stats "Permalink to this headline")

Display stats.

`stats.``stats`(*dir\_stats='data/stats/'*)[[source]](_modules/stats.html#stats)[¶](#stats.stats "Permalink to this definition")

`stats.``compress_stats`(*filename*)[[source]](_modules/stats.html#compress_stats)[¶](#stats.compress_stats "Permalink to this definition")

`stats.``average`(*content*)[[source]](_modules/stats.html#average)[¶](#stats.average "Permalink to this definition")

Calculate average.

| Parameters: | **content** (*list*) – values |

| Returns: | average |

Package[¶](#package "Permalink to this headline")

-------------------------------------------------

These packages provide all functions and class that crawler need.

### swiftea\_bot.module module[¶](#module-swiftea_bot.module "Permalink to this headline")

Define several functions for all crawler’s class.

`swiftea_bot.module.``tell`(*message*, *error\_code=''*, *severity=1*)[[source]](_modules/swiftea_bot/module.html#tell)[¶](#swiftea_bot.module.tell "Permalink to this definition")

Manage newspaper.

Print in console that program doing and save a copy with time in event file.

| Parameters: | * **message** (*str*) – message to print and write

* **error\_code** (*int*) – (optional) error code, if given call errors() with given message

* **severity** (*int*) – 1 is default severity, -1 add 4 spaces befor message,

0 add 2 spaces befor the message, 2 uppercase and underline message.

 |

`swiftea_bot.module.``errors`(*message*, *error\_code*)[[source]](_modules/swiftea_bot/module.html#errors)[¶](#swiftea_bot.module.errors "Permalink to this definition")

Write the error report with the time in errors file.

Normaly call by tell() when a error\_code parameter is given.

| Parameters: | * **message** (*str*) – message to print and write

* **error\_code** (*int*) – error code

 |

`swiftea_bot.module.``create_dirs`()[[source]](_modules/swiftea_bot/module.html#create_dirs)[¶](#swiftea_bot.module.create_dirs "Permalink to this definition")

Manage crawler’s runing.

Test lot of things:

create config directory

create doc file if doesn’t exists

create config file if it doesn’t exists

create links directory if it doesn’t exists

create index directory if it doesn’t exists

`swiftea_bot.module.``def_links`()[[source]](_modules/swiftea_bot/module.html#def_links)[¶](#swiftea_bot.module.def_links "Permalink to this definition")

Create directory of links if it doesn’t exist

Ask to user what doing if there isn’t basic links.

Create a basic links file if user what it.

`swiftea_bot.module.``is_index`()[[source]](_modules/swiftea_bot/module.html#is_index)[¶](#swiftea_bot.module.is_index "Permalink to this definition")

Check if there is a saved inverted-index file.

| Returns: | True if there is one |

`swiftea_bot.module.``can_add_doc`(*docs*, *new\_doc*)[[source]](_modules/swiftea_bot/module.html#can_add_doc)[¶](#swiftea_bot.module.can_add_doc "Permalink to this definition")

to avoid documents duplicate, look for all url doc.

Parse self.infos of Crawler and return True if new\_doc isn’t in it.

| Parameters: | * **docs** (*list*) – the documents to check

* **new\_doc** (*dict*) – the doc to add

 |

| Returns: | True if can add the doc |

`swiftea_bot.module.``remove_duplicates`(*old\_list*)[[source]](_modules/swiftea_bot/module.html#remove_duplicates)[¶](#swiftea_bot.module.remove_duplicates "Permalink to this definition")

Remove duplicates from a list.

| Parameters: | **old\_list** (*list*) – list to clean |

| Returns: | list without duplicates |

`swiftea_bot.module.``stats_webpages`(*begining*, *end*)[[source]](_modules/swiftea_bot/module.html#stats_webpages)[¶](#swiftea_bot.module.stats_webpages "Permalink to this definition")

Write the time in second to crawl 10 webpages.

| Parameters: | * **begining** (*int*) – time before starting crawl 10 webpages

* **end** (*int*) – time after crawled 10 webpages

 |

`swiftea_bot.module.``stats_send_index`(*begining*, *end*)[[source]](_modules/swiftea_bot/module.html#stats_send_index)[¶](#swiftea_bot.module.stats_send_index "Permalink to this definition")

Time spent between two sending of index

`swiftea_bot.module.``convert_keys`(*inverted\_index*)[[source]](_modules/swiftea_bot/module.html#convert_keys)[¶](#swiftea_bot.module.convert_keys "Permalink to this definition")

Convert str words keys into int from inverted-index.

Json convert doc id key in str, must convert in int.

| Parameters: | **inverted\_index** – inverted\_index to convert |

| Tyep inverted\_index: |

|  | dict |

| Returns: | converted inverted-index |

### swiftea\_bot.data module[¶](#module-swiftea_bot.data "Permalink to this headline")

Define required data for crawler.

### swiftea\_bot.file\_manager module[¶](#module-swiftea_bot.file_manager "Permalink to this headline")

Swiftea-Crawler use lot a files. For example to manage configurations, stuck links…

Here is a class who manager files of crawler.

*class* `swiftea_bot.file_manager.``FileManager`[[source]](_modules/swiftea_bot/file_manager.html#FileManager)[¶](#swiftea_bot.file_manager.FileManager "Permalink to this definition")

File manager for Swiftea-Crawler.

Save and read links, read and write configuration variables,

read inverted-index from json file saved and from file using when send it.

Create configuration file if doesn’t exists or read it.

`check_stop_crawling`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.check_stop_crawling)[¶](#swiftea_bot.file_manager.FileManager.check_stop_crawling "Permalink to this definition")

Check if the user want to stop program.

`save_config`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.save_config)[¶](#swiftea_bot.file_manager.FileManager.save_config "Permalink to this definition")

Save all configurations in config file.

`save_links`(*links*)[[source]](_modules/swiftea_bot/file_manager.html#FileManager.save_links)[¶](#swiftea_bot.file_manager.FileManager.save_links "Permalink to this definition")

Save found links in file.

Save link in a file without doublons.

| Parameters: | **links** (*list*) – links to save |

`ckeck_size_links`(*links*)[[source]](_modules/swiftea_bot/file_manager.html#FileManager.ckeck_size_links)[¶](#swiftea_bot.file_manager.FileManager.ckeck_size_links "Permalink to this definition")

Check number of links in file.

| Parameters: | **links** (*str*) – links saved in file |

`check_size_files`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.check_size_files)[¶](#swiftea_bot.file_manager.FileManager.check_size_files "Permalink to this definition")

`get_url`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.get_url)[¶](#swiftea_bot.file_manager.FileManager.get_url "Permalink to this definition")

Get url of next webpage.

Check the size of curent reading links and increment it if over.

| Returns: | url of webpage to crawl |

`save_inverted_index`(*inverted\_index*)[[source]](_modules/swiftea_bot/file_manager.html#FileManager.save_inverted_index)[¶](#swiftea_bot.file_manager.FileManager.save_inverted_index "Permalink to this definition")

Save inverted-index in local.

Save it in a .json file when can’t send.

| Parameters: | **inverted\_index** (*dict*) – inverted-index |

`get_inverted_index`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.get_inverted_index)[¶](#swiftea_bot.file_manager.FileManager.get_inverted_index "Permalink to this definition")

Get inverted-index in local.

Call after a connxion error. Read a .json file conatin inverted-index.

Delete this file after reading.

| Returns: | inverted-index |

`read_inverted_index`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.read_inverted_index)[¶](#swiftea_bot.file_manager.FileManager.read_inverted_index "Permalink to this definition")

Get inverted-index in local.

Call after sending inverted-index without error.

Read all files created for sending inverted-index.

| Returns: | inverted-index |

`get_lists_words`()[[source]](_modules/swiftea_bot/file_manager.html#FileManager.get_lists_words)[¶](#swiftea_bot.file_manager.FileManager.get_lists_words "Permalink to this definition")

Get lists words from data

Check for dirs lists words, create it if don’t exists.

| Returns: | stopwords, badwords |

### crawling.web\_connection module[¶](#module-crawling.web_connection "Permalink to this headline")

Connection to webpage are manage with requests module.

Thoses errors are waiting for: timeout with socket module and with urllib3 mudule

and all RequestException errors.

*class* `crawling.web_connection.``WebConnection`[[source]](_modules/crawling/web_connection.html#WebConnection)[¶](#crawling.web_connection.WebConnection "Permalink to this definition")

Manage the web connection with the page to crawl.

`get_code`(*url*)[[source]](_modules/crawling/web_connection.html#WebConnection.get_code)[¶](#crawling.web_connection.WebConnection.get_code "Permalink to this definition")

Get source code of given url.

| Parameters: | **url** (*str*) – url of webpage |

| Returns: | source code, True if no take links, score and new url (redirection) |

`send_request`(*url*)[[source]](_modules/crawling/web_connection.html#WebConnection.send_request)[¶](#crawling.web_connection.WebConnection.send_request "Permalink to this definition")

`search_encoding`(*headers*, *code*)[[source]](_modules/crawling/web_connection.html#WebConnection.search_encoding)[¶](#crawling.web_connection.WebConnection.search_encoding "Permalink to this definition")

Searche encoding of webpage in source code.

If an encoding is found in source code, score is 1, but if not

score is 0 and encoding is utf-8.

| Parameters: | * **headers** (*dict*) – hearders of requests

* **code** (*str*) – source code

 |

| Returns: | encoding of webpage and it score |

`check_robots_perm`(*url*)[[source]](_modules/crawling/web_connection.html#WebConnection.check_robots_perm)[¶](#crawling.web_connection.WebConnection.check_robots_perm "Permalink to this definition")

Check robots.txt for permission.

| Parameters: | **url** (*str*) – webpage url |

| Returns: | True if can crawl |

`duplicate_content`(*request1*, *url*)[[source]](_modules/crawling/web_connection.html#WebConnection.duplicate_content)[¶](#crawling.web_connection.WebConnection.duplicate_content "Permalink to this definition")

Avoid param duplicate.

Compare source codes with params and whitout.

Return url whitout params if it’s the same content.

| Parameters: | **request** (*requests.models.Response*) – request |

| Returns: | url, source code |

### crawling.connection module[¶](#module-crawling.connection "Permalink to this headline")

Define several functions WebConnection.

`crawling.connection.``no_connection`(*url='https://github.com'*)[[source]](_modules/crawling/connection.html#no_connection)[¶](#crawling.connection.no_connection "Permalink to this definition")

Check connection.

Try to connect to swiftea website.

| Parameters: | **url** – url use by test |

| Returns: | True if no connection |

`crawling.connection.``is_nofollow`(*url*)[[source]](_modules/crawling/connection.html#is_nofollow)[¶](#crawling.connection.is_nofollow "Permalink to this definition")

Check if take links.

Search !nofollow! at the end of url, remove it if found.

| Parameters: | **url** (*str*) – webpage url |

| Returns: | True if nofollow and url |

`crawling.connection.``duplicate_content`(*code1*, *code2*)[[source]](_modules/crawling/connection.html#duplicate_content)[¶](#crawling.connection.duplicate_content "Permalink to this definition")

Compare code1 and code2.

| Parameters: | * **code1** (*str*) – first code to compare

* **code2** (*str*) – second code to compare

 |

`crawling.connection.``all_urls`(*request*)[[source]](_modules/crawling/connection.html#all_urls)[¶](#crawling.connection.all_urls "Permalink to this definition")

Return all urls from request.history.

| Parameters: | * **request** (*requests.models.Response*) – request

* **first** (*str*) – list start with the url if given

 |

| Returns: | list of redirected urls, first is the last one |

### crawling.site\_informations module[¶](#module-crawling.site_informations "Permalink to this headline")

After parse source code, data extracted must be classify and clean.

Here is a class who use the html parser and manage all results.

*class* `crawling.site_informations.``SiteInformations`[[source]](_modules/crawling/site_informations.html#SiteInformations)[¶](#crawling.site_informations.SiteInformations "Permalink to this definition")

Class to manage searches in source codes.

`set_listswords`(*stopwords*, *badwords*)[[source]](_modules/crawling/site_informations.html#SiteInformations.set_listswords)[¶](#crawling.site_informations.SiteInformations.set_listswords "Permalink to this definition")

`get_infos`(*url*, *code*, *nofollow*, *score*)[[source]](_modules/crawling/site_informations.html#SiteInformations.get_infos)[¶](#crawling.site_informations.SiteInformations.get_infos "Permalink to this definition")

Manager all searches of webpage’s informations.

| Parameters: | * **url** (*str*) – url of webpage

* **score** (*int*) – score of webpage

* **code** (*str*) – source code of webpage

* **nofollow** (*bool*) – if we take links of webpage

 |

| Returns: | links, title, description, key words, language,

score, number of words |

`detect_language`(*keywords*)[[source]](_modules/crawling/site_informations.html#SiteInformations.detect_language)[¶](#crawling.site_informations.SiteInformations.detect_language "Permalink to this definition")

Detect language of webpage if not given.

| Parameters: | **keywords** (*list*) – keywords of webpage used for detecting |

| Returns: | language found |

`clean_links`(*links*, *base\_url=None*)[[source]](_modules/crawling/site_informations.html#SiteInformations.clean_links)[¶](#crawling.site_informations.SiteInformations.clean_links "Permalink to this definition")

Clean webpage’s links: rebuild urls with base url and

remove anchors, mailto, javascript, .index.

| Parameters: | **links** (*list*) – links to clean |

| Returns: | cleanen links without duplicate |

`clean_favicon`(*favicon*, *base\_url*)[[source]](_modules/crawling/site_informations.html#SiteInformations.clean_favicon)[¶](#crawling.site_informations.SiteInformations.clean_favicon "Permalink to this definition")

Clean favicon.

| Parameters: | **favicon** (*str*) – favicon url to clean |

| Returns: | cleaned favicon |

`clean_keywords`(*dirty\_keywords*, *language*)[[source]](_modules/crawling/site_informations.html#SiteInformations.clean_keywords)[¶](#crawling.site_informations.SiteInformations.clean_keywords "Permalink to this definition")

Clean found keywords.

Delete stopwords, bad chars, two letter less word and split word1-word2

| Parameters: | **keywords** (*list*) – keywords to clean |

| Returns: | list of cleaned keywords |

`sane_search`(*keywords*, *language*, *max\_ratio=0.2*)[[source]](_modules/crawling/site_informations.html#SiteInformations.sane_search)[¶](#crawling.site_informations.SiteInformations.sane_search "Permalink to this definition")

Filter adults websites.

| Param: | keywords: webpage’s keywords |

| Pram language: | found website language |

| Returns: | True or False |

### crawling.searches module[¶](#module-crawling.searches "Permalink to this headline")

Define several functions SiteInformations.

`crawling.searches.``clean_text`(*text*)[[source]](_modules/crawling/searches.html#clean_text)[¶](#crawling.searches.clean_text "Permalink to this definition")

Clean up text by removing tabulation, blank and carriage return.

| Parameters: | **text** (*str*) – text to clean\_text |

| Returns: | cleaned text |

`crawling.searches.``get_base_url`(*url*)[[source]](_modules/crawling/searches.html#get_base_url)[¶](#crawling.searches.get_base_url "Permalink to this definition")

Get base url using urlparse.

| Parameters: | **url** (*str*) – url |

| Returns: | base url of given url |

`crawling.searches.``is_homepage`(*url*)[[source]](_modules/crawling/searches.html#is_homepage)[¶](#crawling.searches.is_homepage "Permalink to this definition")

Check if url is the homepage.

If there is only two ‘/’ and two ‘.’ if www and one otherwise.

| Parameters: | **url** (*str*) – url to check |

| Returns: | True or False |

`crawling.searches.``clean_link`(*url*, *base\_url=None*)[[source]](_modules/crawling/searches.html#clean_link)[¶](#crawling.searches.clean_link "Permalink to this definition")

Clean a link.

Rebuild url with base url, pass mailto and javascript,

remove anchors, pass if more than 5 query, pass if more than 255 chars,

remove /index.xxx, remove last /.

| Parameters: | * **url** (*str*) – links to clean

* **base\_url** – base url for rebuilding, can be None if

 |

| Returns: | cleaned link |

`crawling.searches.``capitalize`(*text*)[[source]](_modules/crawling/searches.html#capitalize)[¶](#crawling.searches.capitalize "Permalink to this definition")

Upper the first letter of given text

| Parameters: | **text** (*str*) – text |

| Returns: | text |

`crawling.searches.``stats_links`(*stats*)[[source]](_modules/crawling/searches.html#stats_links)[¶](#crawling.searches.stats_links "Permalink to this definition")

Write the number of links for statistics.

| Parameters: | **stat** (*int*) – number of list in a webpage |

### crawling.parsers module[¶](#module-crawling.parsers "Permalink to this headline")

Data of webpage are geted by the python html.parser.

Here is two parser, the first one for all informations and

the sencond one only for encoding.

*class* `crawling.parsers.``ExtractData`[[source]](_modules/crawling/parsers.html#ExtractData)[¶](#crawling.parsers.ExtractData "Permalink to this definition")

Bases: `html.parser.HTMLParser`

Html parser for extract data.

self.object : the type of text for title, description and keywords

dict(attrs).get(‘content’) : convert attrs in a dict and retrun the value

Data could be extract:

title

language

description

links with nofollow and noindex

stylesheet

favicon

keywords: h1, h2, h3, strong, em

`re_init`()[[source]](_modules/crawling/parsers.html#ExtractData.re_init)[¶](#crawling.parsers.ExtractData.re_init "Permalink to this definition")

Call when met html tag, put back all variables to default.

`handle_starttag`(*tag*, *attrs*)[[source]](_modules/crawling/parsers.html#ExtractData.handle_starttag)[¶](#crawling.parsers.ExtractData.handle_starttag "Permalink to this definition")

Call when parser met a starting tag.

| Parameters: | * **tag** (*str*) – starting tag

* **attrs** (*list*) – attributes: [(‘name’, ‘language’), (‘content’, ‘fr’)]

 |

`handle_data`(*data*)[[source]](_modules/crawling/parsers.html#ExtractData.handle_data)[¶](#crawling.parsers.ExtractData.handle_data "Permalink to this definition")

Call when parser met data.

| Parameters: | **tag** (*str*) – starting tag |

`handle_endtag`(*tag*)[[source]](_modules/crawling/parsers.html#ExtractData.handle_endtag)[¶](#crawling.parsers.ExtractData.handle_endtag "Permalink to this definition")

Call when parser met a ending tag.

| Parameters: | * **tag** (*str*) – starting tag

* **attrs** (*list*) – attributes

 |

`handle_entityref`(*name*)[[source]](_modules/crawling/parsers.html#ExtractData.handle_entityref)[¶](#crawling.parsers.ExtractData.handle_entityref "Permalink to this definition")

`handle_charref`(*name*)[[source]](_modules/crawling/parsers.html#ExtractData.handle_charref)[¶](#crawling.parsers.ExtractData.handle_charref "Permalink to this definition")

`crawling.parsers.``meta`(*attrs*)[[source]](_modules/crawling/parsers.html#meta)[¶](#crawling.parsers.meta "Permalink to this definition")

Manager searches in meat tag.

Can find:

<meta name=’description’ content=’my description’/>

<meta name=’language’ content=’en’/>

<meta http-equiv=’content-language’ content=’en’/>

| Apram attrs: | attributes of meta tag |

| Returns: | language, description, object |

`crawling.parsers.``can_append`(*url*, *rel*)[[source]](_modules/crawling/parsers.html#can_append)[¶](#crawling.parsers.can_append "Permalink to this definition")

Check rel attrs to know if crawler can take this the link.

Add !nofollow! at the end of the url if can’t follow links of url.

| Parameters: | * **url** (*str*) – url to add

* **rel** (*str*) – rel attrs in a tag

 |

| Returns: | None if can’t add it, otherwise return url |

*class* `crawling.parsers.``ExtractEncoding`[[source]](_modules/crawling/parsers.html#ExtractEncoding)[¶](#crawling.parsers.ExtractEncoding "Permalink to this definition")

Bases: `html.parser.HTMLParser`

Html parser for extract encoding from source code.

`handle_starttag`(*tag*, *attrs*)[[source]](_modules/crawling/parsers.html#ExtractEncoding.handle_starttag)[¶](#crawling.parsers.ExtractEncoding.handle_starttag "Permalink to this definition")

Call when parser met a starting tag.

| Parameters: | * **tag** (*str*) – starting tag

* **attrs** (*list*) – attributes

 |

### database.database module[¶](#module-database.database "Permalink to this headline")

Define several functions for DatabaseSwiftea.

`database.database.``url_is_secure`(*url*)[[source]](_modules/database/database.html#url_is_secure)[¶](#database.database.url_is_secure "Permalink to this definition")

Check if given url is secure (https).

| Parameters: | **url** (*str*) – url to check |

| Returns: | True if url is secure |

`database.database.``convert_secure`(*url*)[[source]](_modules/database/database.html#convert_secure)[¶](#database.database.convert_secure "Permalink to this definition")

Convert https to http and http to https.

| Parameters: | **url** (*str*) – url to convert |

| Returns: | converted url |

### database.database\_manager module[¶](#module-database.database_manager "Permalink to this headline")

*class* `database.database_manager.``DatabaseManager`(*host*, *user*, *password*, *name*)[[source]](_modules/database/database_manager.html#DatabaseManager)[¶](#database.database_manager.DatabaseManager "Permalink to this definition")

Class to manage query to Database using PyMySQL.

How to: create a subclass

result, response = self.send\_comand(command, data=tuple(), all=False)

if ‘error’ in response:

> 

> print(‘An error occured.’)

where result are data asked and response a message.

| Parameters: | * **host** (*str*) – hostname of the sftp server

* **user** (*str*) – username to use for connection

* **password** (*str*) – password to use for connection

* **name** (*str*) – name of database

 |

`set_name`(*name*)[[source]](_modules/database/database_manager.html#DatabaseManager.set_name)[¶](#database.database_manager.DatabaseManager.set_name "Permalink to this definition")

Set base name

| Parameters: | **name** (*str*) – new base name |

`connection`()[[source]](_modules/database/database_manager.html#DatabaseManager.connection)[¶](#database.database_manager.DatabaseManager.connection "Permalink to this definition")

Connect to database.

`close_connection`()[[source]](_modules/database/database_manager.html#DatabaseManager.close_connection)[¶](#database.database_manager.DatabaseManager.close_connection "Permalink to this definition")

Close database connection.

`send_command`(*command*, *data=()*, *fetchall=False*)[[source]](_modules/database/database_manager.html#DatabaseManager.send_command)[¶](#database.database_manager.DatabaseManager.send_command "Permalink to this definition")

Send a query to database.

Catch timeout and OperationalError.

| Parameters: | * **data** (*tuple*) – data attached to query

* **fetchall** (*bool*) – True if return all results

 |

| Returns: | result of the query and status message |

### database.database\_swiftea module[¶](#module-database.database_swiftea "Permalink to this headline")

*class* `database.database_swiftea.``DatabaseSwiftea`(*host*, *user*, *password*, *name*, *table*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea)[¶](#database.database_swiftea.DatabaseSwiftea "Permalink to this definition")

Bases: [`database.database\_manager.DatabaseManager`](#database.database_manager.DatabaseManager "database.database_manager.DatabaseManager")

Class to manage Swiftea database.

| Parameters: | * **host** (*str*) – hostname of the sftp server

* **user** (*str*) – username to use for connection

* **password** (*str*) – password to use for connection

* **name** (*str*) – name of database

 |

`send_doc`(*webpage\_infos*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.send_doc)[¶](#database.database_swiftea.DatabaseSwiftea.send_doc "Permalink to this definition")

send documents informations to database.

| Parameters: | **infos** (*list*) – informations to send to database |

| Returns: | True if an error occured |

`update`(*infos*, *popularity*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.update)[¶](#database.database_swiftea.DatabaseSwiftea.update "Permalink to this definition")

Update a document in database.

| Parameters: | * **infos** (*dict**(**)*) – doc infos

* **popularity** (*int*) – new doc popularity

 |

| Returns: | True is an arror occured |

`insert`(*infos*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.insert)[¶](#database.database_swiftea.DatabaseSwiftea.insert "Permalink to this definition")

Insert a new document in database.

| Parameters: | **infos** (*dict**(**)*) – doc infos |

| Returns: | True is an arror occured |

`get_doc_id`(*url*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.get_doc_id)[¶](#database.database_swiftea.DatabaseSwiftea.get_doc_id "Permalink to this definition")

Get id of a document in database.

| Parameters: | **url** (*str*) – url of webpage |

| Returns: | id of webpage or None if not found |

`del_one_doc`(*url*, *table=None*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.del_one_doc)[¶](#database.database_swiftea.DatabaseSwiftea.del_one_doc "Permalink to this definition")

Delete document corresponding to url.

| Parameters: | **url** (*str*) – url of webpage |

| Returns: | status message |

`suggestions`()[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.suggestions)[¶](#database.database_swiftea.DatabaseSwiftea.suggestions "Permalink to this definition")

Get the five first URLs from Suggestion table and delete them.

| Returns: | list of url in Suggestion table and delete them |

`doc_exists`(*url*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.doc_exists)[¶](#database.database_swiftea.DatabaseSwiftea.doc_exists "Permalink to this definition")

Check if url is in database.

| Parameters: | **url** (*str*) – url corresponding to doc |

| Returns: | True if doc exists |

`https_duplicate`(*old\_url*)[[source]](_modules/database/database_swiftea.html#DatabaseSwiftea.https_duplicate)[¶](#database.database_swiftea.DatabaseSwiftea.https_duplicate "Permalink to this definition")

Avoid https and http duplicate.

If old url is secure (https), must delete insecure url if exists,

then return secure url (old url).

If old url is insecure (http), must delete it if secure url exists,

then return secure url (new url)

| Parameters: | **old\_url** (*str*) – old url |

| Returns: | url to add and url to delete |

### index.index module[¶](#module-index.index "Permalink to this headline")

Define several functions for inverted-index.

`index.index.``count_files_index`(*index*)[[source]](_modules/index/index.html#count_files_index)[¶](#index.index.count_files_index "Permalink to this definition")

Return number of file to download are uplaod

Parse languages and letters from the given index.

| Returns: | int |

`index.index.``stats_dl_index`(*begining*, *end*)[[source]](_modules/index/index.html#stats_dl_index)[¶](#index.index.stats_dl_index "Permalink to this definition")

Write the time to download inverted-index.

| Parameters: | * **begining** (*int*) – time download inverted-index

* **end** (*int*) – time after download inverted-index

 |

`index.index.``stats_ul_index`(*begining*, *end*)[[source]](_modules/index/index.html#stats_ul_index)[¶](#index.index.stats_ul_index "Permalink to this definition")

Write the time to upload inverted-index.

| Parameters: | * **begining** (*int*) – time before send inverted-index

* **end** (*int*) – time after send inverted-index

 |

### index.inverted\_index module[¶](#module-index.inverted_index "Permalink to this headline")

*class* `index.inverted_index.``InvertedIndex`[[source]](_modules/index/inverted_index.html#InvertedIndex)[¶](#index.inverted_index.InvertedIndex "Permalink to this definition")

Manage inverted-index for crawler.

Inverted-index is a dict, each keys are language

> 

> -> values are a dict, each keys are first letter

> 

> 

> -> values are dict, each keys are two first letters

> 

> 

> -> values are dict, each keys are word

> 

> 

> -> values are dict, each keys are id

> 

> 

> -> values are int : tf

> 

> 

> 

example:

[‘FR’][‘A’][‘av’][‘avion’][21] is tf of word ‘avion’ in doc 21 in french.

`setInvertedIndex`(*inverted\_index*)[[source]](_modules/index/inverted_index.html#InvertedIndex.setInvertedIndex)[¶](#index.inverted_index.InvertedIndex.setInvertedIndex "Permalink to this definition")

Define inverted-index at the beginning.

| Parameters: | **inverted\_index** (*dict*) – inverted-index |

`getInvertedIndex`()[[source]](_modules/index/inverted_index.html#InvertedIndex.getInvertedIndex)[¶](#index.inverted_index.InvertedIndex.getInvertedIndex "Permalink to this definition")

| Returns: | inverted-index |

`add_doc`(*keywords*, *doc\_id*, *language*)[[source]](_modules/index/inverted_index.html#InvertedIndex.add_doc)[¶](#index.inverted_index.InvertedIndex.add_doc "Permalink to this definition")

Add all words of a doc in inverted-index.

| Parameters: | * **keywords** (*list*) – all word in doc\_id

* **doc\_id** (*int*) – id of the doc in database

* **language** (*str*) – language of word

 |

`add_word`(*word\_infos*, *doc\_id*, *nb\_words*)[[source]](_modules/index/inverted_index.html#InvertedIndex.add_word)[¶](#index.inverted_index.InvertedIndex.add_word "Permalink to this definition")

Add a word in inverted-index.

| Parameters: | * **word\_infos** (*dict*) – word infos: word, language, occurence,

first letter and two first letters

* **doc\_id** (*int*) – id of the doc in database

* **nb\_words** (*int*) – number of words in the doc\_id

 |

`delete_word`(*word*, *language*, *first\_letter*, *filename*)[[source]](_modules/index/inverted_index.html#InvertedIndex.delete_word)[¶](#index.inverted_index.InvertedIndex.delete_word "Permalink to this definition")

Delete a word in inverted-index.

| Parameters: | * **word** (*str*) – word to delete

* **language** (*str*) – language of word

* **first\_letter** (*str*) – first letter of word

* **filename** (*str*) – two first letters of word

 |

`delete_id_word`(*word\_infos*, *doc\_id*)[[source]](_modules/index/inverted_index.html#InvertedIndex.delete_id_word)[¶](#index.inverted_index.InvertedIndex.delete_id_word "Permalink to this definition")

Delete a id of a word in inverted-index

This method delete a word from a document.

Remove a words from a doc.

| Parameters: | * **word\_infos** (*dict*) – word infos: word, language, first letter and two first letters

* **doc\_id** (*int*) – id of the doc in database

 |

`delete_doc_id`(*doc\_id*)[[source]](_modules/index/inverted_index.html#InvertedIndex.delete_doc_id)[¶](#index.inverted_index.InvertedIndex.delete_doc_id "Permalink to this definition")

Delete a id in inverted-index.

| Parameters: | **doc\_id** (*int*) – id to delete |

### index.sftp\_manager module[¶](#index-sftp-manager-module "Permalink to this headline")

### index.sftp\_swiftea module[¶](#index-sftp-swiftea-module "Permalink to this headline")

Tests[¶](#tests "Permalink to this headline")

---------------------------------------------

Tests for Swiftea-Crawler using pytest.

### tests.run\_tests[¶](#tests-run-tests "Permalink to this headline")

### tests.swiftea\_bot\_test[¶](#tests-swiftea-bot-test "Permalink to this headline")

### tests.crawling\_test[¶](#tests-crawling-test "Permalink to this headline")

### tests.database\_test[¶](#tests-database-test "Permalink to this headline")

### tests.index\_test[¶](#tests-index-test "Permalink to this headline")

### tests.crawler\_test[¶](#tests-crawler-test "Permalink to this headline")

### tests.global\_test[¶](#tests-global-test "Permalink to this headline")

### tests.test\_data[¶](#tests-test-data "Permalink to this headline")

Indices and tables[¶](#indices-and-tables "Permalink to this headline")

=======================================================================

* [Index](genindex.html)

* [Module Index](py-modindex.html)

* [Search Page](search.html)

