<commit_before>\section{Related work}
2-related-work.tex

Web Object Identification for Web Automation and Meta-Search,
\\
http://www.dbai.tuwien.ac.at/proj/ta\\
mcrow/download/Kordomatis2013WIMS.pdf

Self-supervised Automated Wrapper Generation for Weblog Data Extraction,
\\
https://github.com/OlivierBlanvillai\\
n/blogforever-crawler-publication/raw\\
/master/papers/bncod\_published.pdf

Web Data Extraction, Applications and Techniques: A Survey,
\\
http://www.emilio.ferrara.name/wp-co\\
ntent/uploads/2011/07/survey-csur.pdf

Archiving Data Objects using Web Feeds,
\\
http://hal.archives-ouvertes.fr/docs\\
/00/53/79/62/PDF/iwawienna.pdf

Intelligent and Adaptive Crawling of Web Applications for Web Archiving,
\\
http://pierre.senellart.com/publicat\\
ions/faheem2013intelligent.pdf

Zero-cost Labelling with Web Feeds for Weblog Data Extraction,
\\
http://www2013.org/companion/p73.pdf\\


OXPath: A Language for Scalable, Memory-efficient Data Extraction from Web Applications,
\\
http://www.vldb.org/pvldb/vol4/p1016-furche.pdf<commit_msg>First partial attempt to related work<commit_after>\section{Related work}

Web archiving is a challenging and very interesting problem and has been the subject of many studies as well as the goal of several projects.
A fraction of these focuses on the specific problem of blog harvesting and archiving.
These studies and projects focus on the unique nature of blogs as dynamic web resources with a usually common set of attributes.
Each entry in a blog, commonly called a post, has a title, an author, a publication date, some content and possibly some descriptive tags.
Blogs are also strongly associated with web feeds, using technologies such as RSS and Atom (expand and reference).
Web feeds are structured documents (often XML-based) that advertize and provide access to a website's content.
Given their format, web feeds constitute an excellent source for blog archiving.
They do, however, present some problems: (a) web feeds contain only the latest posts of a blog (usually 20) and (b) web feeds may only provide a summary or a fraction of the entire content of each blog post.
More advanced methods and techniques have to be employed in order to achieve quality blog archiving.
In this section we review some studies and projects that introduce and describe such techniques.
\comment{Maybe this is repetitive, some of these things should already have been mentioned in the introduction}

\comment{list the projects and compare them to what we do}

Web Object Identification for Web Automation and Meta-Search\\
http://www.dbai.tuwien.ac.at/proj/tamcrow/download/Kordomatis2013WIMS.pdf\\
In this paper we deal with the problem of finding specific web objects on previously unseen web pages. The problem consists of the sub problems of how to describe the desired objects (e.g., a specific submit button)and how to identify them on new web pages. This general problem is fundamental to many fields of research like Web Data Extraction, Web Form Understanding, Web Automation, etc.
Our approach uses features based on the visual perceivable  characteristics of web objects for object description and machine learning techniques for object identification. Applied to a meta-search setting, this allows for a very universal solution. Here, the common elements of the considered search forms can be identified on previously unseen search pages, as long as they follow the same design principles.

Self-supervised Automated Wrapper Generation for Weblog Data Extraction\\
https://github.com/OlivierBlanvillain/blogforever-crawler-publication/raw/master/papers/bncod\_published.pdf\\

Web Data Extraction, Applications and Techniques: A Survey\\
http://www.emilio.ferrara.name/wp-content/uploads/2011/07/survey-csur.pdf\\

Archiving Data Objects using Web Feeds\\
http://hal.archives-ouvertes.fr/docs/00/53/79/62/PDF/iwawienna.pdf\\

Intelligent and Adaptive Crawling of Web Applications for Web Archiving\\
http://pierre.senellart.com/publications/faheem2013intelligent.pdf\\

Zero-cost Labelling with Web Feeds for Weblog Data Extraction\\
http://www2013.org/companion/p73.pdf\\

OXPath: A Language for Scalable, Memory-efficient Data Extraction from Web Applications\\
http://www.vldb.org/pvldb/vol4/p1016-furche.pdf\\
{'fraction_non_alphanumeric': 0.07228070175438596, 'fraction_numerical': 0.016140350877192983, 'mean_word_length': 4.611548556430447, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5770051', 'n_tokens_mistral': 1247, 'n_tokens_neox': 1141, 'n_words': 502}