<commit_before># gofeed

Gofeed was inspired by feed43.com. It is disigned to extract feeds from websites which do not provide any. Gofeed can even parse several websites into one full-text feed. 

This simple program was written when I started to learn golang. So I tried to reinvent everything I need, including a simple crawler which took good use of cache and a very simple rss2.0 feed generator.

## Features

* Like feed43.com, you can extract feed titles, feed links and the feed descriptions with the following predefined patterns. Note that all these patterns are **lazy** and perform **leftmost** match, which means they will match as few characters as possible.
    *  {any}: match any character including newline
    *  {title}: title of feed entry, matched against the Feed.URL page
    *  {link}: hyper link of feed entry, matched against the Feed.URL page
    *  {description}: full-text description of feed entry, matched against the corresponding {link} page
* The crawler knows well when to request new html and when to use local html cache. This will save a lot of bandwidth.
* Several websites in one full-text feed.
 
## Things need to be improved

*  The crawler only understands tcp protocol.
*  Lacking URL normalization.
*  Little documentation.

## More functions on the todo list

1. <del>Cache old requests: use sqlite to cache downloaded web pages and save their lastmod time.</del>
2. Add alternative methods to extract feed title, link and description from html<commit_msg>update readme, add todo entry<commit_after># gofeed

Gofeed was inspired by feed43.com. It is disigned to extract feeds from websites which do not provide any. Gofeed can even parse several websites into one full-text feed. 

This simple program was written when I started to learn golang. So I tried to reinvent everything I need, including a simple crawler which took good use of cache and a very simple rss2.0 feed generator.

## Features

* Like feed43.com, you can extract feed titles, feed links and the feed descriptions with the following predefined patterns. Note that all these patterns are **lazy** and perform **leftmost** match, which means they will match as few characters as possible.
    *  {any}: match any character including newline
    *  {title}: title of feed entry, matched against the Feed.URL page
    *  {link}: hyper link of feed entry, matched against the Feed.URL page
    *  {description}: full-text description of feed entry, matched against the corresponding {link} page
* The crawler knows well when to request new html and when to use local html cache. This will save a lot of bandwidth.
* Several websites in one full-text feed.
 
## Things need to be improved

*  The crawler only understands tcp protocol.
*  Lacking URL normalization.
*  Little documentation.

## More functions on the todo list

1. <del>Cache old requests: use sqlite to cache downloaded web pages and save their lastmod time.</del>
2. Download html files for each feed target defined in the configuration in separate goroutines. 
3. Add alternative methods to extract feed title, link and description from html
{'fraction_non_alphanumeric': 0.052210121716848176, 'fraction_numerical': 0.005445227418321589, 'mean_word_length': 4.0946166394779775, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5021651', 'n_tokens_mistral': 777, 'n_tokens_neox': 747, 'n_words': 468}