<commit_before>homepage: https://sr.ht/~geyaeb/haskell-readability
changelog-type: markdown
hash: 1c9e709f84c0aaded16fe5c4498690d17822ae3cee31583e59f7a3b858b424ec
test-bench-deps: {}
maintainer: <EMAIL>
synopsis: Extracts text of main article from HTML document
changelog: |
  # Revision history for readability

  ## 0.0.1.0 – 2020-06-26

  * First version
basic-deps:
  bytestring: ==0.10.*
  xml-conduit: '>=1.7 && <2'
  base: '>=4.7 && <5'
  text: ==1.2.*
  html-conduit: ==1.3.*
  containers: ==0.6.*
  readability: -any
  optparse-applicative: ==0.15.*
  http-conduit: ==2.3.*
all-versions:
- 0.0.1.0
author: <NAME>
latest: 0.0.1.0
description-type: markdown
description: |
  # readability

  Give `readability` an HTML document and it will detect and extract text of the article while removing everything unnecessary like menus, advertisements or sidebars. It is more or less reimplementation of [python-readability](https://github.com/buriy/python-readability).

  The package contains both a library and simple executable.

  ## Example of using `readability` executable

  Having an article that looks like following image:

  ![Original HTML](https://hg.sr.ht/~geyaeb/haskell-readability/raw/doc/orig.png "Original HTML")

  we can extract text by calling:

  ```shell
  $> readability https://mises.org/wire/why-central-banks-are-threat-our-savings
  ```

  and we get the following HTML:

  ![Extracted text](https://hg.sr.ht/~geyaeb/haskell-readability/raw/doc/readable.png "Extracted text")


  If we are interested in plain text, we can further use `pandoc`:

  ```text
  $> readability https://mises.org/wire/why-central-banks-are-threat-our-savings | pandoc -f html -t plain

  The US personal savings rate jumped to 33 percent in April from 12.7
  percent in March and 8 percent in April last year. An increase in
  savings is regarded by popular economics as less expenditure on
  consumption. Since consumption expenditure is considered as the main
  driving force of the economy, obviously a rebound in savings, which
  implies less consumption, cannot be good for economic activity, so it is
  held. Saving and wealth—what is the relation?

  To maintain their life and well-being, individuals require access to
  consumer goods. An increase in various consumer goods permits an
  increase in individuals’ living standards. What allows an increase in
  the production of consumer goods is the maintenance and the enhancement
  of the infrastructure of an economy. With better infrastructure, a
  greater quantity and better quality of consumer goods could be generated
  and more real wealth can be produced.

  The enhancement and the maintenance of the infrastructure becomes
  possible because of the availability of final consumer goods that
  sustain the various individuals who are busy expanding and maintaining
  the infrastructure. It is the producers of final consumer goods who pay
  the various individuals engaged in maintenaning and enhancing the
  infrastructure. The producers of final consumer goods pay these
  individuals (i.e., the intermediary producers) out of the saved or
  unconsumed production of final consumer goods.

  Note that when a producer of final consumer goods decides to save more,
  i.e., to consume less, the fall in his consumption is offset by the
  increase in the consumption of individuals who are engaged in the
  intermediary stages of production. This means that overall consumption
  is not declining because of an increase in saving—as popular thinking
  has it.
  ```

  Had we not processed the article through `readability`, we would have gotten:

  ``` text
  $> curl https://mises.org/wire/why-central-banks-are-threat-our-savings | pandoc -f html -t plain

  Skip to main content

  [Home]

  Toggle navigation

  -   Blog
  -   <NAME>
  -   Books
  -   Podcast
  -   Video
  -   Events
  -   Store
  -   Graduate Program

  -   Ver en Español

  Stay Connected

  GO

  SUPPORT MISES

  JOIN OR RENEW TODAY

  SUPPORT MISES

  JOIN OR RENEW TODAY

  <NAME>

  GET NEWS AND ARTICLES IN YOUR INBOXPrint

  A

  A

  Home | Wire | Why Central Banks Are a Threat to Our Savings

  Why Central Banks Are a Threat to Our Savings

  -   [dollars]

  0 Views

  Tags

  Money and Banking

  06/25/2020<NAME>

  The US personal savings rate jumped to 33 percent in April from 12.7
  percent in March and 8 percent in April last year. An increase in
  savings is regarded by popular economics as less expenditure on
  consumption. Since consumption expenditure is considered as the main
  driving force of the economy, obviously a rebound in savings, which
  implies less consumption, cannot be good for economic activity, so it is
  held. Saving and wealth—what is the relation?
  ```

  ## Contribute

  Project is hosted at https://sr.ht/~geyaeb/haskell-readability/ . The homepage provides links to [Mercurial repository](https://hg.sr.ht/~geyaeb/haskell-readability), [mailing list](https://lists.sr.ht/~geyaeb/haskell-readability) and [ticket tracker](https://todo.sr.ht/~geyaeb/haskell-readability).

  Patches, suggestions, questions and general discussions can be send to the [mailing list](https://lists.sr.ht/~geyaeb/haskell-readability). Detailed information about sending patches by email can be found at [https://man.sr.ht/hg.sr.ht/email.md](https://man.sr.ht/hg.sr.ht/email.md).<commit_msg>Update from Hackage at 2020-07-09T11:41:27Z<commit_after>homepage: https://sr.ht/~geyaeb/haskell-readability
changelog-type: markdown
hash: c2e61b876478bd18cfe5ed02f6f9286d149a2e834a218369ab1fc4d384f1df79
test-bench-deps: {}
maintainer: <EMAIL>
synopsis: Extracts text of main article from HTML document
changelog: |
  # Revision history for readability

  ## 0.1.0.0 – 2020-07-09

  * Added reading of data from standard input
  * Added option `extract` to command line to choose what to print
  * Added extraction of title and short title

  ## 0.0.1.0 – 2020-06-26

  * First version
basic-deps:
  bytestring: ==0.10.*
  xml-conduit: '>=1.7 && <2'
  base: '>=4.7 && <5'
  text: ==1.2.*
  html-conduit: ==1.3.*
  containers: ==0.6.*
  readability: -any
  optparse-applicative: ==0.15.*
  http-conduit: ==2.3.*
  aeson: '>=1.4 && <1.6'
all-versions:
- 0.0.1.0
- 0.1.0.0
author: <NAME>
latest: 0.1.0.0
description-type: markdown
description: |
  # readability

  Give `readability` an HTML document and it will detect and extract text of the article while removing everything unnecessary like menus, advertisements or sidebars. It is more or less reimplementation of [python-readability](https://github.com/buriy/python-readability).

  The package contains both a library and simple executable.

  ## Example of using `readability` executable

  Having an article that looks like following image:

  ![Original HTML](https://hg.sr.ht/~geyaeb/haskell-readability/raw/doc/orig.png "Original HTML")

  we can extract text by calling:

  ```shell
  $> readability https://mises.org/wire/why-central-banks-are-threat-our-savings
  ```

  and we get the following HTML:

  ![Extracted text](https://hg.sr.ht/~geyaeb/haskell-readability/raw/doc/readable.png "Extracted text")


  If we are interested in plain text, we can further use `pandoc`:

  ```text
  $> readability https://mises.org/wire/why-central-banks-are-threat-our-savings | pandoc -f html -t plain

  The US personal savings rate jumped to 33 percent in April from 12.7
  percent in March and 8 percent in April last year. An increase in
  savings is regarded by popular economics as less expenditure on
  consumption. Since consumption expenditure is considered as the main
  driving force of the economy, obviously a rebound in savings, which
  implies less consumption, cannot be good for economic activity, so it is
  held. Saving and wealth—what is the relation?

  To maintain their life and well-being, individuals require access to
  consumer goods. An increase in various consumer goods permits an
  increase in individuals’ living standards. What allows an increase in
  the production of consumer goods is the maintenance and the enhancement
  of the infrastructure of an economy. With better infrastructure, a
  greater quantity and better quality of consumer goods could be generated
  and more real wealth can be produced.

  The enhancement and the maintenance of the infrastructure becomes
  possible because of the availability of final consumer goods that
  sustain the various individuals who are busy expanding and maintaining
  the infrastructure. It is the producers of final consumer goods who pay
  the various individuals engaged in maintenaning and enhancing the
  infrastructure. The producers of final consumer goods pay these
  individuals (i.e., the intermediary producers) out of the saved or
  unconsumed production of final consumer goods.

  Note that when a producer of final consumer goods decides to save more,
  i.e., to consume less, the fall in his consumption is offset by the
  increase in the consumption of individuals who are engaged in the
  intermediary stages of production. This means that overall consumption
  is not declining because of an increase in saving—as popular thinking
  has it.
  ```

  Had we not processed the article through `readability`, we would have gotten:

  ``` text
  $> curl https://mises.org/wire/why-central-banks-are-threat-our-savings | pandoc -f html -t plain

  Skip to main content

  [Home]

  Toggle navigation

  -   Blog
  -   Mises Wire
  -   Books
  -   Podcast
  -   Video
  -   Events
  -   Store
  -   Graduate Program

  -   Ver en Español

  Stay Connected

  GO

  SUPPORT MISES

  JOIN OR RENEW TODAY

  SUPPORT MISES

  JOIN OR RENEW TODAY

  Mises Wire

  GET NEWS AND ARTICLES IN YOUR INBOXPrint

  A

  A

  Home | Wire | Why Central Banks Are a Threat to Our Savings

  Why Central Banks Are a Threat to Our Savings

  -   [dollars]

  0 Views

  Tags

  Money and Banking

  06/25/2020<NAME>

  The US personal savings rate jumped to 33 percent in April from 12.7
  percent in March and 8 percent in April last year. An increase in
  savings is regarded by popular economics as less expenditure on
  consumption. Since consumption expenditure is considered as the main
  driving force of the economy, obviously a rebound in savings, which
  implies less consumption, cannot be good for economic activity, so it is
  held. Saving and wealth—what is the relation?
  ```

  We can also print only title or short title of the article:

  ``` shell
  $> readability --extract shortTitle https://mises.org/wire/why-central-banks-are-threat-our-savings
  Why Central Banks Are a Threat to Our Savings
  ```

  Or we can print all available information as JSON for further processing:

  ``` shell
  $> readability --extract all https://mises.org/wire/why-central-banks-are-threat-our-savings | jq '.'
  {
    "article": "…",
    "shortTitle": "Why Central Banks Are a Threat to Our Savings",
    "title": "Why Central Banks Are a Threat to Our Savings | Mises Wire"
  }
  ```

  Raw HTML can be also provided using standard input when SOURCE is omitted:

  ``` shell
  $> curl -s https://mises.org/wire/why-central-banks-are-threat-our-savings | readability -e title
  Why Central Banks Are a Threat to Our Savings | Mises Wire
  ```

  ## Contribute

  Project is hosted at https://sr.ht/~geyaeb/haskell-readability/ . The homepage provides links to [Mercurial repository](https://hg.sr.ht/~geyaeb/haskell-readability), [mailing list](https://lists.sr.ht/~geyaeb/haskell-readability) and [ticket tracker](https://todo.sr.ht/~geyaeb/haskell-readability).

  Patches, suggestions, questions and general discussions can be send to the [mailing list](https://lists.sr.ht/~geyaeb/haskell-readability). Detailed information about sending patches by email can be found at [https://man.sr.ht/hg.sr.ht/email.md](https://man.sr.ht/hg.sr.ht/email.md).
{'fraction_non_alphanumeric': 0.08063021316033364, 'fraction_numerical': 0.019799477630802934, 'mean_word_length': 3.63129145532579, 'pattern_counts': {'":': 3, '<': 16, '<?xml version=': 0, '>': 25, 'https://': 31, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5974568', 'n_tokens_mistral': 3767, 'n_tokens_neox': 3390, 'n_words': 1428}