<commit_before># tq

tq is command line utility  that performs performs an HTML element selection on HTML content passed to the stdin. Using css selectors that everybody knows.

Since input comes from stdin and output is sent to stout. It can easily be be used inside traditional UNIX pipelines to extract content from webpages and html files.

tq provides extra formating options such as json-encoding or newlines squashing, so it can pley nicely with everyones favourite command line tooling.


## Instalation

	sudo pip install https://github.com/plainas/tq/zipball/stable


## Example usage

Get headlines from hacker news

	curl https://news.ycombinator.com/news | tq -tj ".title a"

Get the title of an html document stored in a file

	cat mydocument.html | tq -t title

Get all the images from a webpage

	TODO: add this example


Notice that tq doesn't provide a way to make http requests or read files. You can use your favorite HTTP client, or provide the html source from any source you want.

For a modern, user friendly http client, check httpie. Or you can just use curl, wget, netcat, etc.

## Command options

  * `SELECTOR`
    A css selector

  * `-a ATTRIBUTE --attr=ATTRIBUTE`<commit_msg>added examples snippet to readme<commit_after># tq

tq is command line utility  that performs performs an HTML element selection on HTML content passed to the stdin. Using css selectors that everybody knows.

Since input comes from stdin and output is sent to stout. It can easily be be used inside traditional UNIX pipelines to extract content from webpages and html files.

tq provides extra formating options such as json-encoding or newlines squashing, so it can play nicely with everyones favourite command line tooling.


## Instalation

	sudo pip install https://github.com/plainas/tq/zipball/stable


## Example usage

Get headlines from hacker news

	curl https://news.ycombinator.com/news | tq -tj ".title a"

Get the title of an html document stored in a file

	cat mydocument.html | tq -t title

Get all the images from a webpage

	curl -s 'http://example.com/' | tq  "img" -a src | wget -i


Notice that tq doesn't provide a way to make http requests or read files. You can use your favorite HTTP client, or provide the html source from any source you want.

For a modern, user friendly http client, check httpie. Or you can just use curl, wget, netcat, etc.

## Command options

  * `SELECTOR`
    A css selector

  * `-a ATTRIBUTE --attr=ATTRIBUTE`
{'fraction_non_alphanumeric': 0.05463375151760421, 'fraction_numerical': 0.0, 'mean_word_length': 3.772200772200772, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6212564', 'n_tokens_mistral': 708, 'n_tokens_neox': 669, 'n_words': 370}