<commit_before># qurl

[![License][License-Image]][License-Url]
[![ReportCard][ReportCard-Image]][ReportCard-Url]

## Table of contents

* [Introduction](https://github.com/repejota/qurl#introduction)
* [User Documentation](https://github.com/repejota/qurl#user-documentation)
	* [Basic Usage](https://github.com/repejota/qurl#basic-usage)
	* [Header Queries](https://github.com/repejota/qurl#header-queries)
	* [Selector Queries](https://github.com/repejota/qurl#selector-queries)
* [Developer Documentation](https://github.com/repejota/qurl#developer-documentation)
* [Continuous Integration](https://github.com/repejota/qurl#continuous-integration)
  * [Tests](https://github.com/repejota/qurl#license)
  * [Coverage](https://github.com/repejota/qurl#coverage)
* [License](https://github.com/repejota/qurl#license)

## Introduction

Qurl is a drop-in and easy to deploy microservice who exposes an HTTP API you can use to extract content from any web page as JSON. Any information available at a given public URL can be extracted using selector queries (check examples below).

## User Documentation

User documentation

### Basic Usage

The most simple query only returns the HTTP Status code response when fetching url contents.

Example:

```
$ curl -s http://qurl.io/q?url=https://example.com | json_pp 
{
   "url" : "https://www.example.com",
   "status" : 200
}
```

> We use `-s` curl flag to enable silent or quiet mode. So curl won't show progress meter or error messages. It will still output the resulting data.

> Also the result is piped to `json_pp` command to pretty print the JSON data.

> See man pages [curl(1)](http://www.manpagez.com/man/1/curl/) and [json_pp(1)](http://www.manpagez.com/man/1/json_pp/) for more information.

### Header Queries

Get any HTTP headers defined in the response, just adding a `header` parameter with the name of the header you want to get as value to the URL query.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Content-Type' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Content-Type" : [
         "text/html; charset=utf-8"
      ]
   }
}
```

Notice that the resulting value is not an string but a list of strings. This is because a single header can have multiple values in the same response. 

In this case we will also return a list of values for the response header so you'll be able to iterate over them.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Set-Cookie' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Set-Cookie" : [
         "_octo=GH1.1.1872640429.1506145235; domain=.github.com; path=/; expires=Mon, 23 Sep 2019 05:40:35 -0000",
         "logged_in=no; domain=.github.com; path=/; expires=Wed, 23 Sep 2037 05:40:35 -0000; secure; HttpOnly",
         "_gh_sess=<KEY>YUlsT1h3YXZTYTErVWJyNCsvRnFFcE5zNVdxUGczUzVKSlo1bTZnPSJ9--8dd729a38c9647489eb95438071a0a5bf083edb6; path=/; secure; HttpOnly"
      ]
   }
}
```

It is possible also to query more than one response header at once. Just append as much `header` parameters with their names as you neeed to the query URL.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Date&header=Cache-Control' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Cache-Control" : [
         "no-cache"
      ],
      "Date" : [
         "Sat, 23 Sep 2017 05:47:30 GMT"
      ]
   }
}
 
```

If the header you query is not present in the response, we will still add it to the result and its value will be `null`.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=foobar' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "foobar" : null
   }
}
```

Finally if you query for an specific header more than once, the result will be only populated once so bandwitch will be saved and the response data will be simpler.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Date&header=Date' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Date" : [
         "Sat, 23 Sep 2017 06:03:56 GMT"
      ]
   }
}
```

### Selector Queries

selector queries

## Developer Documentation

Developer documentation

## Continuous integration

### Tests

* Develop: [![CircleCI](https://circleci.com/gh/repejota/qurl/tree/develop.svg?style=svg)](https://circleci.com/gh/repejota/qurl/tree/develop)
* Master: [![CircleCI](https://circleci.com/gh/repejota/qurl/tree/master.svg?style=svg)](https://circleci.com/gh/repejota/qurl/tree/master)

### Coverage

* Develop: [![Coverage Status](https://coveralls.io/repos/github/repejota/qurl/badge.svg?branch=develop)](https://coveralls.io/github/repejota/qurl?branch=develop)<commit_msg>Footnotes on headers<commit_after># qurl

[![License][License-Image]][License-Url]
[![ReportCard][ReportCard-Image]][ReportCard-Url]

## Table of contents

* [Introduction](https://github.com/repejota/qurl#introduction)
* [User Documentation](https://github.com/repejota/qurl#user-documentation)
	* [Basic Usage](https://github.com/repejota/qurl#basic-usage)
	* [Header Queries](https://github.com/repejota/qurl#header-queries)
	* [Selector Queries](https://github.com/repejota/qurl#selector-queries)
* [Developer Documentation](https://github.com/repejota/qurl#developer-documentation)
* [Continuous Integration](https://github.com/repejota/qurl#continuous-integration)
  * [Tests](https://github.com/repejota/qurl#license)
  * [Coverage](https://github.com/repejota/qurl#coverage)
* [License](https://github.com/repejota/qurl#license)

## Introduction

Qurl is a drop-in and easy to deploy microservice who exposes an HTTP API you can use to extract content from any web page as JSON. Any information available at a given public URL can be extracted using selector queries (check examples below).

## User Documentation

User documentation

### Basic Usage

The most simple query only returns the HTTP Status code response when fetching url contents.

Example:

```
$ curl -s http://qurl.io/q?url=https://example.com | json_pp 
{
   "url" : "https://www.example.com",
   "status" : 200
}
```

> We use `-s` curl flag to enable silent or quiet mode. So curl won't show progress meter or error messages. It will still output the resulting data.

> Also the result is piped to `json_pp` command to pretty print the JSON data.

> See man pages [curl(1)](http://www.manpagez.com/man/1/curl/) and [json_pp(1)](http://www.manpagez.com/man/1/json_pp/) for more information.

### Header Queries

Get any HTTP headers defined in the response, just adding a `header` parameter with the name of the header you want to get as value to the URL query.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Content-Type' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Content-Type" : [
         "text/html; charset=utf-8"
      ]
   }
}
```

Notice that the resulting value is not an string but a list of strings. This is because a single header can have multiple values in the same response. 

In this case we will also return a list of values for the response header so you'll be able to iterate over them.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Set-Cookie' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Set-Cookie" : [
         "_octo=GH1.1.1872640429.1506145235; domain=.github.com; path=/; expires=Mon, 23 Sep 2019 05:40:35 -0000",
         "logged_in=no; domain=.github.com; path=/; expires=Wed, 23 Sep 2037 05:40:35 -0000; secure; HttpOnly",
         "_gh_sess=eyJzZXNzaW9uX2lkIjoiNGU3ZmEwMGUzN2RkOTFkYmYyYjFhM2RmODA3YTc4M2QiLCJsYXN0X3JlYWRfZnJvbV9yZXBsaWNhcyI6MTUwNjE0NTIzNTY2NywiX2NzcmZfdG9rZW4iOiJWMXFCYUlsT1h3YXZTYTErVWJyNCsvRnFFcE5zNVdxUGczUzVKSlo1bTZnPSJ9--8dd729a38c9647489eb95438071a0a5bf083edb6; path=/; secure; HttpOnly"
      ]
   }
}
```

It is possible also to query more than one response header at once. Just append as much `header` parameters with their names as you neeed to the query URL.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Date&header=Cache-Control' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Cache-Control" : [
         "no-cache"
      ],
      "Date" : [
         "Sat, 23 Sep 2017 05:47:30 GMT"
      ]
   }
}
 
```

If the header you query is not present in the response, we will still add it to the result and its value will be `null`.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=foobar' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "foobar" : null
   }
}
```

Finally if you query for an specific header more than once, the result will be only populated once so bandwitch will be saved and the response data will be simpler.

Example:

```
$ curl -s 'http://localhost:8080/q?url=https://github.com&header=Date&header=Date' | json_pp
{
   "url" : "https://github.com",
   "status" : 200,
   "headers" : {
      "Date" : [
         "Sat, 23 Sep 2017 06:03:56 GMT"
      ]
   }
}
```

> Mozila Developer Network website offers good and accurated documentation about what [HTTP Headers](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers) are their use. Their documentation is also available in multiple languages.

> For an updated list of standarizeed HTTP Headers, you can check the [RFC4229](https://tools.ietf.org/html/rfc4229) and the [IANA Message Headers](https://www.iana.org/assignments/message-headers/message-headers.xhtml) document. 

> Wikipedia also maintains a list of standard and non-standard [Response HTTP Headers](https://en.wikipedia.org/wiki/List_of_HTTP_header_fields#Response_fields) in case you want more information.

### Selector Queries

selector queries

## Developer Documentation

Developer documentation

## Continuous integration

### Tests

* Develop: [![CircleCI](https://circleci.com/gh/repejota/qurl/tree/develop.svg?style=svg)](https://circleci.com/gh/repejota/qurl/tree/develop)
* Master: [![CircleCI](https://circleci.com/gh/repejota/qurl/tree/master.svg?style=svg)](https://circleci.com/gh/repejota/qurl/tree/master)

### Coverage

* Develop: [![Coverage Status](https://coveralls.io/repos/github/repejota/qurl/badge.svg?branch=develop)](https://coveralls.io/github/repejota/qurl?branch=develop)
{'fraction_non_alphanumeric': 0.14171883892999432, 'fraction_numerical': 0.0313982166571808, 'mean_word_length': 3.97311320754717, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 13, 'https://': 60, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '153015', 'n_tokens_mistral': 3937, 'n_tokens_neox': 3586, 'n_words': 918}