<commit_before>MetaInvestigator
=============

MetaInvestigator is a library for web scraping, inspired by [MetaInspector](https://github.com/jaimeiniesta/metainspector).

You can get its title, images, charset, description, keywords, meta tags...etc

## Usage
You can use your favorite HTTP Client. HTTPoison, HTTPotion, tesla...etc

```elixir
html = HTTPClient.get!("https://github.com/nekova").body
page = MetaInvestigator.fetch(html)
# => %{best_image: "https://avatars1.githubusercontent.com/u/3464295?v=3&s=400",
#  best_title: "nekova (ಠ_ಠ) · GitHub",
#  images: ["https://avatars3.githubusercontent.com/u/3464295?v=3&s=460",
#   "https://assets-cdn.github.com/images/spinners/octocat-spinner-128.gif"],
#  meta: %MetaInvestigator.Meta{charset: "utf-8", keywords: nil,
#   og_image: "https://avatars1.githubusercontent.com/u/3464295?v=3&s=400",
#   og_title: "nekova (ಠ_ಠ)", og_type: "profile",
#   og_url: "https://github.com/nekova"}, title: "nekova (ಠ_ಠ) · GitHub"}
page.og_image
#=> "https://avatars3.githubusercontent.com/u/3464295?v=3&s=460"
page.best_title
#=> "nekova (ಠ_ಠ) · GitHub"
```

## Installation
First, add MetaInvestigator to your mix.exs dependencies:

```elixir
def dpes do
  [{:metainvestigator, "~> 0.0.2"}]
end
```

and run ```$ mix deps.get```.

```elixir
def application do
  [applications: [:metainvestigator]]
end
```

## LICENSE
```
Copyright © 2015 nekova <<EMAIL>>

This work is free. You can redistribute it and/or modify it under the
terms of the MIT License. See the LICENSE file for more details.
```
<commit_msg>Add 'how-to-confirm-the-operation' to README<commit_after>MetaInvestigator
=============

MetaInvestigator is a library for web scraping, inspired by [MetaInspector](https://github.com/jaimeiniesta/metainspector).

You can get its title, images, charset, description, keywords, meta tags...etc

## Usage
You can use your favorite HTTP Client. HTTPoison, HTTPotion, tesla...etc

```elixir
html = HTTPClient.get!("https://github.com/nekova").body
page = MetaInvestigator.fetch(html)
# => %{best_image: "https://avatars1.githubusercontent.com/u/3464295?v=3&s=400",
#  best_title: "nekova (ಠ_ಠ) · GitHub",
#  images: ["https://avatars3.githubusercontent.com/u/3464295?v=3&s=460",
#   "https://assets-cdn.github.com/images/spinners/octocat-spinner-128.gif"],
#  meta: %MetaInvestigator.Meta{charset: "utf-8", keywords: nil,
#   og_image: "https://avatars1.githubusercontent.com/u/3464295?v=3&s=400",
#   og_title: "nekova (ಠ_ಠ)", og_type: "profile",
#   og_url: "https://github.com/nekova"}, title: "nekova (ಠ_ಠ) · GitHub"}
page.og_image
#=> "https://avatars3.githubusercontent.com/u/3464295?v=3&s=460"
page.best_title
#=> "nekova (ಠ_ಠ) · GitHub"
```

## Installation
First, add MetaInvestigator to your mix.exs dependencies:

```elixir
def dpes do
  [{:metainvestigator, "~> 0.0.2"}]
end
```

and run ```$ mix deps.get```.

```elixir
def application do
  [applications: [:metainvestigator]]
end
```

## How to confirm the operation with iex
```elixir
$> cd metainvestigator
$> iex -S mix
iex(1)> HTTPoison.start
{:ok, [:idna, :hackney, :httpoison]}
iex(2)> html = HTTPoison.get!("URL").body
iex(3)> page = MetaInvestigator.fetch(html)
```

## LICENSE
```
Copyright © 2015 nekova <<EMAIL>>

This work is free. You can redistribute it and/or modify it under the
terms of the MIT License. See the LICENSE file for more details.
```

{'fraction_non_alphanumeric': 0.17697543651968037, 'fraction_numerical': 0.03580941106836342, 'mean_word_length': 5.156648451730419, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 20, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5719311', 'n_tokens_mistral': 1348, 'n_tokens_neox': 1247, 'n_words': 290}