<commit_before># wikipedia

tool for extracting plain text from wikipedia articles

## Installing:

a gem is available, so fire up your terminal:

````
$ gem install wikipedia
````

## Usage:

it's easy:

````ruby
irb(main):001:0* require 'wikipedia'
irb(main):002:0>
irb(main):003:0* connor = Wikipedia::article '<NAME>'
irb(main):004:0* connor.first     # just the first paragraph
"<NAME> is a fictional character and the main protagonist of the Terminator franchise.
Created by writer and director <NAME>, the character is first referred to in the 1984 film The Terminator 
and first appears, portrayed by teenage actor <NAME>, in its 1991 sequel Terminator 2: Judgment Day.
The character is subsequently portrayed by 23-year-old <NAME> in the 2003 film Terminator 3: Rise of the Machines
and by 19-year-old <NAME> in the 2007 television series Terminator: The Sarah Connor Chronicles.
English actor <NAME> portrays Connor in the film series' fourth installment, Terminator Salvation."
````

There's a simple method for checking term's ambiguity, an array of those other terms will be provided in the future.

A good example is 'apple' which may refer to the company, to the fruit, etc.

````ruby
irb(main):001:0> require 'wikipedia'
irb(main):002:0> apple = Wikipedia::article 'apple'
irb(main):003:0> apple.ambiguous?
=> true
````

## TODO

* Integrate it with the [Opensearch API] (http://www.mediawiki.org/wiki/API%3aOpensearch).
* Provide a method for classifying text based on context (using data from Wikipedia's disambiguation pages).
* Switch to Nokogiri or provide support for both Nokogiri and Hpricot?<commit_msg>removing an extra line from irb example<commit_after># wikipedia

tool for extracting plain text from wikipedia articles

## Installing:

a gem is available, so fire up your terminal:

````
$ gem install wikipedia
````

## Usage:

it's easy:

````ruby
irb(main):001:0* require 'wikipedia'
irb(main):002:0>
irb(main):003:0* connor = Wikipedia::article '<NAME>'
irb(main):004:0> connor.first     # just the first paragraph
"<NAME> is a fictional character and the main protagonist of the Terminator franchise.
Created by writer and director <NAME>, the character is first referred to in the 1984 film The Terminator 
and first appears, portrayed by teenage actor <NAME>, in its 1991 sequel Terminator 2: Judgment Day.
The character is subsequently portrayed by 23-year-old <NAME> in the 2003 film Terminator 3: Rise of the Machines
and by 19-year-old <NAME> in the 2007 television series Terminator: The Sarah Connor Chronicles.
English actor <NAME> portrays Connor in the film series' fourth installment, Terminator Salvation."
````

There's a simple method for checking term's ambiguity, an array of those other terms will be provided in the future.

A good example is 'apple' which may refer to the company, to the fruit, etc.

````ruby
irb(main):001:0> require 'wikipedia'
irb(main):002:0> apple = Wikipedia::article 'apple'
irb(main):003:0> apple.ambiguous?
=> true
````

## TODO

* Integrate it with the [Opensearch API] (http://www.mediawiki.org/wiki/API%3aOpensearch).
* Provide a method for classifying text based on context (using data from Wikipedia's disambiguation pages).
* Switch to Nokogiri or provide support for both Nokogiri and Hpricot?
{'fraction_non_alphanumeric': 0.09628275441803778, 'fraction_numerical': 0.031078610603290677, 'mean_word_length': 5.136448598130841, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1477041', 'n_tokens_mistral': 1065, 'n_tokens_neox': 924, 'n_words': 431}