<commit_before>biblicit
=============

Extract citations from PDFs.


# Usage

```ruby
  # Extract metadata from a file using the code from CiteSeerX
  result = Biblicit.extract(file: "myfile.pdf", tool: :citeseer)

  # Extract metadata from the contents of a PDF using cb2bib
  result = Biblicit.extract(contents: IO.read("myfile.pdf"), tool: :cb2bib, remote: true)

  # See reference information for "myfile.pdf"
  hash = result.header
```


# Algorithms

### CiteSeer (default)

Wrapper around Perl code extracted from [CiteSeerX](http://citeseer.ist.psu.edu/). 

Uses a model trained with the [svm-light](http://svmlight.joachims.org/) Support Vector Machine library to extract citation data for the PDF itself, and then uses [ParsCit](http://aye.comp.nus.edu.sg/parsCit/)'s model trained with the [CRF++](http://code.google.com/p/crfpp/) Conditional Random Fields library to parse citations from the PDF's bibliography, if any.

### cb2Bib

Wrapper around [cb2Bib](http://www.molspaces.com/cb2bib/) in command-line mode.

Uses an apparently less-sophisticated parsing algorithm than the CiteSeerX code to parse metadata, but then, if :remote=true, scrapes one of a large number of journal or public repository websites for a structured version of the citation data.


# Requirements

There are a lot, but you may not need all of them, depending on your use case.


## Required to support various input file formats

Different tools are used for different input file formats.

#### PDF - [Poppler](http://poppler.freedesktop.org/)

This provides `pdftotext`. You could install `xpdf` instead.

##### On Debian/Ubuntu

    sudo apt-get install poppler-utils

##### On OS X with Homebrew

    brew install poppler

#### Postscript - [Ghostscript](http://www.ghostscript.com/)

This provides `ps2ascii`.

##### On Debian/Ubuntu

    sudo apt-get install ghostscript

##### On OS X with Homebrew

    brew install ghostscript

#### Other (e.g. docx) - [AbiWord](http://www.abisource.com/)

This provides `abiword`.

##### On Debian/Ubuntu

    sudo apt-get install abiword

##### On OS X

As of writing, you're out of luck, because AbiWord doesn't compile on recent versions of OS X. According to their website, however, this is being actively worked on.


## Required to use the CiteSeer algorithm

#### CRF++

Required for bibliography extraction (reference information for works cited in the input).

You can specify where you have installed CRF++ by setting the CRFPP_HOME environment variable.
 
##### From source

    wget http://crfpp.googlecode.com/files/CRF%2B%2B-0.57.tar.gz
    tar xvzf CRF++-0.57.tar.gz
    cd CRF++-0.57
    ./configure 
    make
    sudo make install

##### On Debian/Ubuntu

    sudo apt-add-repository 'deb http://cl.naist.jp/~eric-n/ubuntu-nlp oneiric all'
    sudo apt-get update
    sudo apt-get install libcrf++

##### On OS X with Homebrew

    brew install crf++

#### svm-light

Required for header extraction (reference information for the input work itself).

The included model requires version 5, not the current version. You can specify where you have installed svm-light by setting the SVM_LIGHT_HOME environment variable.

##### From source

    mkdir svm_light5
    cd svm_light5
    wget http://download.joachims.org/svm_light/v5.00/svm_light.tar.gz
    tar -xzf svm_light.tar.gz
    make
    sudo ln -s $(readlink -f "$(dirname svm_classify)/$(basename svm_classify)") /usr/bin/svm_classify5
    sudo ln -s $(readlink -f "$(dirname svm_learn)/$(basename svm_learn)") /usr/bin/svm_learn5

Note: On OS X you'll need to use `greadlink` instead of `readlink` if you have `coreutils` installed, or another workaround for the absence of `-f`.

#### Perl modules

##### From CPAN

    sudo cpan install DBI
    sudo cpan install Digest::SHA1
    sudo cpan install Log::Log4perl
    sudo cpan install Log::Dispatch
    sudo cpan install String::Approx


## Required to use the cb2bib algorithm

#### cb2Bib
<commit_msg>Fix README to remove unnecessary perl dependencies and provide better svm-light install instructions<commit_after>biblicit
=============

Extract citations from PDFs.


# Usage

```ruby
  # Extract metadata from a file using the code from CiteSeerX
  result = Biblicit.extract(file: "myfile.pdf", tool: :citeseer)

  # Extract metadata from the contents of a PDF using cb2bib
  result = Biblicit.extract(contents: IO.read("myfile.pdf"), tool: :cb2bib, remote: true)

  # See reference information for "myfile.pdf"
  hash = result.header
```


# Algorithms

### CiteSeer (default)

Wrapper around Perl code extracted from [CiteSeerX](http://citeseer.ist.psu.edu/). 

Uses a model trained with the [svm-light](http://svmlight.joachims.org/) Support Vector Machine library to extract citation data for the PDF itself, and then uses [ParsCit](http://aye.comp.nus.edu.sg/parsCit/)'s model trained with the [CRF++](http://code.google.com/p/crfpp/) Conditional Random Fields library to parse citations from the PDF's bibliography, if any.

### cb2Bib

Wrapper around [cb2Bib](http://www.molspaces.com/cb2bib/) in command-line mode.

Uses an apparently less-sophisticated parsing algorithm than the CiteSeerX code to parse metadata, but then, if :remote=true, scrapes one of a large number of journal or public repository websites for a structured version of the citation data. Warning: sometimes it finds the wrong work!


# Requirements

There are a lot, but you may not need all of them, depending on your use case.


## Required to support various input file formats

Different tools are used for different input file formats.

#### PDF - [Poppler](http://poppler.freedesktop.org/)

This provides `pdftotext`. You could install `xpdf` instead.

##### On Debian/Ubuntu

    sudo apt-get install poppler-utils

##### On OS X with Homebrew

    brew install poppler

#### Postscript - [Ghostscript](http://www.ghostscript.com/)

This provides `ps2ascii`.

##### On Debian/Ubuntu

    sudo apt-get install ghostscript

##### On OS X with Homebrew

    brew install ghostscript

#### Other (e.g. docx) - [AbiWord](http://www.abisource.com/)

This provides `abiword`.

##### On Debian/Ubuntu

    sudo apt-get install abiword

##### On OS X

As of writing, you're out of luck, because AbiWord doesn't compile on recent versions of OS X. According to their website, however, this is being actively worked on.


## Required to use the CiteSeer algorithm

#### CRF++

Required for bibliography extraction (reference information for works cited in the input).

You can specify where you have installed CRF++ by setting the CRFPP_HOME environment variable.
 
##### From source

    wget http://crfpp.googlecode.com/files/CRF%2B%2B-0.57.tar.gz
    tar xvzf CRF++-0.57.tar.gz
    cd CRF++-0.57
    ./configure 
    make
    sudo make install

##### On Debian/Ubuntu

    sudo apt-add-repository 'deb http://cl.naist.jp/~eric-n/ubuntu-nlp oneiric all'
    sudo apt-get update
    sudo apt-get install libcrf++

##### On OS X with Homebrew

    brew install crf++

#### svm-light

Required for header extraction (reference information for the input work itself).

The included model requires version 5, not the current version. You can specify where you have installed svm-light by setting the SVM_LIGHT_HOME environment variable.

##### From source

    mkdir svm_light5
    cd svm_light5
    wget http://download.joachims.org/svm_light/v5.00/svm_light.tar.gz
    tar -xzf svm_light.tar.gz
    make
    echo "export SVM_LIGHT_HOME=`pwd`" >> ~/.profile # or .bashrc or whatever
    source ~/.profile

#### Perl modules

##### From CPAN

    sudo cpan install Digest::SHA1
    sudo cpan install String::Approx

## Required to use the cb2bib algorithm

#### cb2Bib

{'fraction_non_alphanumeric': 0.10785204412719014, 'fraction_numerical': 0.007138221933809215, 'mean_word_length': 3.8192620387742338, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2574039', 'n_tokens_mistral': 2535, 'n_tokens_neox': 2307, 'n_words': 882}