<commit_before># node-tika #

Provides text extraction, metadata extraction, mime-type detection, text-encoding detection and language detection. All via a native Java bridge with the Apache Tika content-analysis toolkit. Bundles [Tika 1.6](http://tika.apache.org/1.6/index.html).

[![Build Status](https://travis-ci.org/mattcg/node-tika.png?branch=master)](https://travis-ci.org/mattcg/node-tika)

Depends on [node-java](https://github.com/joeferner/node-java), which itself requires the JDK and Python 2 (not 3) to compile.

Requires JDK 7. Run `node version` to check the version that `node-java` is using. If the wrong version is reported even if you installed JDK 1.7, make sure `JAVA_HOME` is set to the correct path then delete `node_modules/java` and rerun `npm install`.

## API ##

```javascript
var tika = require('tika');
```

All methods that accept a `uri` parameter accept relative or absolute file paths and `http:` or `https:` URLs.

### tika.extract(uri, [contentType,] cb) ###

Extract both text and metadata from a file. Content type is optional but would help Tika in some cases.

```javascript
tika.extract('test/data/file.pdf', function(err, text, meta) {
	assert.equal(text.trim(), 'Just some text.');
	assert.deepEqual(meta.producer, 'LibreOffice 4.1');
});
```

### tika.text(uri, [contentType,] cb) ###

Extract text from a file.

```javascript
tika.text('test/data/file.pdf', function(err, text) {
	assert.equal(text.trim(), 'Just some text.');
});
```

### tika.meta(uri, [contentType,] cb) ###

Extract metadata from a file. Returns an object with names as keys.

```javascript
tika.meta('test/data/file.pdf', function(err, meta) {
	assert.deepEqual(meta.producer, 'LibreOffice 4.1');
});
```

### tika.type(uri, [withCharset,] cb) ###

Detect the content-type of a file.

```javascript
tika.type('test/data/file.pdf', function(err, contentType) {
	assert.equal(contentType, 'application/pdf');
});
```

The `withCharset` parameter defaults to `false`. If set to `true`, then the charset will be appended to the mime-type.

```javascript
tika.type('test/data/file.txt', true, function(err, contentType) {
	assert.equal(contentType, 'text/plain; charset=ISO-8859-1');
});
```

### tika.charset(uri, cb) ###

Detect the character set (text encoding) of a file.

```javascript
tika.charset('test/data/file.txt', true, function(err, charset) {
	assert.equal(charset, 'ISO-8859-1');
});
```

### tika.language(string, cb) ###

Detect the language a given string is written in.

```javascript
tika.language('This is just some text in English.', function(err, language, reasonablyCertain) {
	assert.equal(language, 'en');
});
```
<commit_msg>Update docs<commit_after># node-tika #

Provides text extraction, metadata extraction, mime-type detection, text-encoding detection and language detection. All via a native Java bridge with the Apache Tika content-analysis toolkit. Bundles [Tika 1.6](http://tika.apache.org/1.6/index.html).

[![Build Status](https://travis-ci.org/mattcg/node-tika.png?branch=master)](https://travis-ci.org/mattcg/node-tika)

Depends on [node-java](https://github.com/joeferner/node-java), which itself requires the JDK and Python 2 (not 3) to compile.

Requires JDK 7. Run `node version` to check the version that `node-java` is using. If the wrong version is reported even if you installed JDK 1.7, make sure `JAVA_HOME` is set to the correct path then delete `node_modules/java` and rerun `npm install`.

## API ##

```javascript
var tika = require('tika');
```

All methods that accept a `uri` parameter accept relative or absolute file paths and `http:` or `https:` URLs.

### tika.extract(uri, [contentType,] cb) ###

Extract both text and metadata from a file. Content type is optional but would help Tika in some cases.

```javascript
tika.extract('test/data/file.pdf', function(err, text, meta) {
	console.log(text); // Logs 'Just some text'.
	console.log(meta.producer); // Logs 'LibreOffice 4.1'.
});
```

We can even extract directly from the Web.

```javascript
tika.extract('http://www.ohchr.org/EN/UDHR/Documents/UDHR_Translations/eng.pdf', function(err, text, meta) {
	// ...
});
```

### tika.text(uri, [contentType,] cb) ###

Extract text from a file.

```javascript
tika.text('test/data/file.pdf', function(err, text) {
	console.log(text);
});
```

### tika.meta(uri, [contentType,] cb) ###

Extract metadata from a file. Returns an object with names as keys.

```javascript
tika.meta('test/data/file.pdf', function(err, meta) {
	console.log(meta.producer); // Logs 'LibreOffice 4.1'.
});
```

### tika.type(uri, [withCharset,] cb) ###

Detect the content-type of a file.

```javascript
tika.type('test/data/file.pdf', function(err, contentType) {
	console.log(contentType); // Logs 'application/pdf'.
});
```

The `withCharset` parameter defaults to `false`. If set to `true`, then the charset will be appended to the mime-type.

```javascript
tika.type('test/data/file.txt', true, function(err, contentType) {
	console.log(contentType); // Logs 'text/plain; charset=ISO-8859-1'.
});
```

### tika.charset(uri, cb) ###

Detect the character set (text encoding) of a file.

```javascript
tika.charset('test/data/file.txt', true, function(err, charset) {
	console.log(charset); // Logs 'ISO-8859-1'.
});
```

### tika.language(string, cb) ###

Detect the language a given string is written in.

```javascript
tika.language('This is just some text in English.', function(err, language, reasonablyCertain) {
	console.log(language); // Logs 'en'.
	console.log(reasonablyCertain); // Logs true or false.
});
```

{'fraction_non_alphanumeric': 0.16361679224973089, 'fraction_numerical': 0.008252601363473269, 'mean_word_length': 4.671414038657172, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4498603', 'n_tokens_mistral': 1913, 'n_tokens_neox': 1829, 'n_words': 557}