<commit_before>---
title: "Computing TF-IDF with Spark and Clojure"
layout: article
description: "Tutorial showing a complete sample project using Gorillalabs Sparkling to compute Term Frequency / Inverse Document Frequency"
---


## About this guide

This guide assumes, that you have basic knowledge of Sparkling, e.g. by reading the [getting started guide](/sparkling/articles/getting_started.html). It will walk you through a complete (albeit small) project. The task fulfilled in the project will compute  [tf-idf](https://en.wikipedia.org/wiki/Tf-idf) for a set of documents. Tf-idf stands for Term-frequency / inverse document frequency and is used to determine relevance of a document in a given corpus for a given query. Thus, given a set of documents, our project will yield a term vector for each document.

The code for this guide can be found in the master branch of our ["getting started repo"](https://github.com/gorillalabs/sparkling-getting-started/). However, you should be able to setup the complete project simply by following this guide.

 * [Basic explanation of Tf-idf](#tf-idf)
 * [Setting up a project with leinigen](#new-project)
 * [(Insta)REPL-driven development](#REPL)
 * [Tests for your pure Clojure functions](#tests)
 * [Dealing with with Scalas `Tuple2` from Apache Spark](#tuple)
 * [Tests for your Spark units](#spark-tests)


 * [parallelizing data to a Spark Dataset](#rdds)
 * [reading data from a file into a Spark Dataset](#external)
 * [performing transformations on the Datasets](#rdd-operations)
 * [working with key-value-pairs](#keyvalue)
 * [calling actions on the Datasets.](#rdd-actions)


##<a name="tf-idf"/> What is tf-idf?

TF-IDF (term frequency-inverse document frequency) is a way to score the importance of terms in a document based on how frequently they appear across a collection of documents (corpus). The tf-idf weight of a term in a document is the product of its `tf` weight:

`tf(t, d) = (number of times term t appears in document d) / (total number of terms in document d)`

and its `idf` weight:

`idf(t) = ln(total number of documents in corpus / (1 + number of documents with term t))`

##<a name="new-project"/> Setting up a new project using lein

Make sure you have [Leiningen](http://leiningen.org/) installed, I'm currently using version 2.5.1.

    ➜  tmp  lein version
    Leiningen 2.5.1 on Java 1.7.0_60 Java HotSpot(TM) 64-Bit Server VM

Now, create a new project named "tf-idf".

    ➜  tmp  lein new tf-idf
    Generating a project called tf-idf based on the 'default' template.
    The default template is intended for library projects, not applications.
    To see other templates (app, plugin, etc), try `lein help new`.

Now, open that project in your IDE of choice. Personally, I use IntelliJ/Cursive, but for the sake of this tutorial, let's use [Light Table](http://lighttable.com/), as a easy-going common denominator. We start by editing `project.clj`, adding a dependency to `[gorillalabs/sparkling 1.1.0]`.

The file should now look like this:

```Clojure<commit_msg>try to fix formatting<commit_after>---
title: "Computing TF-IDF with Spark and Clojure"
layout: article
description: "Tutorial showing a complete sample project using Gorillalabs Sparkling to compute Term Frequency / Inverse Document Frequency"
---


## About this guide

This guide assumes, that you have basic knowledge of Sparkling, e.g. by reading the [getting started guide](/sparkling/articles/getting_started.html). It will walk you through a complete (albeit small) project. The task fulfilled in the project will compute  [tf-idf](https://en.wikipedia.org/wiki/Tf-idf) for a set of documents. Tf-idf stands for Term-frequency / inverse document frequency and is used to determine relevance of a document in a given corpus for a given query. Thus, given a set of documents, our project will yield a term vector for each document.

The code for this guide can be found in the master branch of our ["getting started repo"](https://github.com/gorillalabs/sparkling-getting-started/). However, you should be able to setup the complete project simply by following this guide.

 * [Basic explanation of Tf-idf](#tf-idf)
 * [Setting up a project with leinigen](#new-project)
 * [(Insta)REPL-driven development](#REPL)
 * [Tests for your pure Clojure functions](#tests)
 * [Dealing with with Scalas `Tuple2` from Apache Spark](#tuple)
 * [Tests for your Spark units](#spark-tests)


 * [parallelizing data to a Spark Dataset](#rdds)
 * [reading data from a file into a Spark Dataset](#external)
 * [performing transformations on the Datasets](#rdd-operations)
 * [working with key-value-pairs](#keyvalue)
 * [calling actions on the Datasets.](#rdd-actions)


##<a name="tf-idf"/> What is tf-idf?

TF-IDF (term frequency-inverse document frequency) is a way to score the importance of terms in a document based on how frequently they appear across a collection of documents (corpus). The tf-idf weight of a term in a document is the product of its `tf` weight:

`tf(t, d) = (number of times term t appears in document d) / (total number of terms in document d)`

and its `idf` weight:

`idf(t) = ln(total number of documents in corpus / (1 + number of documents with term t))`

##<a name="new-project"/> Setting up a new project using lein

Make sure you have [Leiningen](http://leiningen.org/) installed, I'm currently using version 2.5.1.

    ➜  tmp  lein version
    Leiningen 2.5.1 on Java 1.7.0_60 Java HotSpot(TM) 64-Bit Server VM

Now, create a new project named "tf-idf".

    ➜  tmp  lein new tf-idf
    Generating a project called tf-idf based on the 'default' template.
    The default template is intended for library projects, not applications.
    To see other templates (app, plugin, etc), try `lein help new`.

Now, open that project in your IDE of choice. Personally, I use IntelliJ/Cursive, but for the sake of this tutorial, let's use [Light Table](http://lighttable.com/), as a easy-going common denominator. We start by editing `project.clj`, adding a dependency to `[gorillalabs/sparkling 1.1.0]`.

The file should now look like this:

```clojure
{'fraction_non_alphanumeric': 0.09432114882506527, 'fraction_numerical': 0.00587467362924282, 'mean_word_length': 4.211734693877551, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5902988', 'n_tokens_mistral': 1872, 'n_tokens_neox': 1737, 'n_words': 828}