<commit_before>---
layout: lesson
title: Theoretical Background - Hashing
category: git-noms-data-models
tags:
---

## Prerequisites

Form a pair with another participant, and make sure that at least one of you is running Linux or Mac OS X.  You should already know how to use [redirection and piping on the command line](../redirection-and-piping).

## Goals (Learning Objectives)

After doing this tutorial you will know:

- What hashing means
- What features a hash function usually has
- A couple purposes for which hashing is used
- The names of a couple widely-used cryptographic hash functions

## Activities

Open up your command line.  Each activity below consists of sets of commands to execute on the command line, interspersed with questions and commentary to  discuss with your partner.

### Activity 1: Deterministic

First, let's create a simple text file:

```bash
cat > test.txt
```

Now enter the following text and end with a blank line and an EOF (end-of-file signal; `^D`).  Pay careful attention to the capitalization, spacing, and punctuation!

```
One fish, two fish
Red fish, blue fish
```

We can get a computational fingerprint for this file, called a "hash", by using the `md5` command.  (It may be called `md5sum` on your computer.)

```bash
md5 test.txt
```

On my computer, this command prints out:

```
MD5 (test.txt) = ab07f7960958aa4bf2ec2409089e9e13
```

On your computer, the output may be formatted a little differently, but the *hash*, the `ab07f...` stuff, should be exactly the same.  A "hash" is a value computed by a *hash function*, and this page is all about learning the characteristics of hash functions.  A hash function takes arbitrary data as input, and computes a hash as its output:

```
f : data -> hash
```

Many different hash functions have been invented; the `md5` program runs a hash function called "MD5".  Every hash function is *deterministic*: given the same input, it will always compute the same result (the hash), no matter who runs it, when, on what computer.  (Or even if they compute it by hand!)  Even though the hash looks totally random, I know that your computer will produce the same one that I did for the same input file, because we were both running the same hash function: MD5.  This is the first feature of a hash function:

> A hash function is **deterministic**: the same input always produces the same output.


### Activity 2: Fixed-Size Output

Let's create a bigger file to work with:<commit_msg>Mention SHA-1<commit_after>---
layout: lesson
title: Theoretical Background - Hashing
category: git-noms-data-models
tags:
---

## Prerequisites

Form a pair with another participant, and make sure that at least one of you is running Linux or Mac OS X.  You should already know how to use [redirection and piping on the command line](../redirection-and-piping).

## Goals (Learning Objectives)

After doing this tutorial you will know:

- What hashing means
- What features a hash function usually has
- A couple purposes for which hashing is used
- The names of a couple widely-used cryptographic hash functions

## Activities

Open up your command line.  Each activity below consists of sets of commands to execute on the command line, interspersed with questions and commentary to  discuss with your partner.

### Activity 1: Deterministic

First, let's create a simple text file:

```bash
cat > test.txt
```

Now enter the following text and end with a blank line and an EOF (end-of-file signal; `^D`).  Pay careful attention to the capitalization, spacing, and punctuation!

```
One fish, two fish
Red fish, blue fish
```

We can get a computational fingerprint for this file, called a "hash", by using the `md5` command.  (It may be called `md5sum` on your computer.)

```bash
md5 test.txt
```

On my computer, this command prints out:

```
MD5 (test.txt) = ab07f7960958aa4bf2ec2409089e9e13
```

On your computer, the output may be formatted a little differently, but the *hash*, the `ab07f...` stuff, should be exactly the same.  A "hash" is a value computed by a *hash function*, and this page is all about learning the characteristics of hash functions.  A hash function takes arbitrary data as input, and computes a hash as its output:

```
f : data -> hash
```

Many different hash functions have been invented; the `md5` program runs a hash function called "MD5".  Every hash function is *deterministic*: given the same input, it will always compute the same result (the hash), no matter who runs it, when, on what computer.  (Or even if they compute it by hand!)  Even though the hash looks totally random, I know that your computer will produce the same one that I did for the same input file, because we were both running the same hash function: MD5.  This is the first feature of a hash function:

> A hash function is **deterministic**: the same input always produces the same output.

Though we'll use MD5 for the rest of this lesson, another prominent hash function is called SHA-1.  Given the same input, SHA-1 generally produces a *different* hash than MD5 does.  But SHA-1 has all the same general features of a hash function that we'll learn about by studying MD5.


### Activity 2: Fixed-Size Output

Let's create a bigger file to work with:
{'fraction_non_alphanumeric': 0.07074752097635392, 'fraction_numerical': 0.013539282990083905, 'mean_word_length': 4.127077223851417, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2645238', 'n_tokens_mistral': 1476, 'n_tokens_neox': 1381, 'n_words': 811}