<commit_before># Strengthening Passwords to Defend Against John

In this exercise we will obtain the password of an unsuspecting user by "cracking" the hashed copy of that user's password. Then we will use a password manager to strengthen our own passwords in order to protect ourselves against the same sort of attacks. Our goal is to understand why and how using [password management software](https://en.wikipedia.org/wiki/Password_manager) makes having a digital life both safer and easier at the same time.

In other words, you will perform a *[password cracking](https://en.wikipedia.org/wiki/Password_cracking) attack* and learn how to stop password crackers from cracking your own passwords.

1. [Objectives](#objectives)
1. [Scenario](#scenario)
1. [Prerequisites](#prerequisites)
1. [Set up](#set-up)
    * [John the Ripper installation on Windows](#jtr-installation-on-windows)
    * [John the Ripper installation on macOS](#jtr-installation-on-macos)
    * [John the Ripper installation on GNU/Linux](#jtr-installation-on-gnulinux)
1. [Practice](#practice)
    1. [Introduction](#introduction)
    1. [Sanity check](#sanity-check)
1. [Discussion](#discussion)
    * [Technical errors in the Mr. Robot scene](#technical-errors-in-the-mr-robot-scene)
    * [`passwd` versus `shadow` files](#passwd-versus-shadow-files)
    * [Hash string formats](#hash-string-formats)
    * [Salted versus unsalted hashes](#salted-versus-unsalted-hashes)
    * [Calculating password strength](#calculating-password-strength)
1. [Additional references](#additional-references)

# Objectives

When you complete this exercise, you will have acquired the following capabilities:

* The ability to make and use passwords for your user accounts that are practically invincible against many attackers.
* The ability to replace old passwords with strong new passwords easily, on whatever schedule you want.
* The ability to determine other users's original passwords, if theirs are weak, despite sophisticated attempts to protect them.
* The ability to gauge the approximate relative strength of a password using only "the naked eye."

# Scenario

[![Hacking Tyrell Wellick scene, using the `john` password cracker.](https://web.archive.org/web/20161213025246/https://hackertarget.com/mrrobot/wget-shellshock-john.png)](Mr.%20Robot%20S01E02%20-%20Hacking%20Tyrell%20Wellick.mp4?raw=true)

> Hacking <NAME>, you'd think it'd be tricky but it wasn't. Evil Corp's corporate mail servers? They haven't patched anything since Shellshock. He doesn't even use two-step verification. His password was just his wife's maiden name and Sweden's independence date, `66`. One "six" away from the obvious, but still. This is bad noob practice.

You are <NAME>, ordinary cybersecurity engineer by day, vigilante hacker by night. In order to uncover Evil Corp's next nefarious plot, you must hack into the corporate email account of their Senior Vice President of Technology, <NAME>. Fortunately for you, Evil Corp's mail servers run a webmail system vulnerable to an exploit called Shellshock, which has enabled you to obtain files listing the usernames and passwords of everyone who uses the system. However, the passwords are obfuscated using a one-way mathematical function called a *hash* (or *digest*).

In order to recover the password, you must "crack" its hashed equivalent, revealing the original. Once cracked, you may impersonate <NAME> to the Evil Corp mail server by using his username and password to log in as him, thereby gaining access to his emails. Moreover, to make sure that Tyrell Wellick is unable to hack you back, you must set all your online accounts to use "uncrackable" passwords to protect yourself against the very same attack.

# Prerequisites

To participate in this exercise, you must have:

* A computer running any modern version of:
    * Windows,
    * macOS, or
    * any flavor of the GNU/Linux operating system.
* An active Internet connection (for downloading the required tools in the [set up](#set-up) step; you do not need an Internet connection once you have the tools installed).

> :beginner: :computer: This exercise requires the use of a command line, or "terminal." If you don't know what that means, or if you do but you feel intimidated by that, consider spending an hour at [Codecademy's interactive tutorial, "Learn the Command Line"](https://www.codecademy.com/learn/learn-the-command-line) (for macOS or GNU/Linux users) or reviewing [Computer Hope's article, "How to use the Windows command line (DOS)"](http://www.computerhope.com/issues/chusedos.htm) (for Windows users). You don't need to complete the whole tutorial or article to understand this exercise, but it will dramatically improve your comprehension of this exercise's mechanics.

# Set up

In addition to your laptop or desktop computer, you will need to acquire the following tools.

* [KeePassX](https://www.keepassx.org/)
* [<NAME> password cracker](http://www.openwall.com/john/), often written as *JtR*

There are pre-built versions of KeePassX for both Windows and macOS available for download from the [KeePassX downloads page](https://www.keepassx.org/downloads). For GNU/Linux users, KeePassX is ordinarily provided as a standard software package directly from your operating system's distributor. Use your system's default package manager to install it.

> :bulb: If you already use a different password manager or prefer to choose one of the many different [recommended password managers](https://prism-break.org/en/all/#password-managers), you can still follow along with the overwhelming majority of this exercise. The fundamental features of all password managers are, generally speaking, equivalent to one another. What matters is that you use one, not necessarily which one you use.

There are several different procedures to install John the Ripper available to you. If you're comfortable doing so, you can follow the above link, download JtR's source code, and install the software configuration of your choosing. (See [Beginner's Guide to Installing from Source](http://moi.vonos.net/linux/beginners-installing-from-source/) if you're not comfortable, but want to try that anyway.) The rest of this section will describe more common approaches to installing <NAME> Ripper in a little more detail.

## JtR installation on Windows

Two pre-built Windows versions of <NAME> Ripper are available. One is an "official" build, compiled by the tool's original developers, and the second is a "community-enhanced" build, which bundles (literally) hundreds of additional features written by volunteers into the application itself. For the purposes of this exercise, it does not matter which version you choose to get, although the community-enhanced version is recommended. To install <NAME>, take the following steps:

**Do this:**

1. Go to [the John the Ripper homepage](http://www.openwall.com/john/).
1. Download the Windows version of your choice. At the time of this writing, these are:
    * [<NAME> Ripper 1.8.0-jumbo-1 community-enhanced version for Windows](http://www.openwall.com/john/j/john180j1w.zip) (recommended)
    * [John the Ripper 1.7.9 official free version for Windows](http://www.openwall.com/john/h/john179w2.zip)
1. Unzip the file. Usually, this is as simple as right-clicking on it and selecting *Extract All* from the context-menu. (See [the Windows Help "Compress and uncompress files (zip files)" article](https://support.microsoft.com/en-us/help/14200/windows-compress-uncompress-zip-files).)
    * This will produce a folder, inside of which is another folder named `run`. Inside *that* folder is a file called `john` (or `john.exe` if your computer is [set to display file extensions](http://www.computerhope.com/issues/ch000573.htm)). That file is the main John the Ripper application. Unlike most Windows applications, this program must be run [using the Windows Command Prompt](http://www.computerhope.com/issues/chusedos.htm); you cannot simply double-click it. However, `john.exe` is now successfully "installed."

## JtR installation on macOS

A pre-built macOS version of John the Ripper is available for modern Apple's Macintosh computers, that is, Macs with an Intel processor. (See [About System Information on your Mac](https://support.apple.com/en-us/HT203001) to learn how to find out what processor your Mac has.) If you have such a Mac, then to install John the Ripper, take the following steps.

**Do this:**

1. Go to [John the Ripper's macOS community-contributed builds page](http://download.openwall.net/pub/projects/john/contrib/macosx/).
1. Download the latest version available. (As of this writing, that is [`john-1.7.9-jumbo-7-macosx-Intel-2.zip`](http://download.openwall.net/pub/projects/john/contrib/macosx/john-1.7.9-jumbo-7-macosx-Intel-2.zip).)
1. Unzip the file. Usually, this is as simple as double-clicking on it.
    * This will produce a folder, inside of which is another folder named `run`. Inside *that* folder is a file called `john`. That file is the main John the Ripper application. Unlike a classic macOS application, this program must be run [using the Terminal](http://guides.macrumors.com/Terminal); you cannot simply double-click it. However, `john` is now successfully "installed."

## JtR installation on GNU/Linux

> :construction: TK-TODO: These instructions are currently missing. Please add them!

# Practice

We'll begin by ensuring you have successfully completed the [set up](#set-up) steps. This process will also introduce the fundamentals that you need to understand to complete the rest of the exercise.

## Introduction

Recovering passwords is generally accomplished in one of two ways. The approach we'll be taking in this exercise, *offline hash cracking*, is by far the more common of the two. We say it is "offline" because we do it entirely on our own computer, without needing an active network connection, after already acquiring the "hashed" password(s) through some other means.

> :beginner: If you've never encountered a "hash" in this context before, this exercise might seem complex already. Searching the Internet for "hash" probably reveals more pages about cannabis than computer security at first, but adding "computer" to your searches will get you [many](http://www.webopedia.com/TERM/H/hashing.html), [many](http://unixwiz.net/techtips/iguide-crypto-hashes.html), [many](https://en.wikipedia.org/wiki/Hash_function) more [relevant results](http://www.computerhope.com/jargon/h/hashing.htm). Another tip for beginners, especially if you've found the regular Wikipedia article impenetrable, is to check if there is a "simple" version of the Wikipedia page. In this case, you're in luck: the [Simple Wikipedia article for "Cryptographic hash function"](https://simple.wikipedia.org/wiki/Cryptographic_hash_function) is relatively straightforward by comparison.
> 
> In the end, a "hash" in the context of computer security is simply a value that ("cryptographically") represents some other value. The idea, in theory, is that two different original values will never be represented by the same two ultimate values after they have been "hashed," and that it is infeasible to recover the original value from the hashed value. ([Watch these few minutes from "Crypto 101"](https://www.youtube.com/watch?v=3rmCGsCYJF8&t=20m6s) for a further explanation and some pictures depicting this.) In practice, however, weaknesses in cryptographic hash functions (whether by design due to flaws in their algorithm, or by mistakes introduced through their actual implementation) sometimes result in two different values being hashed to the same value. When this happens, the hash function is said to be "cryptographically broken," because if you can hash two different values and get the same resulting value (a situation known as a *hash collision*), then there is little point to the hash from a security perspective in the first place.
> 
> For the purposes of this exercise, we will assume there are no flaws in the hashing algorithm. In reality, this "sanity check" hash value uses a cryptographically broken algorithm called [SHA-1](https://en.wikipedia.org/wiki/SHA-1). To learn more about how SHA-1 is broken, and how to construct a *hash collision attack* yourself, read [Boston Key Party 2017 CTF: Prudentialv2](https://github.com/meitar/CTF/blob/master/2017/BKP/cloud/Prudentialv2/README.md).

The alternative to offline hash cracking is *online password guessing*. We say it's "online" because it's as simple as repeatedly attempting to log in to an active, remote system, such as an email service or online banking website. Of course, most websites don't allow visitors to try logging in to an account too many times. After some number of failed guesses, they will usually lock out the account. Besides, it would be unbearably slow to have to reload the login page over and over again after each guess. Therein lies the key concept: speed.

A password can, theoretically, be any sequence of characters at all. In practice, many systems place limits on what they consider valid passwords, or limit the length of the password. For instance, many websites require your password to be "between 4 to 12 characters, and must contain only letters, numbers, or punctuation marks," or something like that. The set of possibilities that a given password might be is called a *search space*. Given infinite time, it would be a simple matter of trying every single possibility until one of your guesses is right. This technique is called a *brute force search*. In practice, however, we never have anything close to infinite time. If it took us, say, two hundred years to find <NAME>'s password, we would be long dead, Evil Corp's nefarious plan will surely have gone unopposed, and we have still not even gotten close to infinite time!

This means, to be successful, we only have two techniques available to us:

* Guess more passwords faster.
* Make smarter guesses sooner.

Guessing more passwords faster is pretty intuitive: the faster we can make guesses, the less time it will take to perform an exhaustive, brute-force search. This is simply a function of what computer you have. The better, faster, stronger, more expensive your computers are, the more guesses you can make per second. Making smarter guesses sooner involves much more subtlety, so that's where we'll be focusing the majority of our time in this lab.

## Sanity check

First, let's make sure you successfully installed John the Ripper (JtR) and that its `john` program is working correctly. We'll do this by giving `john` two files. One file contains a fictional username and hashed password combination. The second file will contain a (correct) "guess" of the un-hashed password. Given these two files, `john` should be able to apply the password guess in the one file to the hashed version of the password in the other file and tell us that the guess is correct.

> :beginner: It's very easy to create a hashed version of a password, or any content, such as an arbitrary file. To make the hash in this sanity check, I used [the `shasum(1)` command](https://linux.die.net/man/1/shasum) on a GNU/Linux system:
> 
> ```sh
> $ echo -n "Sup3rs3kr3tP@24431w0rd" | shasum --algorithm 1
> ced91977849c44fd009ba437c14c1b74f632fae6  -
> ```
> 
> Similarly, Windows users can use [Notepad](https://en.wikipedia.org/wiki/Microsoft_Notepad) and a utility called the [File Checksum Integrity Verifier](http://support.microsoft.com/kb/841290).

The file containing the username and hashed password pair is called `sanitycheck.password.txt`. It contains one line that looks like this:

```
fsociety:ced91977849c44fd009ba437c14c1b74f632fae6
```

On the left, at the start of the file, is a username. In this case, that's `fsociety`. Then there is the *field separator* character, a colon (`:`). Finally, there is the *hash value*. That hash is the obfuscated version of the password we need to guess.

The file containing our guesses is called `sanitycheck.wordlist.txt`. It contains four lines and looks like this:

```
randomGuess
alsoWrong
Sup3rs3kr3tP@24431w0rd
someotherword
```

Each guess is written on its own line. We call each guess a *word*, even if there are multiple human-language words on a single line. We call the whole file a *wordlist*, or sometimes a *dictionary* (because it is a "list of words," of course).

When we give the wordlist to `john`, what we're doing is asking `john` to hash each of these "words" one at a time and then compare the resulting hash value to the hash value stored in the password file. When `john` finds a word in the wordlist that hashes to the same hash value as the hashed password, it will tell us that it found a match. When we find a matching value, we say we have "*cracked* the hash."

If `john` is working correctly, we can expect that the third word in the wordlist, `Sup3rs3kr3tP@24431w0rd`, will result in a match. If that guess were missing (as it is in the `sanitycheck.no-crack.wordlist.txt` file), we would expect to see `john` report its failure to find any matches.

To perform the sanity check, we'll use this command invocation:

```sh
./john --wordlist=sanitycheck.wordlist.txt sanitycheck.password.txt
```

Breaking this invocation down:

* The first part, `./john`, runs the program `john` in the current directory (`./`).
* The first option, `--wordlist`, tells JtR to read its guesses from the file given by the value (`=`) of the option, in this case the contents of the file `sanitycheck.wordlist.txt`.
* The final argument, `sanitycheck.password.txt`, is the file containing the hashed passwords we want to crack.<commit_msg>Note Windows-specific directory syntax.<commit_after># Strengthening Passwords to Defend Against John

In this exercise we will obtain the password of an unsuspecting user by "cracking" the hashed copy of that user's password. Then we will use a password manager to strengthen our own passwords in order to protect ourselves against the same sort of attacks. Our goal is to understand why and how using [password management software](https://en.wikipedia.org/wiki/Password_manager) makes having a digital life both safer and easier at the same time.

In other words, you will perform a *[password cracking](https://en.wikipedia.org/wiki/Password_cracking) attack* and learn how to stop password crackers from cracking your own passwords.

1. [Objectives](#objectives)
1. [Scenario](#scenario)
1. [Prerequisites](#prerequisites)
1. [Set up](#set-up)
    * [John the Ripper installation on Windows](#jtr-installation-on-windows)
    * [John the Ripper installation on macOS](#jtr-installation-on-macos)
    * [John the Ripper installation on GNU/Linux](#jtr-installation-on-gnulinux)
1. [Practice](#practice)
    1. [Introduction](#introduction)
    1. [Sanity check](#sanity-check)
1. [Discussion](#discussion)
    * [Technical errors in the Mr. Robot scene](#technical-errors-in-the-mr-robot-scene)
    * [`passwd` versus `shadow` files](#passwd-versus-shadow-files)
    * [Hash string formats](#hash-string-formats)
    * [Salted versus unsalted hashes](#salted-versus-unsalted-hashes)
    * [Calculating password strength](#calculating-password-strength)
1. [Additional references](#additional-references)

# Objectives

When you complete this exercise, you will have acquired the following capabilities:

* The ability to make and use passwords for your user accounts that are practically invincible against many attackers.
* The ability to replace old passwords with strong new passwords easily, on whatever schedule you want.
* The ability to determine other users's original passwords, if theirs are weak, despite sophisticated attempts to protect them.
* The ability to gauge the approximate relative strength of a password using only "the naked eye."

# Scenario

[![Hacking Tyrell Wellick scene, using the `john` password cracker.](https://web.archive.org/web/20161213025246/https://hackertarget.com/mrrobot/wget-shellshock-john.png)](Mr.%20Robot%20S01E02%20-%20Hacking%20Tyrell%20Wellick.mp4?raw=true)

> Hacking <NAME>, you'd think it'd be tricky but it wasn't. Evil Corp's corporate mail servers? They haven't patched anything since Shellshock. He doesn't even use two-step verification. His password was just his wife's maiden name and Sweden's independence date, `66`. One "six" away from the obvious, but still. This is bad noob practice.

You are <NAME>, ordinary cybersecurity engineer by day, vigilante hacker by night. In order to uncover Evil Corp's next nefarious plot, you must hack into the corporate email account of their Senior Vice President of Technology, <NAME>. Fortunately for you, Evil Corp's mail servers run a webmail system vulnerable to an exploit called Shellshock, which has enabled you to obtain files listing the usernames and passwords of everyone who uses the system. However, the passwords are obfuscated using a one-way mathematical function called a *hash* (or *digest*).

In order to recover the password, you must "crack" its hashed equivalent, revealing the original. Once cracked, you may impersonate <NAME> to the Evil Corp mail server by using his username and password to log in as him, thereby gaining access to his emails. Moreover, to make sure that <NAME> is unable to hack you back, you must set all your online accounts to use "uncrackable" passwords to protect yourself against the very same attack.

# Prerequisites

To participate in this exercise, you must have:

* A computer running any modern version of:
    * Windows,
    * macOS, or
    * any flavor of the GNU/Linux operating system.
* An active Internet connection (for downloading the required tools in the [set up](#set-up) step; you do not need an Internet connection once you have the tools installed).

> :beginner: :computer: This exercise requires the use of a command line, or "terminal." If you don't know what that means, or if you do but you feel intimidated by that, consider spending an hour at [Codecademy's interactive tutorial, "Learn the Command Line"](https://www.codecademy.com/learn/learn-the-command-line) (for macOS or GNU/Linux users) or reviewing [Computer Hope's article, "How to use the Windows command line (DOS)"](http://www.computerhope.com/issues/chusedos.htm) (for Windows users). You don't need to complete the whole tutorial or article to understand this exercise, but it will dramatically improve your comprehension of this exercise's mechanics.

# Set up

In addition to your laptop or desktop computer, you will need to acquire the following tools.

* [KeePassX](https://www.keepassx.org/)
* [<NAME> password cracker](http://www.openwall.com/john/), often written as *JtR*

There are pre-built versions of KeePassX for both Windows and macOS available for download from the [KeePassX downloads page](https://www.keepassx.org/downloads). For GNU/Linux users, KeePassX is ordinarily provided as a standard software package directly from your operating system's distributor. Use your system's default package manager to install it.

> :bulb: If you already use a different password manager or prefer to choose one of the many different [recommended password managers](https://prism-break.org/en/all/#password-managers), you can still follow along with the overwhelming majority of this exercise. The fundamental features of all password managers are, generally speaking, equivalent to one another. What matters is that you use one, not necessarily which one you use.

There are several different procedures to install <NAME> Ripper available to you. If you're comfortable doing so, you can follow the above link, download JtR's source code, and install the software configuration of your choosing. (See [Beginner's Guide to Installing from Source](http://moi.vonos.net/linux/beginners-installing-from-source/) if you're not comfortable, but want to try that anyway.) The rest of this section will describe more common approaches to installing John the Ripper in a little more detail.

## JtR installation on Windows

Two pre-built Windows versions of John the Ripper are available. One is an "official" build, compiled by the tool's original developers, and the second is a "community-enhanced" build, which bundles (literally) hundreds of additional features written by volunteers into the application itself. For the purposes of this exercise, it does not matter which version you choose to get, although the community-enhanced version is recommended. To install John the Ripper, take the following steps:

**Do this:**

1. Go to [the John the Ripper homepage](http://www.openwall.com/john/).
1. Download the Windows version of your choice. At the time of this writing, these are:
    * [John the Ripper 1.8.0-jumbo-1 community-enhanced version for Windows](http://www.openwall.com/john/j/john180j1w.zip) (recommended)
    * [John the Ripper 1.7.9 official free version for Windows](http://www.openwall.com/john/h/john179w2.zip)
1. Unzip the file. Usually, this is as simple as right-clicking on it and selecting *Extract All* from the context-menu. (See [the Windows Help "Compress and uncompress files (zip files)" article](https://support.microsoft.com/en-us/help/14200/windows-compress-uncompress-zip-files).)
    * This will produce a folder, inside of which is another folder named `run`. Inside *that* folder is a file called `john` (or `john.exe` if your computer is [set to display file extensions](http://www.computerhope.com/issues/ch000573.htm)). That file is the main John the Ripper application. Unlike most Windows applications, this program must be run [using the Windows Command Prompt](http://www.computerhope.com/issues/chusedos.htm); you cannot simply double-click it. However, `john.exe` is now successfully "installed."

## JtR installation on macOS

A pre-built macOS version of John the Ripper is available for modern Apple's Macintosh computers, that is, Macs with an Intel processor. (See [About System Information on your Mac](https://support.apple.com/en-us/HT203001) to learn how to find out what processor your Mac has.) If you have such a Mac, then to install John the Ripper, take the following steps.

**Do this:**

1. Go to [John the Ripper's macOS community-contributed builds page](http://download.openwall.net/pub/projects/john/contrib/macosx/).
1. Download the latest version available. (As of this writing, that is [`john-1.7.9-jumbo-7-macosx-Intel-2.zip`](http://download.openwall.net/pub/projects/john/contrib/macosx/john-1.7.9-jumbo-7-macosx-Intel-2.zip).)
1. Unzip the file. Usually, this is as simple as double-clicking on it.
    * This will produce a folder, inside of which is another folder named `run`. Inside *that* folder is a file called `john`. That file is the main John the Ripper application. Unlike a classic macOS application, this program must be run [using the Terminal](http://guides.macrumors.com/Terminal); you cannot simply double-click it. However, `john` is now successfully "installed."

## JtR installation on GNU/Linux

> :construction: TK-TODO: These instructions are currently missing. Please add them!

# Practice

We'll begin by ensuring you have successfully completed the [set up](#set-up) steps. This process will also introduce the fundamentals that you need to understand to complete the rest of the exercise.

## Introduction

Recovering passwords is generally accomplished in one of two ways. The approach we'll be taking in this exercise, *offline hash cracking*, is by far the more common of the two. We say it is "offline" because we do it entirely on our own computer, without needing an active network connection, after already acquiring the "hashed" password(s) through some other means.

> :beginner: If you've never encountered a "hash" in this context before, this exercise might seem complex already. Searching the Internet for "hash" probably reveals more pages about cannabis than computer security at first, but adding "computer" to your searches will get you [many](http://www.webopedia.com/TERM/H/hashing.html), [many](http://unixwiz.net/techtips/iguide-crypto-hashes.html), [many](https://en.wikipedia.org/wiki/Hash_function) more [relevant results](http://www.computerhope.com/jargon/h/hashing.htm). Another tip for beginners, especially if you've found the regular Wikipedia article impenetrable, is to check if there is a "simple" version of the Wikipedia page. In this case, you're in luck: the [Simple Wikipedia article for "Cryptographic hash function"](https://simple.wikipedia.org/wiki/Cryptographic_hash_function) is relatively straightforward by comparison.
> 
> In the end, a "hash" in the context of computer security is simply a value that ("cryptographically") represents some other value. The idea, in theory, is that two different original values will never be represented by the same two ultimate values after they have been "hashed," and that it is infeasible to recover the original value from the hashed value. ([Watch these few minutes from "Crypto 101"](https://www.youtube.com/watch?v=3rmCGsCYJF8&t=20m6s) for a further explanation and some pictures depicting this.) In practice, however, weaknesses in cryptographic hash functions (whether by design due to flaws in their algorithm, or by mistakes introduced through their actual implementation) sometimes result in two different values being hashed to the same value. When this happens, the hash function is said to be "cryptographically broken," because if you can hash two different values and get the same resulting value (a situation known as a *hash collision*), then there is little point to the hash from a security perspective in the first place.
> 
> For the purposes of this exercise, we will assume there are no flaws in the hashing algorithm. In reality, this "sanity check" hash value uses a cryptographically broken algorithm called [SHA-1](https://en.wikipedia.org/wiki/SHA-1). To learn more about how SHA-1 is broken, and how to construct a *hash collision attack* yourself, read [Boston Key Party 2017 CTF: Prudentialv2](https://github.com/meitar/CTF/blob/master/2017/BKP/cloud/Prudentialv2/README.md).

The alternative to offline hash cracking is *online password guessing*. We say it's "online" because it's as simple as repeatedly attempting to log in to an active, remote system, such as an email service or online banking website. Of course, most websites don't allow visitors to try logging in to an account too many times. After some number of failed guesses, they will usually lock out the account. Besides, it would be unbearably slow to have to reload the login page over and over again after each guess. Therein lies the key concept: speed.

A password can, theoretically, be any sequence of characters at all. In practice, many systems place limits on what they consider valid passwords, or limit the length of the password. For instance, many websites require your password to be "between 4 to 12 characters, and must contain only letters, numbers, or punctuation marks," or something like that. The set of possibilities that a given password might be is called a *search space*. Given infinite time, it would be a simple matter of trying every single possibility until one of your guesses is right. This technique is called a *brute force search*. In practice, however, we never have anything close to infinite time. If it took us, say, two hundred years to find <NAME>'s password, we would be long dead, Evil Corp's nefarious plan will surely have gone unopposed, and we have still not even gotten close to infinite time!

This means, to be successful, we only have two techniques available to us:

* Guess more passwords faster.
* Make smarter guesses sooner.

Guessing more passwords faster is pretty intuitive: the faster we can make guesses, the less time it will take to perform an exhaustive, brute-force search. This is simply a function of what computer you have. The better, faster, stronger, more expensive your computers are, the more guesses you can make per second. Making smarter guesses sooner involves much more subtlety, so that's where we'll be focusing the majority of our time in this lab.

## Sanity check

First, let's make sure you successfully installed John the Ripper (JtR) and that its `john` program is working correctly. We'll do this by giving `john` two files. One file contains a fictional username and hashed password combination. The second file will contain a (correct) "guess" of the un-hashed password. Given these two files, `john` should be able to apply the password guess in the one file to the hashed version of the password in the other file and tell us that the guess is correct.

> :beginner: It's very easy to create a hashed version of a password, or any content, such as an arbitrary file. To make the hash in this sanity check, I used [the `shasum(1)` command](https://linux.die.net/man/1/shasum) on a GNU/Linux system:
> 
> ```sh
> $ echo -n "Sup3rs3kr3tP@24431w0rd" | shasum --algorithm 1
> ced91977849c44fd009ba437c14c1b74f632fae6  -
> ```
> 
> Similarly, Windows users can use [Notepad](https://en.wikipedia.org/wiki/Microsoft_Notepad) and a utility called the [File Checksum Integrity Verifier](http://support.microsoft.com/kb/841290).

The file containing the username and hashed password pair is called `sanitycheck.password.txt`. It contains one line that looks like this:

```
fsociety:ced91977849c44fd009ba437c14c1b74f632fae6
```

On the left, at the start of the file, is a username. In this case, that's `fsociety`. Then there is the *field separator* character, a colon (`:`). Finally, there is the *hash value*. That hash is the obfuscated version of the password we need to guess.

The file containing our guesses is called `sanitycheck.wordlist.txt`. It contains four lines and looks like this:

```
randomGuess
alsoWrong
Sup3rs3kr3tP@24431w0rd
someotherword
```

Each guess is written on its own line. We call each guess a *word*, even if there are multiple human-language words on a single line. We call the whole file a *wordlist*, or sometimes a *dictionary* (because it is a "list of words," of course).

When we give the wordlist to `john`, what we're doing is asking `john` to hash each of these "words" one at a time and then compare the resulting hash value to the hash value stored in the password file. When `john` finds a word in the wordlist that hashes to the same hash value as the hashed password, it will tell us that it found a match. When we find a matching value, we say we have "*cracked* the hash."

If `john` is working correctly, we can expect that the third word in the wordlist, `Sup3rs3kr3tP@24431w0rd`, will result in a match. If that guess were missing (as it is in the `sanitycheck.no-crack.wordlist.txt` file), we would expect to see `john` report its failure to find any matches.

To perform the sanity check, we'll use this command invocation on GNU/Linux or macOS systems:

```sh
./john --wordlist=sanitycheck.wordlist.txt sanitycheck.password.txt
```

Windows users will need to reverse the slash, but otherwise can use the same invocation:

```sh
.\john --wordlist=sanitycheck.wordlist.txt sanitycheck.password.txt
```

Breaking this invocation down:

* The first part, `./john` (or `.\john` on Windows), runs the program `john` in the current directory (notated as `./` on GNU/Linux and macOS and `.\` on Windows). The rest of this walkthrough always uses the GNU/Linux and macOS style, so if you're following along on Windows, just remember to reverse the direction of the slash to mean "current directory."
* The first option, `--wordlist`, tells JtR to read its guesses from the file given by the value (`=`) of the option, in this case the contents of the file `sanitycheck.wordlist.txt`.
* The final argument, `sanitycheck.password.txt`, is the file containing the hashed passwords we want to crack.
{'fraction_non_alphanumeric': 0.07575715282623866, 'fraction_numerical': 0.011165387299371946, 'mean_word_length': 4.690279542566709, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 55, 'https://': 34, 'lorem ipsum': 0, 'www.': 26, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1225664', 'n_tokens_mistral': 9888, 'n_tokens_neox': 9183, 'n_words': 4896}