<jupyter_start><jupyter_text>2. Passwords should not be to shortIf we take a look at the first 12 users above we already see some bad passwords. But let's not get ahead of ourselves and start flagging passwords manually. What is the first thing we should check according to the NIST Special Publication 800-63B?  Verifiers SHALL require subscriber-chosen memorized secrets to be at least 8 characters in length.Ok, so the passwords of our users shouldn't be too short. Let's start by checking that!<jupyter_code># Calculating the lengths of users' passwords
users$length <- nchar(users$password)
users$length
# Flagging the users with too short passwords
users$too_short <- users$length<8
users$too_short
# Counting the number of users with too short passwords
# .... YOUR CODE FOR TASK 2 ....
sum(users$too_short)
# Taking a look at the 12 first rows
# .... YOUR CODE FOR TASK 2 ....
head(users, 12)<jupyter_output><empty_output><jupyter_text>3.  Common passwords people useAlready this simple rule flagged a couple of offenders among the first 12 users. Next up in Special Publication 800-63B is the rule that  verifiers SHALL compare the prospective secrets against a list that contains values known to be commonly-used, expected, or compromised.    Passwords obtained from previous breach corpuses.  Dictionary words.  Repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).  Context-specific words, such as the name of the service, the username, and derivatives thereof.  We're going to check these in order and start with Passwords obtained from previous breach corpuses, that is, websites where hackers have leaked all the users' passwords. As many websites don't follow the NIST guidelines and encrypt passwords there now exist large lists of the most popular passwords. Let's start by loading in the 10,000 most common passwords which I've taken from here.<jupyter_code># Reading in the top 10000 passwords
common_passwords <- read_lines("datasets/10_million_password_list_top_10000.txt")

# Taking a look at the top 100
# .... YOUR CODE FOR TASK 3 ....
head(common_passwords, 100)<jupyter_output><empty_output><jupyter_text>4.  Passwords should not be common passwordsThe list of passwords was ordered, with the most common passwords first, and so we shouldn't be surprised to see passwords like 123456 and qwerty above. As hackers also have access to this list of common passwords, it's important that none of our users use these passwords!Let's flag all the passwords in our user database that are among the top 10,000 used passwords.<jupyter_code># Flagging the users with passwords that are common passwords
users$common_password <- users$password %in% common_passwords

# Counting the number of users using common passwords
# .... YOUR CODE FOR TASK 4 ....
sum(users$common_password)
# Taking a look at the 12 first rows
# .... YOUR CODE FOR TASK 4 ....
head(users, 12)<jupyter_output><empty_output><jupyter_text>5. Passwords should not be common wordsAy ay ay! It turns out many of our users use common passwords, and of the first 12 users there are already two. However, as most common passwords also tend to be short, they were already flagged as being too short. What is the next thing we should check?  Verifiers SHALL compare the prospective secrets against a list that contains [...] dictionary words.This follows the same logic as before: It is easy for hackers to check users' passwords against common English words and therefore common English words make bad passwords. Let's check our users' passwords against the top 10,000 English words from Google's Trillion Word Corpus.<jupyter_code># Reading in a list of the 10000 most common words
words <- read_lines("datasets/google-10000-english.txt")

# Flagging the users with passwords that are common words
users$common_word <- users$password %in% words

# Counting the number of users using common words as passwords
# .... YOUR CODE FOR TASK 5 ....
sum(users$common_word)

# Taking a look at the 12 first rows
# .... YOUR CODE FOR TASK 5 ....
head(users, 12)<jupyter_output><empty_output><jupyter_text>6. Passwords should not be your nameIt turns out many of our passwords were common English words too! Next up on the NIST list:  Verifiers SHALL compare the prospective secrets against a list that contains [...] context-specific words, such as the name of the service, the username, and derivatives thereof.Ok, so there are many things we could check here. One thing to notice is that our users' usernames consist of their first names and last names separated by a dot. For now, let's just flag passwords that are the same as either a user's first or last name.<jupyter_code># Extracting first and last names into their own columns
users$first_name <- str_extract(users$user_name, "^\\w+")
users$last_name <- str_extract(users$user_name, "\\w+$")

# Flagging the users with passwords that matches their names
users$uses_name <- users$password == users$first_name  | users$password == users$last_name

# Counting the number of users using names as passwords
# .... YOUR CODE FOR TASK 6 ....
sum(users$uses_name)
# Taking a look at the 12 first rows
# .... YOUR CODE FOR TASK 6 ....
head(users, 12)<jupyter_output><empty_output><jupyter_text>7. Passwords should not be repetitive<NAME> (user number 12 above), what where you thinking!? Ok, so the last thing we are going to check is a bit tricky:  verifiers SHALL compare the prospective secrets [so that they don't contain] repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).This is tricky to check because what is repetitive is hard to define. Is 11111 repetitive? Yes! Is 12345 repetitive? Well, kind of. Is 13579 repetitive? Maybe not..? To check for repetitiveness can be arbitrarily complex, but here we're only going to do something simple. We're going to flag all passwords that contain 4 or more repeated characters.<jupyter_code># Splitting the passwords into vectors of single characters
split_passwords <- str_split(users$password, "")

# Picking out the max number of repeat characters for each password
users$max_repeats <- sapply(split_passwords, function(split_password) {
    # .... YOUR CODE FOR TASK 7 ....
    rle_password <- rle(split_password)
    max(rle_password$lengths)
})


# Flagging the passwords with >= 4 repeats
users$too_many_repeats <- users$max_repeats>=4

# Taking a look at the users with too many repeats
# .... YOUR CODE FOR TASK 7 ....
users$too_many_repeats<jupyter_output><empty_output><jupyter_text>8. All together now!Now we have implemented all the basic tests for bad passwords suggested by NIST Special Publication 800-63B! What's left is just to flag all bad passwords and maybe to send these users an e-mail that strongly suggests they change their password.<jupyter_code># Flagging all passwords that are bad
users$bad_password <- users$password %in% users$too_short | users$common_password | users$common_word | users$uses_name | users$too_many_repeats

# Counting the number of bad passwords
# .... YOUR CODE FOR TASK 8 ....
sum(users$bad_password)
# Looking at the first 100 bad passwords
# .... YOUR CODE FOR TASK 8 ....
head(users$bad_password, 100)<jupyter_output><empty_output><jupyter_text>9. Otherwise, the password should be up to the userIn this notebook, we've implemented the password checks recommended by the NIST Special Publication 800-63B. It's certainly possible to better implement these checks, for example, by using a longer list of common passwords. Also note that the NIST checks in no way guarantee that a chosen password is good, just that it's not obviously bad.Apart from the checks we've implemented above the NIST is also clear with what password rules should not be imposed:  Verifiers SHOULD NOT impose other composition rules (e.g., requiring mixtures of different character types or prohibiting consecutively repeated characters) for memorized secrets. Verifiers SHOULD NOT require memorized secrets to be changed arbitrarily (e.g., periodically).So the next time a website or app tells you to "include both a number, symbol and an upper and lower case character in your password" you should send them a copy of NIST Special Publication 800-63B.<jupyter_code># Enter a password that passes the NIST requirements
# PLEASE DO NOT USE AN EXISTING PASSWORD HERE
new_password <- "<PASSWORD>"<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06696214662194537, 'fraction_numerical': 0.018327743172017248, 'mean_word_length': 4.7420907840440165, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '68791', 'license': 'MIT', 'n_tokens_mistral': 2428, 'n_tokens_neox': 2120, 'n_words': 1130, 'path': 'Bad passwords and the NIST guidelines/notebook.ipynb', 'repo_name': 'pritamnegi/Bad-passwords-and-the-NIST-guidelines', 'size': '8356'}