<jupyter_start><jupyter_text>This section introduces some of the most relevant aspects of working with Python for social scientists. This includes the different data types available and ways to modify them.   Strings Doroth<NAME> is credited with the expression, > Heterosexuality is not normal, it is just common. To store that in Python, create a new variable called `sentence`<jupyter_code>sentence =  'Heterosexuality is not normal, it is just common.'<jupyter_output><empty_output><jupyter_text>The text is surrounded by a single quote (`'`) on each side. To make sure that you typed the tweet correctly, you can type `sentence`:<jupyter_code>sentence<jupyter_output><empty_output><jupyter_text>You can get almost the same response using the `print` function:<jupyter_code>print(sentence)<jupyter_output>Heterosexuality is not normal, it is just common.<jupyter_text>The only difference is that the first response was wrapped in single quotes and the second wasn’t. As a side note, the single quotes weren’t because you put them there. If you used double quotes, Python would still show a single-quote.<jupyter_code>sentence =  "Heterosexuality is not normal, it is just common."

sentence<jupyter_output><empty_output><jupyter_text>In addition to `'` and `"`, strings can also be marked with a `'''`. This last one is particularly useful when your text contains contractions or quotation marks.<jupyter_code>new_sentence = '''According to Parker, "Heterosexuality is not normal, it is just common."'''
print(new_sentence)<jupyter_output>According to Parker, "'Heterosexuality is not normal, it is just common."<jupyter_text>String Exercise 1Create a new string called `food` that is a sentence about your most recent meal. Display the contents of your new string.<jupyter_code># Type your answer here
food = 'I had a pizza for lunch.'
print(food)<jupyter_output>I had a pizza for lunch.<jupyter_text>Python has a few tools for manipulating text, such as `lower` for making the string lower-case.<jupyter_code>sentence.lower()<jupyter_output><empty_output><jupyter_text>This did not alter the original string, however.<jupyter_code>sentence<jupyter_output><empty_output><jupyter_text>In Python, strings are immmutable, meaning once a string is created, it cannot be altered in place. To get around this, we can store the results from the `lower` command in a new variable.<jupyter_code>lower_sentence = sentence.lower()

lower_sentence<jupyter_output><empty_output><jupyter_text>String Exercise 2Create a new, lower cased version of your `food` string.<jupyter_code># Your answer here.

food_lower = food.lower()<jupyter_output><empty_output><jupyter_text>We can also `replace` text within the string.<jupyter_code>sentence.replace("Heterosexuality", "Monogamy")<jupyter_output><empty_output><jupyter_text>`replace` can also be used to remove text without replacement.<jupyter_code>sentence.replace(",", "")<jupyter_output><empty_output><jupyter_text>As before, this does not alter the original string. If you wanted to save the string edits, you would need to create a new variable.<jupyter_code>edited_sentence = sentence.lower()
print(edited_sentence)<jupyter_output>heterosexuality is not normal, it is just common.<jupyter_text>If you were doing a series of manipulations, you could reuse a varaiable name, although it is best practice to keep a version of the original string in case you ever need to go back to it.<jupyter_code>edited_sentence = sentence.lower()
print(edited_sentence)

edited_sentence = edited_sentence.replace(".", "")
print(edited_sentence)<jupyter_output>heterosexuality is not normal, it is just common.
heterosexuality is not normal, it is just common<jupyter_text>You can also stack multiple transformations together, although combining too many may make your code harder to follow.<jupyter_code>edited_sentence.replace(".", "").lower()<jupyter_output><empty_output><jupyter_text>String Exercise 3Create a new string called `boring` that removes the exclamation marks and capitalization from the sentence "Way to go!!!"<jupyter_code># Your answer here.
boring = "Way to go!!!".replace('!','').lower()
print(boring)

# Alternatively:
exciting = "Way to go!!!".replace('!','')
boring   = exciting.replace('!','')
boring   = boring.lower()
print(boring)<jupyter_output>way to go
way to go<jupyter_text>SlicingIf you had a very long text, such as the entire text of a Wikipedia article, you might only want to look at the first few characters. In Python, this is called by slicing.<jupyter_code>sentence
sentence[0:20]<jupyter_output><empty_output><jupyter_text>A slice is signaled with brackets (`[]`). The first number is the starting position, where 0 indicates the beginning. This is followed by a colon (`:`) and then the end position, which, in this case, is a 20. Note that this is splitting on characters, not words.Here is a section from the middle of the string:<jupyter_code>sentence[20:32]<jupyter_output><empty_output><jupyter_text>For convience, if you ommit the number before the colon, it defaults to the string beginning.<jupyter_code>sentence[:40]<jupyter_output><empty_output><jupyter_text>Ommitting the second number defaults to the end.<jupyter_code>sentence[40:]<jupyter_output><empty_output><jupyter_text>Finally, negative numbers are interpreted as distance from the end of the string.<jupyter_code>sentence[-20:]<jupyter_output><empty_output><jupyter_text>Slicing Exercise 1Create a new string called `s` that contains `The weather is hot and humid today.` Find the slices for each of the following :* `The we`* `today.`* `hot and humid`<jupyter_code>#Your answer here.
s = 'The weather is hot and humid today.'
print(s[:6])
print(s[-6:])
print(s[15:-7])<jupyter_output>The we
today.
hot and humid<jupyter_text>NumbersWe can also count the number of characters in a string with the `len` function.<jupyter_code>len(sentence)<jupyter_output><empty_output><jupyter_text>In this case, Python returned an interger instead of string. This also can be stored in a variable.<jupyter_code>sentence_length = len(sentence)
sentence_length<jupyter_output><empty_output><jupyter_text>Numbers Exercise 1What is the length of the sentence `How many dogs do you own?`? Store it in a variable called `sl`.<jupyter_code># Your answer here.
sl = len('How many dogs do you own?')
print(sl)<jupyter_output>25<jupyter_text>Since this is a number, we can do standard math operations with it.<jupyter_code>print(sentence_length * 3)
print(sentence_length / 2)
print(sentence_length + sentence_length)<jupyter_output>98<jupyter_text>Numbers Exercise 2What is one-third the length of `sl`?<jupyter_code># Your answer here
print(sl/3)<jupyter_output>8.333333333333334<jupyter_text>As with strings, these can be saved in new variables.<jupyter_code>double_length = sentence_length + sentence_length

print(double_length)<jupyter_output>98<jupyter_text>These same operators also work with strings.<jupyter_code>print('Frogs' * 3)
print('sociology' + 'political science')<jupyter_output>sociologypolitical science<jupyter_text>The operators can't be used to combine different data types, however.<jupyter_code>print("The sentence was " + sentence_length + "characters.")<jupyter_output><empty_output><jupyter_text>Conviently, Python the `str` function will convert an interger to a string.<jupyter_code>print("The sentence was " + str(sentence_length) + " characters.")<jupyter_output>The sentence was 8 characters.<jupyter_text>I manually had to include the spaces before and after `sentence_length`. Otherwise, it all is smushed together.<jupyter_code>print("The sentence was" + str(sentence_length) + "characters.")<jupyter_output>The sentence was8characters.<jupyter_text>Numbers Exercise 3Print `The length of the word "hippopotamus" is [x].` where `[x]` is the length of the word hippopotamus .<jupyter_code>#Your answer here.
l = len('hippopotamus')
s = 'The length of the word "hippopotamus" is ' + str(l) + '.'
print(s)<jupyter_output>The length of the word "hippopotamus" is 12.<jupyter_text>ListsWe can also `split` the sentence into a series of strings. By default, this splits based on spaces and other whitespace characters such as a line break (`\n`) or tab character (`\t`).<jupyter_code>print(sentence.split())<jupyter_output>['Heterosexuality', 'is', 'not', 'normal,', 'it', 'is', 'just', 'common.']<jupyter_text>What is returned here is a third data type (the first two were strings and intergers) called a list. A list is enclosed in brackets (`[]`) and the items are seperated by commas. In this case each item is in quotation marks because they are all strings. Items in a list, however, can be of any sort.<jupyter_code>my_list = ['Speeches', 7, 'Data']
my_list<jupyter_output><empty_output><jupyter_text>While `len` returned the number of characters in a string, it returns the number the items in a list.<jupyter_code>len(my_list)
sentence_length = len(sentence.split())
sentence_length<jupyter_output><empty_output><jupyter_text>In the second example, the list created by `sentence.split()` is not saved in any way; only its length.  Lists Exercise 1Create a list called `food` that includes at least three things you ate today. Use `len` to count the number of items in the list.<jupyter_code># Your answer here
food = ['pizza slice', 'naan', 'cupcake', 'grape']
print(len(food))<jupyter_output>4<jupyter_text>Like, strings, lists can also be sliced. The first three items of a list:<jupyter_code>words = sentence.split()
print(words[:3])<jupyter_output>['Heterosexuality', 'is', 'not']<jupyter_text>We can also extract specific items from a list by their position. As it did with strings, slicing in Python starts with 0.<jupyter_code>words[0]<jupyter_output><empty_output><jupyter_text>The third word:<jupyter_code>words[3]<jupyter_output><empty_output><jupyter_text>The fifth word from the end:<jupyter_code>words[-5]<jupyter_output><empty_output><jupyter_text>The last two words, returned as a list:<jupyter_code>words[-2:]<jupyter_output><empty_output><jupyter_text>Lists Exercise 2Display the first two items of your `food` list. What is the last item?<jupyter_code># Your answer here
#Sample Answer 

print(food[:2])
print(food[-1])<jupyter_output>['pizza slice', 'naan']
grape<jupyter_text>Unlike a string, lists are mutable. That means that we can remove or as is more frequently the case text analysis, add things to it. This is done with `append`.<jupyter_code>male_words = ['his', 'him', 'father']
male_words.append('brother')
print(male_words)<jupyter_output>['his', 'him', 'father', 'brother']<jupyter_text>Since `append` is changing `male_words`, we do not want to use an `=`. The Python interpreter is editing our original list but not returning anything.<jupyter_code>not_going_to_work = male_words.append('brother')
print(not_going_to_work)<jupyter_output>None<jupyter_text>Lists can be also be combined using `+`.<jupyter_code>gendered_words = male_words + ['her', 'she', 'mother']
print(gendered_words)<jupyter_output>['his', 'him', 'father', 'brother', 'brother', 'her', 'she', 'mother']<jupyter_text>As note above, the items in a list can include a variety of data types. This includes lists.<jupyter_code>gendered_lists = [ male_words ,  ['her', 'she', 'mother'] ]<jupyter_output><empty_output><jupyter_text>Note the two closing brackets next to each other. The first closes the list that ends with 'mother' while the second closes our `gendered_lists`.<jupyter_code>len(gendered_lists)<jupyter_output><empty_output><jupyter_text>`gendered_lists` has a length of two because it contains just two items, each a list of varying lengths.<jupyter_code>print(gendered_lists)<jupyter_output>[['his', 'him', 'father', 'brother', 'brother'], ['her', 'she', 'mother']]<jupyter_text>Lists Exercise 3Add three more items to your `food` list. Use`append` for the first item. For the other two, place them in a new list and then combine the two lists.<jupyter_code># Your answer here
# Sample answer

food.append('panini')
print(food)
food = food + ['burrito', 'box of donuts']
print(food)
print(len(food))<jupyter_output>['pizza slice', 'naan', 'cupcake', 'grape', 'panini']
['pizza slice', 'naan', 'cupcake', 'grape', 'panini', 'burrito', 'box of donuts']
7<jupyter_text>Dictionaries A fourth useful data type is a dictionary. A dictionary is like a list in that it holds multiples items. The items in a list can be identified by their position in the list. In contrast, the values in a dictionary are associated with a keyword. The analogy here is a to a physical dictionary, which has a list of unique words, and each word has a definition. In this case, the entries are called keys, and the definitions, which can be any data type, are called values. Alternatively, you can think of a dictionary as a single row of data from a dataset, where the keys are the variable names.<jupyter_code>respondent = {'sex'   : "female",
              'abany' : 1,
              'educ'  : 'College'}<jupyter_output><empty_output><jupyter_text>Dictionaries are surrounded by curly brackets (`{}`). Each entry is pair consisting of the key, which must be a string, followed, by a colon and then the value. Like in a list, entries are seperated by commas. We can access the contents of a dictionary by enclosing the key in brackets (`[]`).<jupyter_code>respondent['sex']<jupyter_output><empty_output><jupyter_text>If the key is not dictionary, you will get a `KeyError`.<jupyter_code>respondent['gender']<jupyter_output><empty_output><jupyter_text>You can inspect all the keys in a dictionary, in case you forgot or someone else made it.<jupyter_code>respondent.keys()
len(respondent.keys())<jupyter_output><empty_output><jupyter_text>Dictionaries are mutable, so we can change the value of existing keys, remove keys, or add new ones.<jupyter_code>respondent['race'] = 'Black'

print(respondent)
respondent['abany'] = 'Yes'

print(respondent)<jupyter_output>{'sex': 'female', 'abany': 'Yes', 'educ': 'College', 'race': 'Black'}<jupyter_text>Dictionaries Exercise 1Add a new key to the dictionary called `age` with a value of 37. Confirm that you did it correctly by dispaying the value of `age`.<jupyter_code># Your answer here.
respondent['age'] = 37
print(respondent['age'])<jupyter_output>37<jupyter_text>As noted above, while the keys have to be strings, the values can be any data type.<jupyter_code>respondent['children ages'] = [3, 5, 10]

print(respondent)<jupyter_output>{'sex': 'female', 'abany': 'Yes', 'educ': 'College', 'race': 'Black', 'age': 37, 'children ages': [3, 5, 10]}<jupyter_text>Spaces Within the Python community, there are strong norms about how code should be written. Many of these are centered around making the code readable, both by others and by your future self. As a trivial example, `2+2` is allowed, but is almost always written `2 + 2`. Likewise, I defined my respondent dictionary with plenty of white space to maximize readability.<jupyter_code>respondent = {'sex'   : "female",
              'abany' : 1,
              'educ'  : 'College'}<jupyter_output><empty_output><jupyter_text>This is identical to:<jupyter_code>respondent={'sex':'female','abany':1,'educ':'College'}<jupyter_output><empty_output><jupyter_text>but putting it all on one line obscures the logic of the dictionary. In this case, what is a key and what is a value is quite clear in the first version, while distinguishing between the two is more problematic in the single-line version.<jupyter_code>r2 = {'sex':'male',   'abany':1, 'educ':'College'     }
r3 = {'sex':'female', 'abany':0, 'educ':'High School' }
r4 = {'sex':'male',   'abany':0, 'educ':'Some College'}
respondents = [respondent, r2, r3, r4]
respondents<jupyter_output><empty_output><jupyter_text>This now looks a lot like the common data format JSON!  Loops<jupyter_code>for person in respondents:
    print(person['educ'])
for item in [1,2,'bobcat']:
    print(item)<jupyter_output>1
2
bobcat<jupyter_text>Loops Exercise 1Loop over the items in your `food` list. For each item, print its length.<jupyter_code>#Your answer here.
for item in food:
    item_len = len(item)
    print(item_len)<jupyter_output>11
4
7
5
6
7
13<jupyter_text>Functions For those who come from Stata or R background, one of the more striking aspects of Python code is the frequency of user-defined functions. These functions are deployed both for cases where you are surprised that no one has already written a function, such as counting words in a sentence, and for highly-custom situations, such as scraping the contents of a particular web page.   Programming with many small functions tends to make code more readable and easier to debug than code written in a more traditional social science style.A standard function has three parts. First, the function is named. Subsequent line(s) do the legwork of performing the desired task. Finally, the results are returned.  A trivial function that returns the `Hello!` might look like:<jupyter_code>def make_hello():
    word = 'Hello!'
    return word<jupyter_output><empty_output><jupyter_text>Of note, `def` signals that you are defining a function. This is followed by the name of the function. In this case, `make_hello`. Since this function doesn't take any arguments, such as accepting a variable to modify or have any options, it is followed by `()`. The first line ends with a colon.All subsequent lines are indented. The second line creates a new string variable called `word` which contains `Hello!`. The third and final line of the functions returns the value stored in word.<jupyter_code>make_hello()<jupyter_output><empty_output><jupyter_text>More commonly in text analysis, a user-defined function modifies an existing string. In this case, the variable name that will be used within the function is established within the parentheses on the opening line. A second trivial function takes a text string and returns an all-caps version.<jupyter_code>def scream(text):
    text_upper = text.upper()
    return text_upper
scream('Hi there!')<jupyter_output><empty_output><jupyter_text>The `text` and `text_upper` variable only exist within the function. That means that you can pass a variable not called `text` to the function.<jupyter_code>text_upper<jupyter_output><empty_output><jupyter_text>It is a good idea to include a comment within the function that explains the function. This is helpful for other people reading your code and when you return to your own code months and days later.<jupyter_code>def scream(text):
    '''Returns an all-caps version of text string.'''
    text_upper = text.upper()
    return text_upper<jupyter_output><empty_output><jupyter_text>Functions Exercise 1Make a function called "whisper" that replaces all exclamation marks with a period and returns a lower case version of a string. Test it out.<jupyter_code>#Your answer here.

def whisper(text):
    ''''''
    
    return quiet_text
def whisper(text):
    '''Lower case string and replace
       ! with .'''
    quiet_text = text.replace('!','.')
    quiet_text = quiet_text.lower()
    return quiet_text

#Testing it out: 
whisper('ASDFASDFAS!')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09822268687924726, 'fraction_numerical': 0.006272869837950863, 'mean_word_length': 6.200225818592397, 'pattern_counts': {'":': 0, '<': 245, '<?xml version=': 0, '>': 246, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '68', 'dir': 'jupyter-structured-clean-dedup', 'id': '250457', 'license': 'MIT', 'n_tokens_mistral': 6035, 'n_tokens_neox': 5719, 'n_words': 2182, 'path': 'Notebooks/python.ipynb', 'repo_name': 'nealcaren/Text-Mining-with-Python-Notebooks', 'size': '19132'}