<jupyter_start><jupyter_text>*If you click the Interactive button above, you can run the code on this page and complete each of the exercises. If you click the Notebook button, you will be redirected to an interactive Jupyter notebook. Both of these options are powered by [Binder](https://mybinder.org).**The answers to each of the exercises can be revealed by selecting the plus button on the right side of the page below each exercise.*  Python 101This section introduces some of the most relevant aspects of working with Python for social scientists. This includes the different data types available and ways to modify them.  In 2009, as part of his first State of the Union address, President <NAME> said: > Let us invest in our people without leaving them a mountain of debtTo store that in Python, create a new variable called `sentence`<jupyter_code>sentence =  'Let us invest in our people without leaving them a mountain of debt.'<jupyter_output><empty_output><jupyter_text>The text is surrounded by a single quote (`'`) on each side. To make sure that you typed the tweet correctly, you can type `sentence`:<jupyter_code>sentence<jupyter_output><empty_output><jupyter_text>You can get almost the same response using the `print` function:<jupyter_code>print(sentence)<jupyter_output>Let us invest in our people without leaving them a mountain of debt.<jupyter_text>The only difference is that the first response was wrapped in single quotes and the second wasn’t. As a side note, the single quotes weren’t because you put them there. If you used double quotes, Python would still show a single-quote.<jupyter_code>sentence =  "Let us invest in our people without leaving them a mountain of debt."

sentence<jupyter_output><empty_output><jupyter_text>In addition to `'` and `"`, strings can also be marked with a `'''`. This last one is particularly useful when your text contains contractions or quotation marks.<jupyter_code>new_sentence = '''Let's invest in our people without leaving them a mountain of debt.'''
print(new_sentence)<jupyter_output>Let's invest in our people without leaving them a mountain of debt.<jupyter_text>Your turnCreate a new string called food  that is a sentence about your most recent meal. Display the contents of your new string.<jupyter_code>food = 'My standard lunch is a veggie burrito.'
print(food)<jupyter_output>My standard lunch is a veggie burrito.<jupyter_text>StringsPython has a few tools for manipulating text, such as `lower` for making the string lower-case.<jupyter_code>sentence.lower()<jupyter_output><empty_output><jupyter_text>This did not alter the original string, however.<jupyter_code>sentence<jupyter_output><empty_output><jupyter_text>In Python, strings are immmutable, meaning once created, they can not be altered in place. We could store the results in a new variable.<jupyter_code>lower_sentence = sentence.lower()

lower_sentence<jupyter_output><empty_output><jupyter_text>Your turnCreate a new, lower cased version of your food string.<jupyter_code>lower_food = food.lower()

print(lower_food)<jupyter_output><empty_output><jupyter_text>We can also `replace` words within the string.<jupyter_code>sentence.replace("nation", "country")<jupyter_output><empty_output><jupyter_text>`replace` can also be used to remove text by not including anything with the replacement quotation marks.<jupyter_code>sentence.replace(".", "")<jupyter_output><empty_output><jupyter_text>As before, this does not alter the original string. If you wanted to save the string edits, you would need to create a new variable.<jupyter_code>edited_sentence = sentence.lower()
print(edited_sentence)<jupyter_output>let us invest in our people without leaving them a mountain of debt.<jupyter_text>If you were doing a series of manipulations, you could reuse a varaiable name, although it is best practices to keep a version of the original string in case you ever need to go back to it.<jupyter_code>edited_sentence = sentence.lower()
print(edited_sentence)

edited_sentence = edited_sentence.replace(".", "")
print(edited_sentence)<jupyter_output>let us invest in our people without leaving them a mountain of debt.
let us invest in our people without leaving them a mountain of debt<jupyter_text>You can also stack multiple transformations together, although combining too many may make your code harder to follow.<jupyter_code>edited_sentence.replace(".", "").lower()<jupyter_output><empty_output><jupyter_text>Your turnCreate a new string called boring that removes the exclamation marks and capitalization from the sentence "Way to go!!!".<jupyter_code>boring = "Way to go!!!".lower().replace('!', '')

print(boring)<jupyter_output><empty_output><jupyter_text>SlicingIf you had a very long text, such as the entire text of the State of the Union, you might only want to look at the first few characters. In Python, this is called by slicing.<jupyter_code>sentence
sentence[0:20]<jupyter_output><empty_output><jupyter_text>A slice is signaled with brackets (`[]`). The first number is the starting position, where 0 indicates the beginning. This is followed by a colon (`:`) and then the end position, which, in this case, is a 20. Note that this is splitting on characters, not words.Here is a section from the middle of the string:<jupyter_code>sentence[20:32]<jupyter_output><empty_output><jupyter_text>For convience, if you ommit the number before the colon, it defaults to the string beginning.<jupyter_code>sentence[:40]<jupyter_output><empty_output><jupyter_text>Ommitting the second number defaults to the end.<jupyter_code>sentence[40:]<jupyter_output><empty_output><jupyter_text>Finally, negative numbers are interpreted as distance from the end of the string.<jupyter_code>sentence[-20:]<jupyter_output><empty_output><jupyter_text>Your turnCreate a new string called `s` that contains `The weather is hot and humid today.` Find the slices for each of the following :* "The w"* "today."* "hot and humid"<jupyter_code>s = 'The weather is hot and humid today.'

print(s[:5])
print(s[-6:])
print(s[15:28])<jupyter_output>The w
today.
hot and humid<jupyter_text>NumbersWe can also count the number of characters in a string with the `len` function.<jupyter_code>len(sentence)<jupyter_output><empty_output><jupyter_text>In this case, Python returned an interger instead of string. This also can be stored in a variable.<jupyter_code>sentence_length = len(sentence)
sentence_length<jupyter_output><empty_output><jupyter_text>Your turnWhat is the length of How many dogs do you own?? Store it in a variable called sl.<jupyter_code>question = 'How many dogs do you own?'
sl = len(question)
print(sl)<jupyter_output><empty_output><jupyter_text>Since the length of a string  is a number, we can do standard math operations with it.<jupyter_code>print(sentence_length * 3)
print(sentence_length / 2)
print(sentence_length + sentence_length)<jupyter_output>136<jupyter_text>Your turnWhat is one-third the length of sl?<jupyter_code>sl/3<jupyter_output><empty_output><jupyter_text>As with strings, these can be saved in new variables.<jupyter_code>double_length = sentence_length + sentence_length

print(double_length)<jupyter_output>136<jupyter_text>These same operators also work with strings.<jupyter_code>print(sentence * 2)
print(sentence + sentence)<jupyter_output>Let us invest in our people without leaving them a mountain of debt.Let us invest in our people without leaving them a mountain of debt.<jupyter_text>The operators can't be used to combine different data types, however.<jupyter_code>print("The sentence was " + sentence_length + "characters.")<jupyter_output><empty_output><jupyter_text>Conviently, the `str` function will convert an interger to a string.<jupyter_code>print("The sentence was " + str(sentence_length) + " characters.")<jupyter_output>The sentence was 68 characters.<jupyter_text>I manually had to include the spaces before and after `sentence_length`. Otherwise, it all is smushed together.<jupyter_code>print("The sentence was" + str(sentence_length) + "characters.")<jupyter_output>The sentence was68characters.<jupyter_text>Your turnPrint `The length of the word "hippopotamus" is [x].` where `[x]` is the length of the word hippopotamus  .<jupyter_code>l = len('hippopotamus')

print('The length of the word "hippopotamus" is ' + str(l) + '.')<jupyter_output><empty_output><jupyter_text>ListsYou can also `split` the sentence into a series of strings. By default, this splits based on spaces and other whitespace characters such as a line break (`\n`) or tab character (`\t`).<jupyter_code>print(sentence.split())<jupyter_output>['Let', 'us', 'invest', 'in', 'our', 'people', 'without', 'leaving', 'them', 'a', 'mountain', 'of', 'debt.']<jupyter_text>What is returned here is a third data type (the first two were strings and intergers) called a list. A list is enclosed in brackets (`[]`) and the items are seperated by commas. In this case each item is in quotation marks because they are all strings. Items in a list, however, can be of any sort.<jupyter_code>my_list = ['Speeches', 7, 'Data']
my_list<jupyter_output><empty_output><jupyter_text>While `len` returned the number of characters in a string, it returns the number the items in a list.<jupyter_code>len(my_list)
sentence_length = len(sentence.split())
sentence_length<jupyter_output><empty_output><jupyter_text>In the second example, the list created by `sentence.split()` is not saved in any way; only its length.  Your turnCreate a list called **ate** that includes at least three things you ate today. Use `len` to count the number of items in the list.<jupyter_code>ate = ['apple', 'dosa', 'pizza slice']

print(len(ate))<jupyter_output><empty_output><jupyter_text>Like, strings, lists can also be sliced. The first three items of a list:<jupyter_code>words = sentence.split()
print(words[:3])<jupyter_output>['Let', 'us', 'invest']<jupyter_text>We can also extract specific items from a list by their position. As it did with strings, slicing in Python starts with 0.<jupyter_code>words[0]<jupyter_output><empty_output><jupyter_text>The third word:<jupyter_code>words[2]<jupyter_output><empty_output><jupyter_text>The fifth word from the end:<jupyter_code>words[-5]<jupyter_output><empty_output><jupyter_text>The last two words:<jupyter_code>words[-2:]<jupyter_output><empty_output><jupyter_text>Your turnDisplay the first two items of your **ate** list. What is the last item?<jupyter_code>print(ate[:2])
print(ate[-1])<jupyter_output>['apple', 'dosa']
pizza slice<jupyter_text>Slicing a list returns a list. If you ask for the first three items, you will get a list made up of those items. In contrast, if your request a specific location, such as `words[2]`, Python returns the specific object stored in the place, which may be a string, number, or event an entire list.  Unlike a string, lists are mutable. That means that we can remove or as is more frequently the case text analysis, add things to it. This is done with `append`.<jupyter_code>male_words = ['his', 'him', 'father']
male_words.append('brother')
print(male_words)<jupyter_output>['his', 'him', 'father', 'brother']<jupyter_text>Since `append` is changing `male_words`, we do not want to use an `=`. The Python interpreter is editing our original list but not returning anything.<jupyter_code>not_going_to_work = male_words.append('brother')
print(not_going_to_work)<jupyter_output>None<jupyter_text>Lists can be also be combined using `+`.<jupyter_code>gendered_words = male_words + ['her', 'she', 'mother']
print(gendered_words)<jupyter_output>['his', 'him', 'father', 'brother', 'brother', 'her', 'she', 'mother']<jupyter_text>As note above, the items in a list can include a variety of data types. This includes lists.<jupyter_code>gendered_lists = [ male_words ,  ['her', 'she', 'mother'] ]<jupyter_output><empty_output><jupyter_text>Note the two closing brackets next to each other. The first closes the list that ends with 'mother' while the second closes our `gendered_lists`.<jupyter_code>len(gendered_lists)<jupyter_output><empty_output><jupyter_text>`gendered_lists` has a length of two because it contains just two items, each a list of varying lengths.<jupyter_code>print(gendered_lists)<jupyter_output>[['his', 'him', 'father', 'brother', 'brother'], ['her', 'she', 'mother']]<jupyter_text>Your turnAdd three more items to your `food` list. Use `append` for one. For the other two, places them in a list and then combine the two lists.<jupyter_code># solution<jupyter_output><empty_output><jupyter_text>Dictionaries A fourth useful data type is a dictionary. A dictionary is like a list in that it holds multiples items. The items in a list can be identified by their position in the list. In contrast, the values in a dictionary are associated with a keyword. The analogy here is a to a physical dictionary, which has a list of unique words, and each word has a definition. In this case, the entries are called keys, and the definitions, which can be any data type, are called values. Alternatively, you can think of a dictionary as a single row of data from a dataset, where the keys are the variable names.<jupyter_code>respondent = {'sex'   : "female",
              'abany' : 1,
              'educ'  : 'College'}<jupyter_output><empty_output><jupyter_text>Dictionaries are surrounded by curly brackets (`{}`). Each entry is pair consisting of the key, which must be a string, followed, by a colon and then the value. Like in a list, entries are seperated by commas. You can access the contents of a dictionary by enclosing the key in brackets (`[]`).<jupyter_code>respondent['sex']<jupyter_output><empty_output><jupyter_text>If the key is not dictionary, you will get a `KeyError`.<jupyter_code>respondent['gender']<jupyter_output><empty_output><jupyter_text>You can inspect all the keys in a dictionary, in case you forgot or someone else made it.<jupyter_code>respondent.keys()
len(respondent.keys())<jupyter_output><empty_output><jupyter_text>Dictionaries are mutable, so we can change the value of existing keys, remove keys, or add new ones.<jupyter_code>respondent['race'] = 'Black'

print(respondent)
respondent['abany'] = 'Yes'

print(respondent)<jupyter_output>{'sex': 'female', 'abany': 'Yes', 'educ': 'College', 'race': 'Black'}<jupyter_text>Your turnAdd a new key to the dictionary called `age` with a value of 37. Confirm that you did it correctly by displaying the value of `age`.<jupyter_code>respondent['age'] = 37

print(respondent['age'])<jupyter_output><empty_output><jupyter_text>As noted above, while the keys have to be strings, the values can be any data type. You could add the ages of the respondent's children as a list.<jupyter_code>respondent['children ages'] = [3, 5, 10]

print(respondent)<jupyter_output>{'sex': 'female', 'abany': 'Yes', 'educ': 'College', 'race': 'Black', 'age': 37, 'children ages': [3, 5, 10]}<jupyter_text>Spaces Within the Python community, there are strong norms about how code should be written. Many of these are centered around have code be readable, both by others and by your future self. As a trivial example, `2+2` is allowed, by is almost always written `2 + 2`. Likewise I defined my respondent dictionary with plenty of white space in order to maximize readability.<jupyter_code>respondent = {'sex'   : "female",
              'abany' : 1,
              'educ'  : 'College'}<jupyter_output><empty_output><jupyter_text>This is identical to:<jupyter_code>respondent={'sex':'female','abany':1,'educ':'College'}<jupyter_output><empty_output><jupyter_text>but putting it all on one line obscures the logic of the dictionary. In this case, what is a key and what is a value is quite clear in the first version, while distinguishing between the two is more problematic in the single-line version.<jupyter_code>r2 = {'sex':'male',   'abany':1, 'educ':'College'     }
r3 = {'sex':'female', 'abany':0, 'educ':'High School' }
r4 = {'sex':'male',   'abany':0, 'educ':'Some College'}
respondents = [respondent, r2, r3, r4]
respondents<jupyter_output><empty_output><jupyter_text>This is now looks a lot like the common data format JSON!  Loops<jupyter_code>for person in respondents:
    print(person['educ'])
for item in [1,2,'bobcat']:
    print(item)<jupyter_output>1
2
bobcat<jupyter_text>Your turnLoop over the items in your `food` list. For each item, print its length.<jupyter_code># Answer<jupyter_output><empty_output><jupyter_text>Functions For those who come from Stata or R background, one of the more striking aspects of Python code is the frequency of user defined functions. They are deployed not just for things where you think there should be a function, like counting words in a sentence, but also for highly-custom situations, such as scraping the contents of a particular web page. This style of programming, with many small functions, tends to make code more readable easier to debug than code written in a more traditional social science style.A standard function has three parts. First the function is named and defined. Subsequent line or lines actually do the thing. Finally the results are returned.  A trivial function that returns the `Hello!` might look like:<jupyter_code>def make_hello():
    word = 'Hello!'
    return word<jupyter_output><empty_output><jupyter_text>Of note, `def` signals that your are defining a function. This is followed by the name of the function. In this case, `make_hello`. Since this function doesn't take any arguments, such as accepting a variable to modify or have any options, it is followed by `()`. The first line ends with a colon.All subsequent lines are indented. The second line creates a new string variable called `word` which contains `Hello!`. The third and final line of the functions returns the value stored in word.<jupyter_code>make_hello()<jupyter_output><empty_output><jupyter_text>More commonly in text analysis, a user-defined function modifies an existing string. In this case, the variable name that will be used within the function is established within the parenthesis on the opening line. A second trivial function takes a text string and returns an all-caps version.<jupyter_code>def scream(text):
    text_upper = text.upper()
    return text_upper
scream('Hi there!')<jupyter_output><empty_output><jupyter_text>The `text` and `text_upper` variable only exist within the function. That means that you can pass a variable not called `text` to the function.<jupyter_code>scream(sentence)<jupyter_output><empty_output><jupyter_text>It also means that everything but the returned `text` disappears.<jupyter_code>text_upper<jupyter_output><empty_output><jupyter_text>It is a good idea to include a comment within the function that explains the function. This is helpful for other people reading your code and when you return to your own code months and days later.<jupyter_code>def scream(text):
    '''Returns an all-caps version of text string.'''
    text_upper = text.upper()
    return text_upper<jupyter_output><empty_output><jupyter_text>Your turnMake a function called `whisper` that replaces all exclamation marks with a period and returns a lower case version of a string. Test it out.<jupyter_code>def whisper(text):
    ''''''
    
    return quite_text<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09147512063508535, 'fraction_numerical': 0.004514087064805687, 'mean_word_length': 6.1944755505785745, 'pattern_counts': {'":': 0, '<': 257, '<?xml version=': 0, '>': 258, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '69', 'dir': 'jupyter-structured-clean-dedup', 'id': '298086', 'license': 'MIT', 'n_tokens_mistral': 5872, 'n_tokens_neox': 5608, 'n_words': 2255, 'path': 'content/getting_started/python.ipynb', 'repo_name': 'nealcaren/textminingwithpython', 'size': '19279'}