<jupyter_start><jupyter_text>Chapter 3 - Strings*This notebook uses code snippets and explanations from [this course](https://github.com/kadarakos/python-course/blob/master/Chapter%2011%20-%20Strings.ipynb).*The first thing you learned was printing a simple sentence: "Hello, world!" This sentence, as any other **text**, was stored by Python as a **string**. Since many disciplines within the Humanities and Social Sciences work with texts, quite naturally we will focus a lot on **manipulating and analyzing texts** in this course. In addition, generally use strings to point to files on out computer (so-called filepaths) or use urls to point to websites (more on both later). Therefore, being able to manipulate strings is a central component of this course (which will come back in the following blocks). This Notebook is devoted to this object type. At the end of this chapter, you will be able to:* define *strings* and understand their *internal representation** understand strings as *sequences** use character indices for *string slicing** combine strings through *printing, concatenation and insertion** compare strings using *comparison operators* and the *in operator** understand strings as *immutable* objects* work with and understand *string methods** understand the difference between *args* and *kwargs* If you want to learn more about these topics, you might find the following links useful:* Documentation: [String methods](https://docs.python.org/3/library/stdtypes.htmlstring-methods)* Documentation: [Literal String Interpolation (f-strings)](https://www.python.org/dev/peps/pep-0498/)* Explanation: [Strings](https://www.tutorialspoint.com/python/python_strings.htm)* Explanation: [F-strings](https://cito.github.io/blog/f-strings/)* Video: [Strings - working with text data](https://www.youtube.com/watch?v=k9TUPpGqYTo)* Video: [Strings](https://www.youtube.com/watch?v=L2IUSArpG98)* Video: [String Indexing and Slicing](https://www.youtube.com/watch?v=lMQRjpgJslI)If you have **questions** about this chapter, please contact **Pia (<EMAIL>)**.  1. Defining and representing stringsA **string** is a sequence of letters/characters which together form a whole (for instance a word, sentence or entire text). In Python, a string is a type of object for which the value is enclosed by single or double quotes. Let's define a few of them:<jupyter_code># Here are some strings:
string_1 = "Hello, world!"
string_2 = 'I ❤️ cheese'      # If you are using Python 2, your computer will not like this.
string_3 = '1,2,3,4,5,6,7,8,9'<jupyter_output><empty_output><jupyter_text>There is no difference in declaring a string with single or double quotes. However, if your string contains a quote symbol it can lead to errors if you try to enclose it with the same quotes.<jupyter_code># Run this cell to see the error generated by the following line.
restaurant = 'Wendy's'<jupyter_output><empty_output><jupyter_text>In the example above the error indicates that there is something wrong with the letter *s*. This is because the single quote closes the string we started, and anything after that is unexpected.To solve this we can enclose the string in double quotes, as follows:<jupyter_code>restaurant = "Wendy's"
# Similarly, we can enclose a string containing double quotes with single quotes:
quotes = 'Using "double" quotes enclosed by a single quote.'<jupyter_output><empty_output><jupyter_text>We can also use the escape character "\" in front of the quote, which will tell Python not to treat this specific quote as the end of the string.<jupyter_code>restaurant = 'Wendy\'s'
print(restaurant)
restaurant = "Wendy\"s"
print(restaurant)<jupyter_output><empty_output><jupyter_text>1.1 Multi-line stringsStrings in Python can also span across **multiple lines**, which can be useful for when you have a very long string, or  when you want to format the output of the string in a certain way. This can be achieved in two ways:1. With single or double quotes, where we manually indicate that the rest of the string continues on the next line with a backslash.2. With three single or double quotes.We will first demonstrate how this would work when you use **one double or single quote**.<jupyter_code># This example also works with single-quotes.
long_string = "A very long string\n\
can be split into multiple\n\
sentences by appending a newline symbol\n\
to the end of the line."

print(long_string)<jupyter_output>A very long string
can be split into multiple
sentences by appending a newline symbol
to the end of the line.<jupyter_text>The **`\n` or newline symbol** indicates that we want to start the rest of the text on a new line in the string, the following **\\** indicates that we want the string to continue on the next line of the code. This difference can be quite hard to understand, but best illustrated with an example where we do not include the `\n` symbol.<jupyter_code>long_string = "A very long string \
can be split into multiple \
sentences by appending a backslash \
to the end of the line."

print(long_string)<jupyter_output>A very long string can be split into multiple sentences by appending a backslash to the end of the line.<jupyter_text>As you can see, Python now interprets this example as a single line of text. If we use the recommended way in Python to write multiline strings, **with triple double or single quotes**, you will see that the `\n` or newline symbol is automatically included.<jupyter_code>long_string = """A very long string
can also be split into multiple 
sentences by enclosing the string
with three double or single quotes."""

print(long_string)

print()

another_long_string = '''A very long string
can also be split into multiple 
sentences by enclosing the string
with three double or single quotes.'''

print(another_long_string)<jupyter_output>A very long string
can also be split into multiple 
sentences by enclosing the string
with three double or single quotes.

A very long string
can also be split into multiple 
sentences by enclosing the string
with three double or single quotes.<jupyter_text>What will happen if you remove the backslash characters in the example? Try it out in the cell below.<jupyter_code>long_string = "A very long string\
can be split into multiple\
sentences by appending a backslash\
to the end of the line."

print(long_string)<jupyter_output>A very long stringcan be split into multiplesentences by appending a backslashto the end of the line.<jupyter_text>1.2 Internal representation: using `repr()` As we have seen above, it is possible to make strings that span multiple lines. Here are two ways to do so:<jupyter_code>multiline_text_1 = """This is a multiline text, so it is enclosed by triple quotes.
Pretty cool stuff!
I always wanted to type more than one line, so today is my lucky day!"""
multiline_text_2 = "This is a multiline text, so it is enclosed by triple quotes.\nPretty cool stuff!\nI always wanted to type more than one line, so today is my lucky day!"
print(multiline_text_1)
print() # this just prints an empty line
print(multiline_text_2)<jupyter_output>This is a multiline text, so it is enclosed by triple quotes.
Pretty cool stuff!
I always wanted to type more than one line, so today is my lucky day!

This is a multiline text, so it is enclosed by triple quotes.
Pretty cool stuff!
I always wanted to type more than one line, so today is my lucky day!<jupyter_text>Internally, these strings are equally represented. We can check that with the double equals sign, which checks if two objects are the same:<jupyter_code>print(multiline_text_1 == multiline_text_2)<jupyter_output>True<jupyter_text>So from this we can conclude that `multiline_text_1` has the same **hidden characters** (in this case **`\n`**, which stands for 'new line') as `multiline_text_2`. You can show that this is indeed true by using the built-in **`repr()`** function (which gives you the Python-internal *repr*esentation of an object).<jupyter_code># Show the internal representation of multiline_text_1.
print(repr(multiline_text_1))
print(repr(multiline_text_2))<jupyter_output>'This is a multiline text, so it is enclosed by triple quotes.\nPretty cool stuff!\nI always wanted to type more than one line, so today is my lucky day!'
'This is a multiline text, so it is enclosed by triple quotes.\nPretty cool stuff!\nI always wanted to type more than one line, so today is my lucky day!'<jupyter_text>Another hidden character that is often used is **`\t`**, which represents tabs:<jupyter_code>colors = "yellow\tgreen\tblue\tred"
print(colors)
print(repr(colors))<jupyter_output>yellow	green	blue	red
'yellow\tgreen\tblue\tred'<jupyter_text>2. Strings as sequences 2.1 String indicesStrings are simply **sequences of characters**. Each character in a string therefore has a position, which can be referred to by the **index number** of the position. The index numbers start at 0 and then increase to the length of the string. You can also start counting backwards using negative indices. The following table shows all characters of the sentence "Sandwiches are yummy" in the first row. The second row and the third row show respectively the **positive and negative indices** for each character:| Characters     | S | a | n | d | w | i | c | h | e | s |    | a  |  r | e  |    |  y |  u |  m |  m |  y ||----------------|---|---|---|---|---|---|---|---|---|---|----|----|----|----|----|----|----|----|----|----|| Positive index | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 || Negative index |-20|-19|-18|-17|-16|-15|-14|-13|-12|-11|-10|-9|-8|-7|-6|-5|-4|-3|-2|-1|You can access the characters of a string as follows:<jupyter_code>my_string = "<NAME> yummy"
print(my_string[1])   
print(my_string[-1])<jupyter_output><empty_output><jupyter_text>**Length:** Python has a built-in function called **`len()`** that lets you compute the length of a sequence. It works like this:<jupyter_code>number_of_characters = len(my_string)
print(number_of_characters) # Note that spaces count as characters too!<jupyter_output><empty_output><jupyter_text>2.2 Slicing and indices applied to stringsBesides using single indices we can also extract a range from a string:<jupyter_code>my_string = "Sandwiches are yummy"
print(my_string[1:4])<jupyter_output>and<jupyter_text>This is called **string slicing**. So how does this notation work?```pythonmy_string[i]                   Get the character at index i.my_string[start:end]           Get the substring starting at 'start' and ending *before* 'end'.my_string[start:end:stepsize]  Get all characters starting from 'start', ending before 'end',                                with a specific step size.```You can also leave parts out:```pythonmy_string[:i]                  Get the substring starting at index 0 and ending just before i.my_string[i:]                  Get the substring starting at i and running all the way to the end.my_string[::i]                 Get a string going from start to end with step size i.```You can also have negative step size. `my_string[::-1]` is the idiomatic way to reverse a string.![Blocks](images/string-slicing.png)**Tip:** Slicing and accessing values via indices is very useful and can be applied to other python objects, which have a fixed sequence, such as lists (we will see how in the subsequent notebooks). Try to understand what is going on with string slicing - it will be very helpful in rest of the course! Do you know what the following statements will print?<jupyter_code>print(my_string[1:4]) 
print(my_string[1:4:1]) 
print(my_string[11:14])
print(my_string[15:])
print(my_string[:9])
print('cow'[::2])
print('cow'[::-2])
# a fun trick to reverse sequences:
print(my_string[::-1])
# You can do something similar with lists (you don't have to understand this is detail now - but we'll show you an
# example already, so you've seen it):

my_list = ['a', 'bunch', 'of', 'words']
print(my_list[3])
print(my_list[2:4])
print(my_list[-1])<jupyter_output>words
['of', 'words']
words<jupyter_text>3. ImmutabilityThe **mutability** of an object refers to whether an object can change or not. Strings are **immutable**, meaning that they *cannot* be changed. It is possible to create a new string-object based on the old one, but we cannot modify the existing string-object. The cells below demonstrate this.<jupyter_code># This is fine, because we are creating a new string. The old one remains unchanged:
fruit = 'guanabana'
island = fruit[:5] 
print(island, 'island')
print(fruit, 'fruit')
# This works because we are creating a new string and overwriting our old one
fruit = fruit[5:] + 'na' 
print(fruit)
# This does not work because now we are trying to change an existing string
fruit[4:5] = 'an' 
print(fruit)
# We could do this with a list though (don't worry about this yet - it is just meant to show the contrast)

fruits = ['cherry', 'blueberry', 'banana']
fruits[2:3] = ['rasperry', 'kiwi']
fruits
# If we want to modify a string by exchanging characters, we need to do:
fruit = fruit[:4] + 'an'
print(fruit)<jupyter_output>guanan<jupyter_text>The reasons for why strings are immutable are beyond the scope of this notebook. Just remember that if you want to modify a string, you need to overwrite the entire string, and you cannot modify parts of it by using individual indices.  4. Comparing stringsIn Python it is possible to use **comparison operators** (as used in conditional statements) on strings. These operators are: * == ('is the same as') * != ('is not the same as')* < ('is smaller than')* <= ('is the same as or smaller than')* > ('is greater than')* \>= ('is the same as or greater than')**Attention** '=' is used to assign a value to a variable whereas '==' is used to compare two values. If you get errors in comparisons, check if you used the correct operator. Some of these symbols are probably familiar to you from your math classes. Most likely, you have used them before to compare numbers. However, we can also use them to compare strings!  There are a number of things we have to know about python when comparing strings:* String comparison is always case-sensitive * Internally, chanracters are represented as numerical values, which can be ranked. You can use the smaller than/greater than operators to put words in lexicographical order. This is similar to the alphabetical order you would use with a dictionary, except that all the uppercase letters come before all the lowercase letters (so first *A, B, C, etc.* and then *a, b, c, etc.*)<jupyter_code>print('a' == 'a')
print('a' != 'b')
print('a' == 'A')  # string comparison is case-sensitive
print('a' < 'b')   # alphabetical order
print('A' < 'a')   # uppercase comes before lowercase
print('B' < 'a')   # uppercase comes before lowercase
print()
print('orange' == 'Orange')
print('orange' > 'Orange')
print('orange' < 'Orange')
print('orange' > 'banana')
print('Orange' > 'banana')<jupyter_output><empty_output><jupyter_text>Another way of comparing strings is to check whether a string is part of another string, which can be done using **the `in` operator**. It returns `True` if the string contains the relevant substring, and `False` if it doesn't. These two values (`True` and `False`) are called **boolean values**, or *booleans* for short. We'll talk about them in more detail later. Here are some examples to try:<jupyter_code>"fun" in "function"
"I" in "Team"
"App" in "apple" # Capitals are not the same as lowercase characters!
"apple" in "apple" 
"applepie" in "apple"<jupyter_output><empty_output><jupyter_text>5. Printing, concatenating and inserting stringsYou will often find yourself concatenating and printing combinations of strings. Consider the following examples:<jupyter_code>print("Hello", "World")
print("Hello " + "World")<jupyter_output><empty_output><jupyter_text>Even though they may look similar, there are two different things happening here. Simply said: the plus in the expression is doing concatenation, but the comma is not doing concatenation.**The 'print()' function**, which we have seen many times now, will print as strings everything in a comma-separated sequence of expressions to your screen, and it will separate the results with single blanks by default. Note that you can mix types: anything that is not already a string is automatically converted to its string representation.<jupyter_code>number = 5
print("I have", number, "apples")<jupyter_output>I have 5 apples<jupyter_text>**String concatenation**, on the other hand, happens when we merge two strings into a single object using the + operator. No single blanks are inserted, and you cannot concatenate mix types. So, if you want to merge a string and an integer, you will need to convert the integer to a string.<jupyter_code>number = 5
print("I have " + str(number) + " apples")<jupyter_output><empty_output><jupyter_text>Optionally, we can assign the concatenated string to a variable:<jupyter_code>my_string = "I have " + str(number) + " apples"
print(my_string)<jupyter_output><empty_output><jupyter_text>In addition to using + to concatenate strings, we can also use the multiplication sign \* in combination with an integer for repeating strings (note that we again need to add a blank after 'apples' if we want it to be inserted):<jupyter_code>my_string = "apples " * 5
print(my_string)<jupyter_output><empty_output><jupyter_text>The difference between "," and "+" when printing and concatenating strings can be confusing at first. Have a look at these examples to get a better sense of their differences.<jupyter_code>print("Hello", "World")
print("Hello" + "World")
print("Hello " + "World")
print(5, "eggs")
print(str(5), "eggs")
print(5 + " eggs")
print(str(5) + " eggs")
text = "Hello" + "World"
print(text)
print(type(text))
text = "Hello", "World"
print(text)
print(type(text))<jupyter_output><empty_output><jupyter_text>5.1 Using f-strings We can imagine that string concatenation can get rather confusing and unreadable if we have more variables. Consider the following example:<jupyter_code>name = "Pia"
age = 26
country = "Austria"
residence = "The Netherlands"

introduction = "Hello. My name is " + name + ". I'm " +  str(age) + " years old and I'm from " + country + \
", but I live in "+ residence +'.'
print(introduction)<jupyter_output>Hello. My name is Pia. I'm 26 years old and I'm from Austria, but I live in the Netherlands.<jupyter_text>Luckily, there is a way to make the code a lot more easy to understand and nicely formatted. In Python, you can use astring formatting mechanism called [Literal String Interpolation](https://www.python.org/dev/peps/pep-0498/). Strings that are formatted using this mechanism are called **f-strings**, after the leading character used to denote such strings, and standing for "formatted strings". It works as follows:<jupyter_code>name="Pia" 
age=26
country="Austria"
residence = "The Netherlands"
introduction = f"Hello. My name is {name}. I'm {age} years old and I'm from {country}, but I live in {residence}."
introduction<jupyter_output><empty_output><jupyter_text>We can even do cool stuff like this with f-strings:<jupyter_code>text = f"Soon, I'm turning {age+1} years."
print(text)<jupyter_output>Soon, I'm turning 27 years.<jupyter_text>Other formatting methods that you may come across include [%-formatting](https://docs.python.org/3/library/stdtypes.htmlprintf-style-string-formatting) and [str.format()](https://docs.python.org/3/library/string.htmlformatstrings), but we recommend that you use f-strings because they are the most intuitive.Using f-strings can be extremely useful if you're dealing with a lot of data you want to modify in a similar way. Suppose you want to create many new files containing data and name them according to a specific system. You can create a kind of template name and then fill in specific information using variables. (More about files later.)  6. String methodsA **method** is a function that is associated with an object. For example, the **string-method `lower()`** turns a string into all lowercase characters, and **the string method `upper()`** makes strings uppercase. You can call this method using the **dot-notation** as shown below:<jupyter_code>string_1 = 'Hello, world!'
print(string_1)         # The original string.
print(string_1.lower()) # Lowercased.
print(string_1.upper())# Uppercased.<jupyter_output>Hello, world!
hello, world!
HELLO, WORLD!
!dlrow ,olleH<jupyter_text>6.1 Learning about methodsSo how do you find out what kind of methods an object has? There are two options:1. Read the documentation. See [here](https://docs.python.org/3.5/library/stdtypes.htmlstring-methods) for the string methods.2. Use the **`dir()` function**, which returns a list of method names (as well as attributes of the object). If you want to know what a specific method does, use the **`help()` function**.Run the code below to see what the output of `dir()` looks like. The method names that start and end with double underscores ('dunder methods') are Python-internal. They are what makes general methods like `len()` work (`len()` internally calls the `string.__len__()` function), and cause Python to know what to do when you, for example, use a for-loop with a string.The other method names indicate common and useful methods.<jupyter_code># Run this cell to see all methods for strings
dir(str)<jupyter_output><empty_output><jupyter_text>If you'd like to know what one of these methods does, you can just use `help()` (or look it up online):<jupyter_code>help(str.upper)<jupyter_output><empty_output><jupyter_text>It's important to note that string methods only *return* the result. They do not change the string itself.<jupyter_code>x = 'test'    # Defining x.
y = x.upper() # Using x.upper(), assigning the result to variable y.
print(y)      # Print y.
print(x)      # Print x. It is unchanged.<jupyter_output><empty_output><jupyter_text>Below we illustrate some of the string methods. Try to understand what is happening. Use the `help()` function to find more information about each of these methods.<jupyter_code># Find out more about each of the methods used below by changing the name of the method
help(str.strip)
s = ' Humpty Dumpty sat on the wall '
print(s)
s = s.strip() 
print(s)

print(s.upper())
print(s.lower())

print(s.count("u"))
print(s.count("U"))

print(s.find('sat'))
print(s.find('t', 12))
print(s.find('q', 12))

print(s.replace('sat on', 'fell off'))

words = s.split()    # This returns a list, which we will talk about later.
for word in words:   # But you can iterate over each word in this manner
    print(word.capitalize())

print('-'.join(words))<jupyter_output><empty_output><jupyter_text>Exercises  Exercise 1: Can you identify and explain the errors in the following lines of code? Correct them please!<jupyter_code>print("A message").
print("A message')
print('A message"')<jupyter_output><empty_output><jupyter_text>Exercise 2: Can you print the following? Try using both positive and negative indices.- the letter 'd' in my_string - the letter 'c' in my_string<jupyter_code>my_string = "Sandwiches are yummy"
# your code here<jupyter_output><empty_output><jupyter_text>Can you print the following? Try using both positive and negative indices.- make a new string containing your first name and print its first letter- print the number of letters in your name<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Exercise 3: Can you print all a's in the word 'banana'?<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Can you print 'banana' in reverse ('ananab')?<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>Can you exchange the first and last characters in `my_string` ('aananb')? Create a new variable `new_string` to store your result.<jupyter_code>my_string = "banana"
new_string = # your code here<jupyter_output><empty_output><jupyter_text>Exercise 4: Find a way to fix the spacing problem below keeping the "+".<jupyter_code>name = "<NAME>"
alterego = "The Hulk"
colour = "Green"
country = "USA"

print("His name is" + name + "and his alter ego is" + alterego + 
      ", a big" + colour + "superhero from the" + country + ".")<jupyter_output><empty_output><jupyter_text>How would you print the same sentence using ","?<jupyter_code>name = "<NAME>"
alterego = "The Hulk"
colour = "Green"
country = "USA"

print("His name is" + name + "and his alter ego is" + alterego + 
      ", a big" + colour + "superhero from the" + country + ".")<jupyter_output><empty_output><jupyter_text>Can you rewrite the code below using an f-string?<jupyter_code>name = "<NAME>"
alterego = "The Hulk"
colour = "green"
country = "the USA"
birth_year = 1969
current_year = 2017

print("His name is " + name + " and his alter ego is " + alterego + 
      ", a big " + colour + " superhero from " + country + ". He was born in " + str(birth_year) + 
      ", so he must be " + str(current_year - birth_year - 1) + " or " + str(current_year - birth_year) + 
      " years old now.")<jupyter_output><empty_output><jupyter_text>Exercise 5: Replace all a's by o's in 'banana' using a string method.<jupyter_code>my_string = "banana"
# your code here<jupyter_output><empty_output><jupyter_text>Remove all spaces in the sentence using a string method.<jupyter_code>my_string = "<NAME> sat on the wall"
# your code here<jupyter_output><empty_output><jupyter_text>What do the methods `lstrip()` and `rstrip()` do? Try them out below.<jupyter_code># find out what lstrip() and rstrip() do<jupyter_output><empty_output><jupyter_text>What do the methods `startswith()` and `endswith()` do? Try them out below.<jupyter_code># find out what startswith() and endswith() do<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09977861498427001, 'fraction_numerical': 0.007962092670990795, 'mean_word_length': 4.997670626601444, 'pattern_counts': {'":': 0, '<': 182, '<?xml version=': 0, '>': 181, 'https://': 12, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '131879', 'license': 'Apache-2.0', 'n_tokens_mistral': 7749, 'n_tokens_neox': 7420, 'n_words': 3312, 'path': 'Chapters/Chapter 03 - Strings.ipynb', 'repo_name': 'rvrheenen/python-for-text-analysis', 'size': '25793'}