<jupyter_start><jupyter_text>_Module 2 lesson 1_: Introduction to Jupyter Notebook and programming in Python    Learning outcomes:                Download and install Anaconda with Jupyter Notebook.        Open your first Jupyter Notebook document.        Learn version control and code management with Git and GitHub.        Write your first line of code.        Learn and apply the core Python variables, types and operators.        Employ variables, types and operators in loops and conditions to perform simple tasks.        Capture and respond efficiently to exceptions.        Use sets to extract unique data from lists.        Develop and use reusable code by encapsulating tasks in functions.        Package functions into flexible and extensible classes.    ---  1.1 Installing Python and Jupyter Notebook with Anaconda![Jupyter overview](images/jupyter-overview.jpg)Anaconda is a suite of popular data science packages for Python and includes Jupyter Notebook which is an interactive notebook allowing you to create and share documents containing live code. This is a Jupyter Notebook and you'll be able to follow along learn as you go.Our first objective is to install Anaconda and create a Jupyter Notebook so that you can start this tutorial. If you want to try it in a browser (including their own tutorial), then [follow this link](https://jupyter.org/try).To install Anaconda, and all the Python packages so that you can work on your own computer:- Go to the [download page on Anaconda](https://www.anaconda.com/download/);![Download Anaconda](images/anaconda-1.jpg)- Definitely choose the **Python 3.x version** for installation. Select the appropriate package for your computer (Windows, MacOS, or Linux). If you're on Windows, you'll also need to decide on 64-bit or 32-bit. Most computers now support 64-bit but, if you're unsure, pick the 32-bit version.- After downloading, run the software and install all the defaults.- You'll see you have the option to launch Anaconda, or Jupyter Notebook. For now, we'll go straight into Jupyter Notebook.- The first page that opens in the Jupyter Notebook folder (or tree) view. Here you can see a list of files. Jupyter Notebooks end with the extension ".ipynb"![Jupyter Notebook tree view](images/jupyter-01.png)- Create a new notebook by clicking on "New" and select "Python 3" as the kernel you wish to run.![Jupyter Notebook create new](images/jupyter-02.png)- This is a new notebook and is what you should see:![Jupyter Notebook](images/jupyter-04.png)- You can get a tour of the notebook interface from `Help -> User Interface Tour`, or you can learn some shortcuts from `Help -> Keyboard shortcuts`.![Jupyter Notebook shortcuts](images/jupyter-03.png)You will need to launch Jupyter Notebooks every time you want to work on these tutorials. Remember to save your `.ipynb` files where you can easily find them. You can also download all of the notebooks from this repository and, if you're running them in Jupyter Notebook, you can run the code here, or click on the text to see how the MarkDown version of HTML is written to code this text.In the next section, you'll learn how to clone this repository.    References:                Anaconda        Project Jupyter        Conda Cheatsheet    ---  1.2 Version control and code management with Git and GitHubThe following tutorial is adapted from [GitHub's 'Hello World' tutorial](https://guides.github.com/activities/hello-world/) and their [Git Handbook](https://guides.github.com/introduction/git-handbook/). GitHub is a particular implementation of Git. It happens to be where these lessons are hosted, but you are welcome to use any other service, such as [GitLab](https://about.gitlab.com/), [Bitbucket](https://bitbucket.org/) or even the venerable [SourceForge](https://sourceforge.net/).[Git](https://git-scm.com/) is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Version control systems track the history of changes as people and teams collaborate on projects together. As the project evolves, teams can run tests, fix bugs, and contribute new code with the confidence that any version can be recovered at any time. Developers can review project history to find out:- Which changes were made?- Who made the changes?- When were the changes made?- Why were changes needed?Git itself runs from the command line (that is, in a terminal window, and outside of the more familiar graphical interface you use to interact with your computer's operating system). There are many commands to learn, but we'll only run through the most basic subset you need for now. Git has a [Basics Video Series](https://git-scm.com/video/get-going) you can follow to learn more.A _repository_, or Git project, encompasses the entire collection of files and folders associated with a project, along with each file’s revision history. The file history appears as snapshots in time called _commits_, and the commits exist as a linked-list relationship, and can be organized into multiple lines of development called _branches_. Because Git is a distributed version control system, repositories are self-contained units and anyone who owns a copy of the repository can access the entire codebase and its history. Using the command line or other ease-of-use interfaces, a git repository also allows for: interaction with the history, cloning, creating branches, committing, merging, comparing changes across versions of code, and more.Working in repositories keeps development projects organized and protected. Developers are encouraged to fix bugs, or create fresh features, without fear of derailing mainline development efforts. Git facilitates this through the use of topic branches: lightweight pointers to commits in history that can be easily created and deprecated when no longer needed.Through platforms like GitHub, Git also provides more opportunities for project transparency and collaboration. Public repositories help teams work together to build the best possible final product.This lesson, for example, exists in a public, open-source GitHub repository with Git version control.[GitHub](https://github.com) is a Git hosting repository that provides developers with tools to ship better code through command line features, issues (threaded discussions), pull requests, and code review. You can learn more about [GitHub's _flow_ here](https://guides.github.com/introduction/flow/).To complete the next part of this tutorial, you will need a [GitHub.com account](http://github.com/) and internet access. You don’t need to know how to code, use the command line, or install Git.![Create a GitHub Account](images/01-02-01-github-create-account.jpg) Create a Tutorial RepositoryA _repository_ is usually used to organize a single project. Repositories can contain folders and files, images, videos, spreadsheets, and data sets – anything your project needs. We recommend including a README, or a file with information about your project. GitHub makes it easy to add one at the same time you create your new repository. It also offers other common options such as a license file.Your `hello-world` repository can be a place where you store ideas, resources, or even share and discuss things with others. You can call it anything you like, but you will need a repository for this lesson and subsequent tutorials, so you may as well use this as an opportunity to get started. Maybe call it `data-wrangling-course`.    For free GitHub accounts, everything you do is public. Don't store important personal information, or express behaviours, that may embarrass you or risk your safety.To create a new repository, you need to be logged in to your GitHub account. Then: - In the upper right corner, next to your avatar or identicon, click __+__ and then select __New repository__.- Name your repository `data-wrangling-course` (or whatever you prefer).- Write a short description.- Select __Initialize this repository with a README__.![Create new repo](images/github-create-new-repo.png)- Click __Create repository__. Create a Branch_Branching_ is the way to work on different versions of a repository at one time.By default your repository has one branch named `master` which is considered to be the definitive branch. We use branches to experiment and make edits before committing them to `master`.When you create a branch off the `master` branch, you’re making a copy, or snapshot, of `master` as it was at that point in time. If someone else made changes to the master branch while you were working on your branch, you could pull in those updates.This diagram shows:- The `master` branch- A new branch called `feature` (because we're doing 'feature work' on this branch)- The journey that `feature` takes before it's merged into `master`![Branching](images/github-branching.png)Have you ever saved different versions of a file? Something like:- `story.txt`- `story-joe-edit.txt`- `story-joe-edit-reviewed.txt`Branches accomplish similar goals in GitHub repositories.At GitHub, developers, writers, and designers use branches for keeping bug fixes and feature work separate from master (production) branch. When a change is ready, they merge their branch into master.To create a new branch:- Go to your new repository `data-wrangling-course`.- Click the drop down at the top of the file list that says __branch: master__.- Type a branch name, `readme-edits`, into the new branch text box.- Select the blue __Create branch__ box or hit _Enter_ on your keyboard.![New branch](images/github-readme-edits.gif)Now you have two branches, `master` and `readme-edits`. They look exactly the same, but not for long! Next we’ll add our changes to the new branch. Make and commit changesNow, you’re on the code view for your `readme-edits` branch, which is a copy of `master`. Let’s make some edits.On GitHub, saved changes are called _commits_. Each commit has an associated _commit message_, which is a description explaining why a particular change was made. Commit messages capture the history of your changes, so other contributors can understand what you’ve done and why.Make and commit changes:- Click the `README.md` file.- Click the pencil icon in the upper right corner of the file view to edit.- In the editor, write a bit about yourself.- Write a commit message that describes your changes.- Click __Commit changes__ button.![Commit](images/github-commit.png)These changes will be made to just the `README` file on your `readme-edits` branch, so now this branch contains content that’s different from `master`. Open a Pull RequestNow that you have changes in a branch off of `master`, you can open a _pull request_.Pull Requests are the heart of collaboration on GitHub. When you open a _pull request_, you’re proposing your changes and requesting that someone review and pull in your contribution and merge them into their branch. Pull requests show _diffs_, or differences, of the content from both branches. The changes, additions, and subtractions are shown in green and red.As soon as you make a commit, you can open a pull request and start a discussion, even before the code is finished.By using GitHub’s @mention system in your pull request message, you can ask for feedback from specific people or teams, whether they’re down the hall or 10 time zones away.You can even open pull requests in your own repository and merge them yourself. It’s a great way to learn the GitHub flow before working on larger projects.__Open a Pull Request for changes to the README:__| __Step__ | __Screenshot__                   ||:------------|:-------------------------|| Click the  __Pull Request__ tab, then from the Pull Request page, click the green __New pull request__ button.  | ![Pull Request tab](images/github-pr-tab.gif) || In the __Example Comparisons__ box, select the branch you made, `readme-edits`, to compare with `master` (the original). | ![Pick Branch](images/github-pick-branch.png) || Look over your changes in the diffs on the Compare page, make sure they’re what you want to submit. | ![Diff](images/github-diff.png) || When you’re satisfied that these are the changes you want to submit, click the big green __Create Pull Request__ button. | ![Create Pull Request](images/github-create-pr.png) || Give your pull request a title and write a brief description of your changes. | ![Pull Request Form](images/github-pr-form.png) |When you’re done with your message, click __Create pull request!__ Merge your Pull RequestIn this final step, it’s time to bring your changes together – merging your readme-edits branch into the master branch.- Click the green __Merge pull request__ button to merge the changes into master.- Click __Confirm merge__.- Go ahead and delete the branch, since its changes have been incorporated, with the __Delete branch__ button in the purple box.![Merge](images/github-merge-button.png)![Delete](images/github-delete-button.png)And that completes the GitHub tutorial. This is what you've done:- Created an open source repository- Started and managed a new branch- Changed a file and committed those changes to GitHub- Opened and merged a Pull Request    Exercise:    GitHub has a Desktop App which you can use to manage your repositories and code directly from your computer. It has an easy-to-use UI for you and will make cloning, editing, pushing and pulling changes to your repositories much easier.    For this exercise:            Review GitHub's desktop app tutorial on your own.        Install the app.        You don't have to, but you might find it easier to also install Notepad++ to edit code and text.        Redo this GitHub tutorial, but directly via the app and on your own desktop. You can edit your `README` file in Notepad++, then save it and push your changes to your repository using the app.        You should create a code directory to save all your code projects. Being very neat and organised about code directories is critical to managing your repositories. Get into the habit now.    In the next section, you'll start learning to code in Python.    Tutorials and References:                GitHub's 'Hello World' tutorial        Git Basics Get Going        GitHub Guides        GitHub YouTube channel        GitHub Desktop App    ---  1.3 Python basicsPython is an interpreted language with a very simple syntax. The first step in learning any new language is producing "Hello, World!".You will have opened a new notebook in __Jupyter Notebook__:![Jupyter Notebook](images/jupyter-04.png)You should save this notebook in a folder dedicated to your tutorials for this course. Give it a name, like `Python coding tutorial 1`, or whatever works for you. You might end up with a directory tree that looks like this:    documents/github-code/data-wrangling-course/Note two things:- The dropdown menu item that says `code`- The layout of the text entry box below that (which starts with `In []:`)This specifies that this block is to be used for code. You can also use it for structured text blocks like this one by selecting `markdown` from the menu list. This tutorial won't teach `markdown`, but you can learn more [here](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet).The tutorial which follows is similar to [Learn Python's](https://www.learnpython.org/), although here you will run the examples on your own computer. Hello, World!For now, let's start with "Hello, World!".Type the below text into the code block in your notebook and hit `Ctrl-Enter` to execute the code:    print("Hello, World!")<jupyter_code>print("Hello, World!")<jupyter_output>Hello, World!<jupyter_text>You just ran your first program. See how Jupyter performs code highlighting for you, changing the colours of the text depending on the nature of the syntax used. `print` is a protected term and gets highlighted in green.This also demonstrates how useful Jupyter Notebook can be. You can treat this just like a document, saving the file, and storing the outputs of your program in the notebook. You could even email this to someone else and, if they have Jupyter Notebook, they could run the notebook and see what you have done. If you save your work in GitHub, you can share links with others on this course and get help or advice if you get stuck.From here on out, we'll simply move through the Python coding tutorial and learn syntax and methods for coding. IndentationPython uses indentation to indicate parts of the code that needs to be executed together. Both tabs and spaces (usually four per level) are supported, and my preference is for tabs. This is the subject of mass debate, but don't worry about it. Whatever you decide to do is fine, but **do not - under any circumstances - mix tab indentation with space indentation.**In this exercise you'll assign a value to a variable, check to see if a comparison is true, and then - based on the result - print.First, spot the little `+` symbol on the menu bar just after the `save` symbol. Click that and you'll get a new box to type the following code into. When you're done, press `Ctrl-Enter` or the `Run` button.<jupyter_code>x = 1
if x == 1:
    # Indented ... and notice how Jupyter Notebook automatically indented for you
    print("x is 1")<jupyter_output>x is 1<jupyter_text>Any non-protected text term can be a variable. Your `x` could just have easily been your name. Usually we name variables as descriptively as possible so that we can read algorithms like text (i.e. the code describes itself).    Syntax                To assign a variable with a specific value, use =        To test whether a variable has a specific value, use the boolean operators:                    equal: ==            not equal: !=            greater-than: &gt;            less-than: &lt;                You can also add helpful comments to your code with the  symbol. Any line starting with a  is not executed. I find it very useful to make detailed notes about my thinking since, often, when you come back to code later you can't remember why you did what you did, or what your code is even supposed to do.     Variables and TypesPython is not "statically-typed". This means you do not have to declare all your variables before you can use them. You can create new variables whenever you want. Python is also "object-oriented", which means that every variable is an object. That will become more important the more experienced you get.Lets go through the core types of variables: NumbersPython supports two types of numbers: integers and floats. Integers are whole numbers (e.g. 7), while floats are fractional (e.g. 7.321). You can also convert integers to floats, and vice versa, but you need to be aware of the risks of doing so.Follow along with the code:<jupyter_code>myint = 7
print(myint)
myfloat = 7.0
print(myfloat)
# Or you could convert the integer you already have
myfloat = float(myint)
# Note how the term `float` is green. It's a protected term.
print(myfloat)
# Now see what happens when you convert a float to an int
myint = int(7.3)
print(myint)<jupyter_output>7<jupyter_text>Note how you lost precision when you converted a `float` to an `int`? Always be careful, since that could be the difference between a door which fits its frame, and one which is far too small. StringsStrings are the Python term for text. You can define these in either single or double quotes. I'll be using double quotes (since you often use a single quote inside text phrases).Try these examples:<jupyter_code>mystring = "Hello, World!"
print(mystring)
# and demonstrating how to use an apostrophe in a string
mystring = "Let's talk about apostrophes..."
print(mystring)<jupyter_output>Hello, World!
Let's talk about apostrophes...<jupyter_text>You can also apply simple operators to your variables, or assign multiple variables simultaneously.<jupyter_code>one = 1
two = 2
three = one + two
print(three)

hello = "Hello,"
world = "World!"
helloworld = hello + " " + world
print(helloworld)

a, b = 3, 4
print(a, b)<jupyter_output>3
Hello, World!
3 4<jupyter_text>Note, though, that mixing variable types causes problems.<jupyter_code>print(one + two + hello)<jupyter_output><empty_output><jupyter_text>Python will throw an error when you make a mistake like this and the error will give you as much detail as it can about what just happened. This is extremely useful when you're attempting to "debug" your code.In this case, you're told: `TypeError: unsupported operand type(s) for +: 'int' and 'str'`And the context should make it clear that you tried to combine two integer variables with a string.You can also combine strings with placeholders for variables:    Syntax                Add variables to a string with F-strings, e.g. F"Variable {x}" will replace the {x} in the string with the value in the variable x        Floating point numbers can get out of hand (imagine including a number with 30 decimal places in a string), and you can format this with F"Variable {x:10.4f}" where 10 is the amount of space allocated to the float (useful if you need to align a column of numbers; you can also leave this out to include all significant digits) and the .4f is the number of decimals after the point. Vary these as you need.<jupyter_code>variable = 1/3 * 100
print(F"Unformated variable: {variable}%")
print(F"Formatted variable: {variable:.3f}%")
print(F"Space formatted variable: {variable:10.1f}%")<jupyter_output>Unformated variable: 33.33333333333333%
Formatted variable: 33.333%
Space formatted variable:       33.3%<jupyter_text>Lists`Lists` are an ordered list of any type of variable. You can combine as many variables as you like, and they could even be of multiple types. Ordinarily, unless you have a specific reason to do so, lists will contain variables of one type.You can also iterate over a list (use each item in a list in sequence).A list is placed between square brackets: `[]`<jupyter_code>mylist = []
mylist.append(1)
mylist.append(2)
mylist.append(3)
# Each item in a list can be addressed directly. 
# The first address in a Python list starts at 0
print(mylist[0])
# The last item in a Python list can be addressed as -1. 
# This is helpful when you don't know how long a list is likely to be.
print(mylist[-1])
# You can also select subsets of the data in a list like this
print(mylist[1:3])

# You can also loop through a list using a `for` statement.
# Note that `x` is a new variable which takes on the value of each item in the list in order.
for x in mylist:
    print(x)<jupyter_output>1
3
[2, 3]
1
2
3<jupyter_text>If you try access an item in a list that isn't there, you'll get an error.<jupyter_code>print(mylist[10])<jupyter_output><empty_output><jupyter_text>Let's put this together with a slightly more complex example. But first, some new syntax:    Syntax                Check what type of variable you have with isinstance, e.g. isinstance(x, float) will be True if x is a float        You've already seen for, but you can get the loop count by wrapping your list in the term enumerate, e.g. for count, x in enumerate(mylist) will give you a count for each item in the list        Sorting a list into numerical or alphabetical order can be done with sort        Getting the number of items in a list is as simple as asking len(list)        If you want to count the number of times a particular variable occurs in a list, use list.count(x) (where x is the variable you're interested in)    Try this for yourself.<jupyter_code># Let's imagine we have a list of unordered names that somehow got some random numbers included.
# For this exercise, we want to print the alphabetised list of names without the numbers.
# This is not the best way of doing the exercise, but it will illustrate a whole bunch of techniques.
names = ["John", 3234, 2342, 3323, "Eric", 234, "Jessica", 734978234, "Lois", 2384]
print(F"Number of names in list: {len(names)}")
# First, let's get rid of all the weird integers.
new_names = []
for n in names:
    if isinstance(n, str):
        # Checking if n is a string
        # And note how we're now indented twice into this new component
        new_names.append(n)
# We should now have only names in the new list. Let's sort them.
new_names.sort()
print(F"Cleaned-up number of names in list: {len(new_names)}")
# Lastly, let's print them.
for i, n in enumerate(new_names):
    # Using both i and n in a formated string
    # Adding 1 to i because lists start at 0
    print(F"{i+1}. {n}")<jupyter_output>Number of names in list: 10
Cleaned-up number of names in list: 4
1. Eric
2. Jessica
3. John
4. Lois<jupyter_text>DictionariesDictionaries are one of the most useful and versatile data types in Python. They're similar to arrays, but consist of `key:value pairs`. Each value stored in a dictionary is accessed by its key, and the value can be any sort of object (string, number, list, etc.).This allows you to create structured records. Dictionaries are placed within `{}`.<jupyter_code>phonebook = {}
phonebook["John"] = {"Phone": "012 794 794",
                     "Email": "<EMAIL>"}
phonebook["Jill"] = {"Phone": "012 345 345",
                     "Email": "<EMAIL>"}
phonebook["Joss"] = {"Phone": "012 321 321",
                     "Email": "<EMAIL>"}
print(phonebook)<jupyter_output>{'John': {'Phone': '012 794 794', 'Email': '<EMAIL>'}, 'Jill': {'Phone': '012 345 345', 'Email': '<EMAIL>'}, 'Joss': {'Phone': '012 321 321', 'Email': '<EMAIL>'}}<jupyter_text>Note that you can nest dictionaries and lists. The above shows you how you can add values to an existing dictionary, or create dictionaries with values.You can iterate over a dictionary just like a list, using the dot term `.items()`. In Python 3, the dictionary maintains the order in which data were added, but older versions of Python don't.<jupyter_code>for name, record in phonebook.items():
    print(F"{name}'s phone number is {record['Phone']}, and their email is {record['Email']}")<jupyter_output>John's phone number is 012 794 794, and their email is <EMAIL>
Jill's phone number is 012 345 345, and their email is <EMAIL>
Joss's phone number is 012 321 321, and their email is <EMAIL><jupyter_text>You add new records as shown above, and you remove records with `del` or `pop`. They each have a different effect.<jupyter_code># First `del`
del phonebook["John"]
for name, record in phonebook.items():
    print("{}'s phone number is {}, and their email is {}".format(name, record["Phone"], record["Email"]))

# Pop returns the record, and deletes it
jill_record = phonebook.pop("Jill")
print(jill_record)
for name, record in phonebook.items():
    # You can see that only Joss is still left in the system
    print("{}'s phone number is {}, and their email is {}".format(name, record["Phone"], record["Email"]))

# If you try and delete a record that isn't in the dictionary, you get an error
del phonebook["John"]<jupyter_output>Jill's phone number is 012 345 345, and their email is <EMAIL>
Joss's phone number is 012 321 321, and their email is <EMAIL>
{'Phone': '012 345 345', 'Email': '<EMAIL>'}
Joss's phone number is 012 321 321, and their email is <EMAIL><jupyter_text>One thing to get into the habit of doing, is to test variables before assuming they have characteristics you're looking for. You can test a dictionary to see if it has a record, and return some default answer if it doesn't have it.You do this with the `.get("key", default)` term. `Default` can be anything, including another variable, or simply `True` or `False`. If you leave `default` blank (i.e. `.get("key")`), then the result will automatically be `False` if there is no record.<jupyter_code># False and True are special terms in Python that allow you to set tests
jill_record = phonebook.get("Jill", False)
if jill_record: # i.e. if you got a record in the previous step
    print(F"Jill's phone number is {jill_record['Phone']}, and their email is {jill_record['Email']}")
else: # the alternative, if `if` returns False
    print("No record found.")<jupyter_output>No record found.<jupyter_text>Basic operatorsOperators are the various algebraic symbols (such as `+`, `-`, `*`, `/`, `%`, etc.). Once you've learned the syntax, programming is mostly mathematics. Arithmetic operatorsAs you would expect, you can use the various mathematical operators with numbers (both integers and floats).<jupyter_code>number = 1 +2 * 3 / 4.0
# Try to predict what the answer will be ... does Python follow order operations hierarchy?
print(number)

# The modulo (%) returns the integer remainder of a division
remainder = 11 % 3
print(remainder)

# Two multiplications is equivalent to a power operation
squared = 7 ** 2
print(squared)
cubed = 2 ** 3
print(cubed)<jupyter_output>2.5
2
49
8<jupyter_text>List operators<jupyter_code>even_numbers = [2, 4, 6, 8]
# One of my first teachers in school said, "People are odd. Numbers are uneven."
# He also said, "<NAME> always ate piles of unshelled peanuts in parliament in Cape Town."
# "You'd know he'd been in parliament by the huge pile of shells on the floor. He also never wore socks."
# "You'll never forget this." And I didn't. I have no idea if it's true.
uneven_numbers = [1, 3, 5, 7]
all_numbers = uneven_numbers + even_numbers
# What do you think will happen?
print(all_numbers)

# You can also repeat sequences of lists
print([1, 2 , 3] * 3)<jupyter_output>[1, 3, 5, 7, 2, 4, 6, 8]
[1, 2, 3, 1, 2, 3, 1, 2, 3]<jupyter_text>We can put this together into a small project.<jupyter_code>x = object() # A generic Python object
y = object()

# Change this code to ensure that x_list and y_list each have 10 repeating objects
# and concat_list is the concatenation of x_list and y_list
x_list = [x]
y_list = [y]
concat_list = []

print(F"x_list contains {len(x_list)} objects")
print(F"y_list contains {len(y_list)} objects")
print(F"big_list contains {len(concat_list)} objects")

# Test your lists
if x_list.count(x) == 10 and y_list.count(y) == 10:
    print("Almost there...")
if concat_list.count(x) == 10 and concat_list.count(y) == 10:
    print("Great!")<jupyter_output>x_list contains 1 objects
y_list contains 1 objects
big_list contains 0 objects<jupyter_text>String operatorsYou can do a surprising amount with operators on strings.<jupyter_code># You've already seen arithmetic concatenations of strings
helloworld = "Hello," + " " + "World!"
print(helloworld)

# You an also multiply strings to form a repeating sequence
manyhellos = "Hello " * 10
print(manyhellos)

# But don't get carried away. Not everything will work.
nohellos = "Hello " / 10
print(nohellos)<jupyter_output>Hello, World!
Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello<jupyter_text>Something to keep in mind is that strings are lists of characters. This means you can perform a number of list operations on strings. And a few new operations.    Syntax                Get the index for the first occurrence of a specific letter with string.index("l") where l is the letter you're looking for        As in lists, count the number of occurrences of a specific letter with string.count("l")        Get slices of strings with string[start:end], e.g. string[3:7]. If you're unsure of the end of a string, remember you can use negative numbers to count from the end, e.g. string[:-3] to get a slice from the first character to the third from the end        You can also "step" through a string with string[start:stop:step], e.g. string[2:6:2] which will skip a character between the characters 2 and 5 (i.e. 6 is the boundary)        You can use a negative "step" to reverse the order of the characters, e.g. string[::-1]        You can convert strings to upper- or lower-case with string.upper() and string.lower()        Test whether a string starts or ends with a substring with:                    string.startswith(substring) which returns True or False            string.endswith(substring) which returns True or False                Use in to test whether a string contains a substring, so substring in string will return True or False        You can split a string into a genuine list with .split(s) where s is the specific character to use for splitting, e.g. s = "," or s = " ". You can see how this might be useful to split up text which contains numeric data.<jupyter_code>a_string = "Hello, World!"
print(F"String length: {len(a_string)}")
# You can get an index of the first occurence of a specific letter
# Remember that Python lists are based at 0; the first letter is index 0
# Also note the use of single quotes inside the double quotes
print(F"Index for first 'o': {a_string.index('o')}")
print(F"Count of 'o': {a_string.count('o')}")
print(F"Slicing between second and fifth characters: {a_string[2:6]}")
print(F"Skipping between 3rd and 2nd-from-last characters: {a_string[3:-2:2]}")
print(F"Reverse text: {a_string[::-1]}")
print(F"Starts with 'Hello': {a_string.startswith('Hello')}")
print(F"Ends with 'Hello': {a_string.endswith('Hello')}")
print(F"Contains 'Goodbye': {'Goodby' in a_string}")
print(F"Split the string: {a_string.split(' ')}")<jupyter_output>String length: 13
Index for first 'o': 4
Count of 'o': 2
Slicing between second and fifth characters: llo,
Skipping between 3rd and 2nd-from-last characters: l,Wr
Reverse text: !dlroW ,olleH
Starts with 'Hello': True
Ends with 'Hello': False
Contains 'Goodbye': False
Split the string: ['Hello,', 'World!']<jupyter_text>ConditionsIn the section on [Indentation](Indentation) you were introduced to the `if` statement and the set of `boolean` operators that allow you to test different variables against each other.To that list of boolean operators are added a new set of comparisons: `and`, `or` and `in`.<jupyter_code># Simple boolean tests
x = 2
print(x == 2)
print(x == 3)
print(x < 3)

# Using `and`
name = "John"
print(name == "John" and x == 2)

# Using `or`
print(name == "John" or name == "Jill")

# Using `in` on lists
print(name in ["John", "Jill", "Jess"])<jupyter_output>True
False
True
True
True
True<jupyter_text>These can be used to create nuanced comparisons using `if`. You can use a series of comparisons with `if`, `elif` and `else`.Remember that code must be indented correctly or you will get unexpected behaviour.<jupyter_code># Unexpected results
x = 2
if x > 2:
    print("Testing x")
print("x > 2")
# Formated correctly
if x == 2:
    print("x == 2")
# Demonstrating more complex if tests
x = 2
y = 10
if x > 2:
    print("x > 2")
elif x == 2 and y > 50:
    print("x == 2 and y > 50")
elif x < 10 or y > 50:
    # But, remember, you don't know WHICH condition was True
    print("x < 10 or y > 50")
else:
    print("Nothing worked.")<jupyter_output>x < 10 or y > 50<jupyter_text>Two special cases are `not` and `is`.    Syntax                not is used to get the opposite of a particular boolean test, e.g. not(False) returns True        is would seem, superficially, to be similar to ==, but it tests for whether the actual objects are the same, not whether the values which the objects reflect are equal.    A quick demonstration.<jupyter_code># Using `not`
name_list1 = ["John", "Jill"]
name_list2 = ["John", "Jill"]
print(not(name_list1 == name_list2))

# Using `is`
name2 = "John"
for n in name_list1:
    print(F"{name2} is {n}: {name2 is n}")
print(name_list1 == name_list2)
print(name_list1 is name_list2)<jupyter_output>False
John is John: True
John is Jill: False
True
False<jupyter_text>LoopsLoops iterate over a given sequence, and - here - it is critical to ensure your indentation is correct or you'll get unexpected results for what is considered inside or outside the loop.There are two types of loop in Python:    Syntax                For loops, for, which loop through a list. There is also some new syntax to use in for loops:                    In Lists you saw enumerate, which allows you to count the loop number            Range creates a list of integers to loop, range(start, stop) creates a list of integers between start and stop, or range(num) creates a zero-based list up to num, or range(start, stop, step) steps through a list in increments of step                While loops, while, which execute while a particular condition is True. And some new syntax for while:                    while is a conditional statement (it requires a test to return True), that means we can use else in a while loop (but not for)<jupyter_code># For loops

for i, x in enumerate(range(2, 8, 2)):
    print(F"{i+1}. Range {x}")
    
# While loops
count = 0
while count < 5:
    print(count)
    count += 1 # A shorthand for count = count + 1
else:
    print("End of while loop reached")<jupyter_output>1. Range 2
2. Range 4
3. Range 6
0
1
2
3
4
End of while loop reached<jupyter_text>Pay close attention to the indentation in that `while` loop. What would happen if `count += 1` were outside the loop?What happens if you need to exit loops early, or miss a step?    Syntax                break exits a while or for loop immediately        continue skips the current loop and returns to the loop conditional<jupyter_code># Break and while conditional
print("Break and while conditional")
count = 0
while True:
    # You may think this would run forever, but ...
    print(count)
    count += 1
    if count >= 5:
        break

# Continue
print("Continue")
for x in range(8):
    # Check if x is uneven
    if (x+1) % 2 == 0:
        continue
    print(x)<jupyter_output>Break and while conditional
0
1
2
3
4
Continue
0
2
4
6<jupyter_text>List comprehensionsOne of the common tasks in coding is to go through a list of items, edit or apply some form of algorithm, and return a new list.Writing long stretches of code to accomplish this is tedious and time-consuming. List comprehensions are an efficient and concise way of achieving exactly that.As an example, imagine we have a sentence where we want to count the length of each word but skip all the "the"s:<jupyter_code>sentence = "for the song and the sword are birthrights sold to an usurer, but I am the last lone highwayman and I am the last adventurer"
words = sentence.split()
word_lengths = []
for word in words:
      if word != "the":
            word_lengths.append(len(word))
print(word_lengths)

# The exact same thing can be achieved with a list comprehension
word_lengths = [len(word) for word in sentence.split(" ") if word != "the"]
print(word_lengths)<jupyter_output>[3, 4, 3, 5, 3, 11, 4, 2, 2, 7, 3, 1, 2, 4, 4, 10, 3, 1, 2, 4, 10]
[3, 4, 3, 5, 3, 11, 4, 2, 2, 7, 3, 1, 2, 4, 4, 10, 3, 1, 2, 4, 10]<jupyter_text>Exception handlingFor the rest of this section of the tutorial, we're going to focus on some more advanced syntax and methodology.In [Strings](Strings) you say how the instruction to concatenate a string with an integer using the `+` operator resulted in an error:<jupyter_code>print(1 + "hello")<jupyter_output><empty_output><jupyter_text>In Python, this is known as an `exception`. The particular exception here is a `TypeError`. Getting exceptions is critical to coding, because it permits you to fix syntax errors, catch glitches where you pass the wrong variables, or your code behaves in unexpected ways.However, once your code goes into production, errors that stop your program entirely are frustrating for the user. More often than not, there is no way to exclude errors and sometimes the only way to find something out is to try it and see if the function causes an error. Many of these errors are entirely expected. For example, if you need a user to enter an integer, you want to prevent them typing in text. Or - in this era of mass hacking - you want to prevent a user trying to including their own code in a text field.When this happens what you want is to way to try to execute your code and then catch any expected exceptions safely.    Syntax                Test and catch exceptions with try and except        Catch specific errors, rather than all errors, since you still need to know about anything unexpected ... otherwise you can spend hours trying to find a mistake which is being deliberately ignored by your program.        Chain exceptions with, e.g. except (IndexError, TypeError):. Here is a link to all the common exceptions.<jupyter_code># An `IndexError` is thrown when you try to address an index in a list that does not exist
# In this example, let's catch that error and do something else

def print_list(l):
    """
    For a given list `l`, of unknown length, try to print out the first
    10 items in the list.
    
    If the list is shorter than 10, fill in the remaining items with `0`.
    """
    for i in range(10):
        try:
            print(l[i])
        except IndexError: 
            print(0)

print_list([1,2,3,4,5,6,7])<jupyter_output>1
2
3
4
5
6
7
0
0
0<jupyter_text>You can also deliberately trigger an exception with `raise`. To go further, and write your own types of exceptions, consider [this explanation](https://stackoverflow.com/a/26938914).<jupyter_code>def print_zero(zero):
    if zero != 0:
        raise ValueError("Not Zero!")
    print(zero)

print_zero(10)<jupyter_output><empty_output><jupyter_text>SetsSets are lists with no duplicate entries. You could probably write a sorting algorithm, or dictionary, to achieve the same end, but sets are faster and more flexible.<jupyter_code># Extract all unique terms in this sentence

print(set("the rain is wet and wet is the rain".split()))<jupyter_output>{'wet', 'is', 'rain', 'the', 'and'}<jupyter_text>Syntax                Create a unique set of terms with set        To get members of a set common to both of two sets, use set1.intersection(set2)        Get the unique members of each of one set and another, use set1.symmetric_difference(set2)        To get the unique members from the asking set (i.e. the one calling the dot function), use set1.difference(set2)        To get all the members of each of two lists, use set1.union(set2)<jupyter_code>set_one = set(["Alice", "Carol", "Dan", "Eve", "Heidi"])
set_two = set(["Bob", "Dan", "Eve", "Grace", "Heidi"])

# Intersection
print(F"Set One intersection: {set_one.intersection(set_two)}")
print(F"Set Two intersection: {set_two.intersection(set_one)}")

# Symmetric difference
print(F"Set One symmetric difference: {set_one.symmetric_difference(set_two)}")
print(F"Set Two symmetric difference: {set_two.symmetric_difference(set_one)}")

# Difference
print(F"Set One difference: {set_one.difference(set_two)}")
print(F"Set Two difference: {set_two.difference(set_one)}")

# Union
print(F"Set One union: {set_one.union(set_two)}")
print(F"Set Two union: {set_two.union(set_one)}")<jupyter_output>Set One intersection: {'Eve', 'Dan', 'Heidi'}
Set Two intersection: {'Eve', 'Dan', 'Heidi'}
Set One symmetric difference: {'Alice', 'Carol', 'Grace', 'Bob'}
Set Two symmetric difference: {'Alice', 'Carol', 'Grace', 'Bob'}
Set One difference: {'Alice', 'Carol'}
Set Two difference: {'Grace', 'Bob'}
Set One union: {'Alice', 'Dan', 'Heidi', 'Bob', 'Eve', 'Carol', 'Grace'}
Set Two union: {'Alice', 'Dan', 'Heidi', 'Bob', 'Eve', 'Carol', 'Grace'}<jupyter_text>And that marks the end of this section of the tutorial.  1.4 Intermediate PythonThe code from Python basics was limited to short snippets. Solving more complex problems means more complex code stretching over hundreds, to thousands, of lines, and - if you want to reuse that code - it's not convenient to copy and paste it multiple times. Worse, any error is magnified, and any changes become tedious to manage.It would be far better to write a discrete module to contain that task, get it absolutely perfect, and call it whenever you want the same problem solved or task executed.In software, this process of packaging up discrete functions into their own modular code is called "abstraction". A complete software system consists of a number of discrete modules all interacting to produce an integrated experience.In Python, these modules are called `functions` and a complete suite of functions grouped around a set of related tasks is called a `library` or `module`. Libraries permit you to inherit a wide variety of powerful software solutions developed and maintained by other people.Python is [open source](https://en.wikipedia.org/wiki/Open-source_software), which means that its source code is released under a licence which permits anyone to study, change, and distribute the software to anyone and for any purpose. Many of the most popular Python libraries are also open source. There are thousands of shared libraries for you to use, and - maybe, when you feel confident enough - to contribute to with your own code. FunctionsFunctions are callable modules of code, some with parameters or arguments (variables you can pass to the function), which performs a task and may return a value. They're a convenient way to package code into discrete blocks, making your overall program more readable, reusable, and saving time.You can also easily share your functions with others, saving them time as well. I snuck one in the section on __Exception handling__ earlier.    Syntax                You structure a function using def, like so:        def function_name(parameters):         code         return response        return is optional, but allows you to return the results of any task performed by the function to the point where the function was called        To test whether an object is a function (i.e. callable), use callable, e.g. callable(function) will return True<jupyter_code># A simple function with no arguments
def say_hello():
    print("Hello, World!")

# Calling it is as simple as this
say_hello()

# And you can test that it's a callable
print(callable(say_hello))<jupyter_output>Hello, World!
True<jupyter_text>An argument can be any variable, such as integers, strings, lists, dictionaries or even other functions. This is where you start realising the importance of leaving comments and explanations in your code because you need to ensure that anyone using a function knows what variables the function expects, and in what order.Functions can also perform calculations and return these to whatever called them.<jupyter_code># A function with two string arguments
def say_hello_to_user(username, greeting):
    # Returns a greeting to a username
    print(F"Hello, {username}! I hope you have a great {greeting}.")

# Call it
say_hello_to_user("Jill", "day")

# Perform a calculation and return it
def sum_two_numbers(x, y):
    # Returns the sum of x + y
    return x + y

sum_two_numbers(5, 10)<jupyter_output>Hello, Jill! I hope you have a great day.<jupyter_text>You can see that swapping `username` and `greeting` in the `say_hello_to_user` function would be confusing, but swapping the numbers in `sum_two_numbers` wouldn't cause a problem.Not only can you call functions from functions, but you can also create variables that are functions, or the result of functions.<jupyter_code>def number_powered(number, exponent):
    # Returns number to the power of exponent
    return number ** exponent

# Jupyter keeps functions available that were called in other cells
# This means `sum_two_numbers` is still available
def sum_and_power(number1, number2, exponent):
    # Returns two numbers summed, and then to an exponent
    summed = sum_two_numbers(number1, number2)
    return number_powered(summed, exponent)

# Call `sum_and_power`
print(sum_and_power(2, 3, 4))<jupyter_output>625<jupyter_text>With careful naming, and plenty of commentary, you can see how you can make your code extremely readable and self-explanatory.A better way of writing comments in functions is called `docstrings`.     Syntax                Docstrings are written as structured text between three sets of inverted commas, e.g. """ This is a docstring """        You can access a function's docstring by calling function.__doc__<jupyter_code>def docstring_example():
    """
    An example function which returns `True`.
    """
    return True

# Printing the docstring
print(docstring_example.__doc__)

# Calling it
print(docstring_example())<jupyter_output>An example function which returns `True`.
    
True<jupyter_text>Classes and ObjectsA complete Python object is an encapsulation of both variables and functions into a single entity. Objects get their variables and functions from `classes`.Classes are where most of the action happens in Python and coding consists, largely, of producing and using classes to perform tasks. A very basic class would look like this:<jupyter_code>class myClass:
    """
    A demonstration class.
    """
    my_variable = "Look, a variable!"
    
    def my_function(self):
        """
        A demonstration class function.
        """
        return "I'm a class function!"

# You call a class by creating a new class object
new_class = myClass()

# You can access class variables or functions with a dotted call, as follows
print(new_class.my_variable)
print(new_class.my_function())

# Access the class docstrings
print(myClass.__doc__)
print(myClass.my_function.__doc__)<jupyter_output>Look, a variable!
I'm a class function!

    A demonstration class.
    

        A demonstration class function.<jupyter_text>Let's unpack the new syntax.    Syntax                You instantiate a class by calling it as class(). If you only called class, without the brackets, you'd gain access to the object itself. This is useful as well, and means you can pass classes around as you would variables.        All variables and functions of a class are reached via the dotted call, .function() or .variable. You can even add new functions and variables to a class you created. Remember, though, these will not exist in new classes you create since you haven't changed the underlying code.        Functions within a class require a base argument that, by convention, is called self. There's a complex explanation as to why self is needed, but - briefly - think of it as the instance of the object itself. So, inside the class, self.function is the way the class calls its component functions.        You can also access the docstrings as you would before.<jupyter_code># Add a new variable to a class instance
new_class1 = myClass()
new_class1.my_variable2 = "Hi, Bob!"
print(new_class1.my_variable2, new_class1.my_variable)

# But, trying to access my_variable2 in new_class causes an error
print(new_class.my_variable2)<jupyter_output>Hi, Bob! Look, a variable!<jupyter_text>Classes can initialise themselves with a set of available variables. This makes the `self` referencing more explicit, and also permits you to pass arguments to your class to set the initial values.    Syntax                Initialise a class with the special function def __init__(self)        Pass arguments to your functions with __init__(self, arguments)        We can also differentiate between arguments, and keyword arguments:                    arguments: these are passed in the usual way, as a single term, e.g. my_function(argument).            keyword arguments: these are passed the way you would think of a dictionary, e.g. my_function(keyword_argument = value). This is also a way to initialise an argument with a default. If you leave out the argument when it has a default it will apply without the function failing.            Functions often need to have numerous arguments and keyword arguments passed to them, and this can get messy. You can also think of a list of arguments like a list, and a list of keyword arguments like a dictionary. A tidier way to deal with this is to reference your arguments and keyword arguments like this, my_function(*args, **kwargs) where *args will be available to the function as an ordered list, and **kwargs as a dictionary.<jupyter_code># A demonstration of all these new concepts

class demoClass:
    """
    A demonstration class with an __init__ function, and a function that takes args and kwargs.
    """
    
    def __init__(self, argument = None):
        """
        A function that is called automatically when the demoClass is initialised.
        """
        self.demo_variable = "Hello, World!"
        self.initial_variable = argument
        
    def demo_class(self, *args, **kwargs):
        """
        A demo class that loops through any args and kwargs provided and prints them.
        """
        for i, a in enumerate(args):
            print("Arg {}: {}".format(i+1, a))
        for k, v in kwargs.items():
            print("{} - {}".format(k, v))
        if kwargs.get(self.initial_variable):
            print(self.demo_variable)
        return True

demo1 = demoClass()
demo2 = demoClass("Bob")

# What was initialised in each demo object?
print(demo1.demo_variable, demo1.initial_variable)
print(demo2.demo_variable, demo2.initial_variable)

# A demo of passing arguments and keyword arguments
args = ["Alice", "Bob", "Carol", "Dave"]
kwargs = {"Alice": "Engineer",
          "Bob": "Consultant",
          "Carol": "Lawyer",
          "Dave": "Doctor"
         }
demo2.demo_class(*args, **kwargs)<jupyter_output>Hello, World! None
Hello, World! Bob
Arg 1: Alice
Arg 2: Bob
Arg 3: Carol
Arg 4: Dave
Alice - Engineer
Bob - Consultant
Carol - Lawyer
Dave - Doctor
Hello, World!
{'fraction_non_alphanumeric': 0.07585115998794818, 'fraction_numerical': 0.010808978608014462, 'mean_word_length': 4.146332008915593, 'pattern_counts': {'":': 11, '<': 135, '<?xml version=': 0, '>': 142, 'https://': 15, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '354948', 'license': 'MIT', 'n_tokens_mistral': 14845, 'n_tokens_neox': 13982, 'n_words': 7497, 'path': 'Module 2 - Lesson 1 - Introduction to Jupyter Notebook and programming in Python.ipynb', 'repo_name': 'whythawk/data-wrangling-and-validation', 'size': '53206'}