<jupyter_start><jupyter_text>Intermediate PythonNow we have unpacked many of the basic features of Python, we will go into some of the more interesting features of Python at the intermediatory stage. SetsSets are part of *Set Theory* whereby it lies as a gathering together of distinct/unique objects. Sets are traditionally **unordered** and contain immutable, unique objects of any data type.If we want to create a set, we can call the built-in set function with a sequence or another iterable object:In the following example, a string is singularized into its characters to build the resulting set x:<jupyter_code>x = set("A Python Tutorial")
x<jupyter_output><empty_output><jupyter_text>Here it has broken down our string into characters and extracted the unique characters in the string. We can pass a list to the built-in set function, as we can see in the following:<jupyter_code>x = set(["Perl", "Python", "Java"])
x<jupyter_output><empty_output><jupyter_text>Now, we want to show what happens, if we pass a tuple with reappearing elements to the set function - in our example the city "Paris":<jupyter_code>cities = set(("Paris", "Lyon", "London","Berlin","Paris","Birmingham"))
cities<jupyter_output><empty_output><jupyter_text>This has of course, removed any duplicates.  Set OperationsAlthough Sets contain objects that are *immutable*, they themselves are *mutable*:<jupyter_code>cities.add("Stuttgart")
cities
second_cities = set(("Hong Kong", "Lyon", "Tokyo", "Paris", "Dubai"))<jupyter_output><empty_output><jupyter_text>We can return the *difference* of two or more sets as a new set:<jupyter_code>cities.difference(second_cities)
# can be abbrievated with - subtract
cities - second_cities<jupyter_output><empty_output><jupyter_text>Or return the *union* between two sets, i.e all the elements that are in **either** set:<jupyter_code>cities.union(second_cities)
# can be abbrievated with pipe |
cities | second_cities<jupyter_output><empty_output><jupyter_text>Or return the *intersection* between two sets, i.e all the elements that are in **both** sets:<jupyter_code>cities.intersection(second_cities)
# can be abbrieviated with &
cities & second_cities<jupyter_output><empty_output><jupyter_text>List ComprehensionsList comprehensions provide a concise way to create lists. It consists of brackets containing an expression followed by a for clause, thenzero or more for or if clauses. The expressions can be anything, meaning you canput in all kinds of objects in lists.The result will be a new list resulting from evaluating the expression in thecontext of the for and if clauses which follow it. The list comprehension always returns a result list. If you used to do it like this:<jupyter_code>old_list = ["apples", "oranges", "grapes"]

new_list = []
for i in old_list:
    new_list.append((i))<jupyter_output><empty_output><jupyter_text>You can obtain the same thing using list comprehension:<jupyter_code>new_list = [i for i in old_list]
new_list<jupyter_output><empty_output><jupyter_text>The list comprehension can include conditionals at the end, such as:    [expression for item in list if conditional]or    for item in list:        if conditional:            expression<jupyter_code>new_list = [i for i in old_list if i == "apples"]
new_list
def square(x):
    return x**2

squares = [square(x) for x in range(10)]
squares<jupyter_output><empty_output><jupyter_text>Or alternatively a list of words:<jupyter_code>listOfWords = ["this","is","a","list","of","words"]
items = [word[0] for word in listOfWords ]
items<jupyter_output><empty_output><jupyter_text>Reading and Writing FilesWhen you’re working with Python, you don’t need to import a library in order to read and write files. It’s handled natively in the language, albeit in a unique manner.The first thing you’ll need to do is use Python’s built-in open function to get a file object.The open function opens a file. It’s simple.When you use the open function, it returns something called a file object. File objects contain methods and attributes that can be used to collect information about the file you opened. They can also be used to manipulate said file.For example, the mode attribute of a file object tells you which mode a file was opened in. And the name attribute tells you the name of the file that the file object has opened.You must understand that a _file_ and _file object_ are two wholly separate – yet related – things.  File TypesWhat you may know as a file is slightly different in Python. In Windows, for example, a file can be any item manipulated, edited or created by the user/OS. That means files can be images, text documents, executables, and much more. Most files are organized by keeping them in individual folders. In Python, a file is categorized as either **text** or **binary**, and the difference between the two file types is important. Text files are structured as a sequence of lines, where each line includes a sequence of characters. This is what you know as code or syntax. Each line is terminated with a special character, called the EOL or End of Line character. There are several types, but the most common is the comma {,} or newline character. It ends the current line and tells the interpreter a new one has begun. A backslash character can also be used, and it tells the interpreter that the next character – following the slash – should be treated as a new line. This character is useful when you don’t want to start a new line in the text itself but in the code.  Open() functionThis works as follows:    file_object = open(filename, mode)    Where the second argument you see, **mode**, informs the interpreter and developer which way the file will be used.The available modes are:- **r**: Read mode which is used when the file is only being read.- **w**: Write mode which is used to edit and write new information into the file- **a**: Appending mode, which is used to add new data to the end of the file.- **r+**: Special read and write mode, which is used to handle both actiosn when working with a fileEvery `open()` should be matched with a `close()`, this allows Python to close the file handle and potentially allow other processes to access it.Let's look at some examples:<jupyter_code>F = open("writing.txt", "w")
F<jupyter_output><empty_output><jupyter_text>This snippet opens the file named “writing” in writing mode so that we can make changes to it. The current information stored within the file is also displayed – or printed – for us to view. Once this has been done, you can move on to call the objects functions. The two most common functions are read and write. Creating a text fileTo get more familiar with text files in Python, let’s create our own and do some additional exercises. Using a simple text editor, let’s create a file. You can name it anything you like, and it’s better to use something you’ll identify with. For the purpose of this tutorial, however, we are going to call it “testfile.txt”. Just create the file and leave it blank. To manipulate the file, write the following in your Python environment (you can copy and paste if you’d like):<jupyter_code>F.write("Hello World")
F.write("This is a new line")
F.write("and this is a second line")
F.write("This is getting repetitive...")

F.close()
!cat writing.txt<jupyter_output><empty_output><jupyter_text>Above we are using **iPython Magicks** to access the command line and print out writing.txt into the console. Notice that we are not actually creating new lines in the file, merely the `write()` function glues whatever you write next to the previous write statement, and we have to manually add `\n` into the string if we want new lines to appear. Read()Let's try reading in a file:<jupyter_code>F = open("test_file.txt", "r")
print(F.read())
F.close()<jupyter_output><empty_output><jupyter_text>This method essentially spits out everything within the file, which is great if we have a small file, but often we want to receive things *line by line*:<jupyter_code>F = open("test_file.txt", "r") 
print(F.readline())
F.close()<jupyter_output><empty_output><jupyter_text>And this returns *all* lines as a Python list.<jupyter_code>F = open("test_file.txt", "r") 
print(F.readlines())
F.close()<jupyter_output><empty_output><jupyter_text>Looping over a file objectWhen you want to read – or return – all the lines from a file in a more memory efficient, and fast manner, you can use the loop over method. The advantage to using this method is that the related code is both simple and easy to read.<jupyter_code>F = open("test_file.txt", "r") 
for line in F: 
    print(line)
F.close()<jupyter_output><empty_output><jupyter_text>With statementYou can also work with file objects using the with statement. It is designed to provide much cleaner syntax and exceptions handling when you are working with code. That explains why it’s good practice to use the with statement where applicable. One bonus of using this method is that any files opened will be closed automatically after you are done. This leaves less to worry about during cleanup. To use the with statement to open a file:<jupyter_code>with open("test_file.txt","r") as F:
    print(F.readlines())<jupyter_output><empty_output><jupyter_text>Splitting Lines in a Text FileAs a final example, let’s explore a unique function that allows you to split the lines taken from a text file. What this is designed to do, is split the string contained in variable data whenever the interpreter encounters a space character.But just because we are going to use it to split lines after a space character, doesn’t mean that’s the only way. You can actually split your text using any character you wish - such as a colon, for instance.The code to do this (also using a with statement) is:<jupyter_code>with open("test_file.txt","r") as f:
    data = f.readlines()

for line in data:
    print(line.split())<jupyter_output><empty_output><jupyter_text>If you wanted to use another letter instead of a space to split your text, you would simply change line.split() to `line.split("l")`. This is known as a **delimiter**.The output for this will be:<jupyter_code>for line in data:
    print(line.split("l"))<jupyter_output><empty_output><jupyter_text>Exception Handling Exceptions handling in Python is very similar to Java. The code, which harbours the risk of an exception, is embedded in a try block. But whereas in Java exceptions are caught by catch clauses, we have statements introduced by an "except" keyword in Python. It's possible to create "custom-made" exceptions: With the raise statement it's possible to force a specified exception to occur.Let's look at a simple example. Assuming we want to ask the user to enter an integer number. If we use a input(), the input will be a string, which we have to cast into an integer. If the input has not been a valid integer, we will generate (raise) a ValueError. We show this in the following interactive session:<jupyter_code>n = int(input("Please enter a number: "))<jupyter_output><empty_output><jupyter_text>With the aid of exception handling, we can write robust code for reading an integer from input:<jupyter_code>while True:
    try:
        n = input("Please enter an integer: ")
        n = int(n)
        break
    except ValueError:
        print("No valid integer! Please try again..")
print("Great, successfully received integer!")<jupyter_output><empty_output><jupyter_text>Multiple Except Clauses A try statement may have more than one except clause for different exceptions. But at most one except clause will be executed.Our next example shows a try clause, in which we open a file for reading, read a line from this file and convert this line into an integer. There are at least two possible exceptions:- an IOError- ValueErrorJust in case we have an additional unnamed except clause for an unexpected error:<jupyter_code>import sys

try:
    f = open('integers.txt')
    s = f.readline()
    i = int(s.strip())
except IOError as e:
    errno, strerror = e.args
    print("I/O error({0}): {1}".format(errno,strerror))
    # e can be printed directly without using .args:
    # print(e)
except ValueError:
    print("No valid integer in line.")
except:
    print("Unexpected error:", sys.exc_info()[0])
    raise<jupyter_output><empty_output><jupyter_text>LambdaDespite much protesting from the founder of the Python programming language, defeat was conceded and these functions including `lambda` now exist in the base Python language.The lambda operator or lambda function is a way to create small anonymous functions, i.e. functions without a name. These functions are throw-away functions, i.e. they are just needed where they have been created. Lambda functions are mainly used in combination with the functions `filter()`, `map()` and `reduce()`.<jupyter_code>sum = lambda x, y: x + y
sum(3,4)<jupyter_output><empty_output><jupyter_text>This acts as shorthand for creating a properly-defined function. Lambda becomes more useful in conjunction with `pandas`, which is a later module down the line. Arbitrary Parameter LengthsThere are many situations in programming, in which the exact number of necessary parameters cannot be determined a-priori. An arbitrary parameter number can be accomplished in Python with so-called tuple references. An asterisk `*` is used in front of the last parameter name to denote it as a tuple reference. This asterisk shouldn't be mistaken with the C syntax, where this notation is connected with pointers.Example:<jupyter_code>def arithmetic_mean(first, *values):
    """
    This function calculates the arithmetic mean of a non-empty
    arbitrary length of numerical values.
    """
    return (first + sum(values)) / (1 + len(values))
arithmetic_mean(2,3,4,5,6)
arithmetic_mean(30,50,10)
arithmetic_mean(75.4340, 504.323923, 1.3493434, 0.548338)<jupyter_output><empty_output><jupyter_text>Alternatively, you can unpack a list or a tuple into a *multiple-parameter* function:<jupyter_code>def f(x, y, z):
    return x*y*z

t = (1.0, 2.0, 3.0)
f(*t)<jupyter_output><empty_output><jupyter_text>Arbitrary Number of 'Keyword' ParametersIn the previous chapter we demonstrated how to pass an arbitrary number of positional parameters to a function. It is also possible to pass an arbitrary number of keyword parameters to a function. To this purpose, we have to use the double asterisk `**`:<jupyter_code>def f(**kwargs):
    print(kwargs)
f()
f(de="German", en="English", fr="French")<jupyter_output><empty_output><jupyter_text>You can think of this as the function essentially creating a dictionary out of the *variable-number* parameters you have passed. A more practical use case: passing a large number of parameters to a function programmatically:<jupyter_code>def f(a, b, x, y):
    return a+b-x*y

d = {"a": 1.3, "b": 0.5, "x": 3.4, "y": 0.6}
f(**d)<jupyter_output><empty_output><jupyter_text>DecoratorsDecorators belong most probably to the most beautiful and most powerful design possibilities in Python, but at the same time the concept is considered by many as complicated to get into. To be precise, the usage of decorates is very easy, but writing decorators can be complicated, especially if you are not experienced with decorators and some functional programming concepts. Even though it is the same underlying concept, we have two different kinds of decorators in Python:- Function decorators- Class decorators A decorator in Python is any callable Python object that is used to modify a function or a class. A reference to a function "func" or a class "C" is passed to a decorator and the decorator returns a modified function or class. The modified functions or classes usually contain calls to the original function "func" or class "C".  A function having multiple namesSee the example below:<jupyter_code>def succ(x):
    return x+1

successor = succ
successor(10)
succ(10)<jupyter_output><empty_output><jupyter_text>Functions inside FunctionsThe concept of having or defining functions inside of a function is completely new to C or C++ programmers:<jupyter_code>def f():
    
    def g():
        print("Local g function here!")
        print("Leaving g now!")
    
    print("Local f function here!")
    print("Calling g now!:")
    g()

f()<jupyter_output><empty_output><jupyter_text>Here is another example with the proper `return` statements:<jupyter_code>def temperature(t):
    def celsius2fahrenheit(x):
        return 9 * x / 5 + 32

    result = "It's " + str(celsius2fahrenheit(t)) + " degrees!" 
    return result

print(temperature(20))<jupyter_output><empty_output><jupyter_text>Functions as ParametersIf you solely look at the previous examples, this doesn't seem to be very usefull. It gets useful in combination with two further powerful possibilities of Python functions. Due to the fact that every parameter of a function is a reference to an object and functions are objects as well, we can pass functions - or better "references to functions" - as parameters to a function. We will demonstrate this in the next simple example:<jupyter_code>def g():
    print("Hi, it's me 'g'")
    print("Thanks for calling me")
    
def f(func):
    print("Hi, it's me 'f'")
    print("I will call 'func' now")
    func()
          
f(g)<jupyter_output><empty_output><jupyter_text>This extends to calling external library functions to create generic functions that could possibly apply a *wide range* of operations:<jupyter_code>import math

def foo(func, array=[1,2,3]):
    print("The function " + func.__name__ + " was passed to foo")
    res = 0
    for x in array:
        res += func(x)
    return res

print(foo(math.sin))
print(foo(math.cos))<jupyter_output><empty_output><jupyter_text>Syntax of decoratorsWe can write our decorator in terms of a statement positioned just before the function we wish to decorate:<jupyter_code>def our_decorator(func):
    def function_wrapper(x):
        print("Before calling " + func.__name__)
        func(x)
        print("After calling " + func.__name__)
    return function_wrapper

@our_decorator
def foo(x):
    print("Foo has been called with " + str(x))

foo("Hi")<jupyter_output><empty_output><jupyter_text>Potential use cases include *checking arguments* with a decorator, see below:<jupyter_code>def argument_test_natural_number(f):
    def helper(x):
        if type(x) == int and x > 0:
            return f(x)
        else:
            raise Exception("Argument is not an integer")
    return helper

@argument_test_natural_number
def factorial(n):
    if n == 1:
        return 1
    else:
        return n * factorial(n-1)
    
for i in range(1,10):
	print(i, factorial(i))

print(factorial(-1))<jupyter_output><empty_output><jupyter_text>*Use Case 2*: Counting Function calls with decorators:<jupyter_code>def call_counter(func):
    def helper(x):
        helper.calls += 1
        return func(x)
    helper.calls = 0

    return helper

@call_counter
def succ(x):
    return x + 1

print(succ.calls)
for i in range(10):
    succ(i)
    
print(succ.calls)<jupyter_output><empty_output><jupyter_text>There is a large amount of room to expand on this topic, but for further details, we recommend you look at https://www.python-course.eu/, which acts as the inspiration for much of this work.  Tasks Task 1Read in all of the lines from The Wolf and the Crane, found in `wolfcran.txt`. Extract all of the unique words from the book using `set`. Print out the number of unique words found.<jupyter_code># your codes here<jupyter_output><empty_output><jupyter_text>Task 2Create a 6 $\times$ 6 matrix of incrementing values, using a **nested list comprehension** technique, but only recording values that are **even**.<jupyter_code># your codes here<jupyter_output><empty_output><jupyter_text>Task 3Write a function `log_sum_product(first, *vals)` that accepts a *arbitrary* number of numbers.Calculating the **product** of a set of proabilities can lead to numerical instability, particularly for values less than 1 (in a probabilistic context). To get around this, we can convert to `log` and `sum` all of the values in logspace, then convert back using `exp`.$$P(\vec{x})=\exp \left(\sum_{i=1}^n \log(\vec{x}_i)\right)$$When summing the values together, we recommend you use `math.fsum` rather than using the default Python `sum` function.When testing your function, try using around 4 or 5 values that are $0 \le x \le 1$.<jupyter_code># your codes here<jupyter_output><empty_output><jupyter_text>Task 4Below is a method from numerical integration for approximating a definite integral:$$\int_a^b f(x)dx \approx \sum_{k=1}^N \frac{f(x_{k-1})+f(x_k)}{2}\Delta x_k $$Write a function `f(x)` which returns:$$f(x)=\cos x$$In addition, write a decorator function that ensures that `f(x)` can only be called where $b-a =\frac{\pi}{2}$. In addition, ensure that `N` is an int. This should ensure that the integral:$$\int_0^{\pi/2} \cos(x) dx = |1|$$You will need to modify the `trapz` function provided to do this. Ensure that you include exception handling into your decorator. Then try the following function calls:    trapz(f, 0, math.pi/2, 10000)    trapz(f, 1, math.pi, 10000)    trapz(f, 0, math.pi, 10000)    trapz(f, math.pi/2, math.pi, 10000)<jupyter_code># your codes here
def trapz(f, a, b, N):
    """
    Calculates composite trapezoidal integration.
    
    Where f is a function passed, which accepts one parameter (x)
        a is the lower-bound
        b is the upper-bound
        N is the size
    
    Returns the Integral.
    """
    h = (b-a)/float(N)
    sum_y = 0
    x = a
    for i in range(N):
        x += h
        sum_y += f(x)
    sum_y += .5 * (f(a) + f(b))
    return sum_y*h
# your function calls here<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08264690994431917, 'fraction_numerical': 0.006534443882011872, 'mean_word_length': 4.720452750723875, 'pattern_counts': {'":': 5, '<': 161, '<?xml version=': 0, '>': 162, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '26052', 'license': 'MIT', 'n_tokens_mistral': 6310, 'n_tokens_neox': 6099, 'n_words': 2857, 'path': '01-Principles/02 Python Intermediate.ipynb', 'repo_name': 'terminal-w/PythonCourse', 'size': '21731'}