<jupyter_start><jupyter_text>Functional Paradigm  PART I : Functions, functional programming. There are several well known paradigms of programming* Imperative - How to do* Declarative - What has to be done* Functional - Subset of Declarative, Sequence of transformations    **Imperative language** is a set of instructions (**statements**) on how to change a **state**.**State** is **values** of all **variables** in all contexts in a giving moment. We can create, mutate and delete variables. We usually use statements, flow control conditional constructs (`if/then/else, try/except`), loops (`for/while`) changing context (`def, class, import`). State is defined by the values of the variables in the various namespaces. In **functional language** we use a notion of evaluating functions rather than changing states. Each function evaluation creates a new object or objects from existing objects. So, we can consider program execution as a sequence of function executions or sequence of mappings between data objects. In order to make complex transformations, functions can be composed from lower-level functions that are easy to understand.  SECTION 0: Function syntax and attributes For the moment you have to know how to define function`def f(a, b, *args, c=10, **kwargs):    """Some function"""    x = 100    return a * x`Official python tutorial: https://docs.python.org/3/tutorial/controlflow.html defining-function**This lecture is mostly for advanced approach to functional programming style in python**  SECTION 1: Functions as objects Everything is an **object** in Python, so **function** does. This is a usually called _First-class object model_. This basically means, that unlike in **C** or **Java**, function objects not only may be invoked, but may be * assigned to other names* passed to other functions* returned from one function to another* embedded in data structures* ... and moreas if they were simple numbers or stringsStill, functions belong to the same general category as other objects.  Assigned to other names<jupyter_code>def f(a):
    print(a)
    
g = f

g('Hello world')<jupyter_output><empty_output><jupyter_text>Passed to other functions<jupyter_code>x = [3, 1, 2, 7, 4]


def key(a):
    return a % 2 

z = sorted(x, key=key)
print(z)<jupyter_output><empty_output><jupyter_text>Returned from other functions<jupyter_code>def add(a, b):
    return a + b
def mul(a, b):
    return a * b
def power (a, b):
    return a ** b
        
# determine what function produces highest value
    
def highest(a, b, functions):
    results = [
        (f, f(a, b))
        for f in functions
        ]
       
      
    sorted_results = sorted(results, key=lambda a: a[1], reverse = True)
    return sorted_results[0][0]

print(highest(a=2, b=4, functions=[add, mul, power]))<jupyter_output><empty_output><jupyter_text>Functions can be assigned names or can be anonymous:           >>> b = 3     >>> f2 = lambda x: x + 10*b     Formally: `lambda argument1, argument2,... argumentN : expression using arguments`**Use cases:** * List sorting `sorted(results, key=lambda a: a[1], reverse = True)`* Function as a parameter`print(highest(a=2, b=4, functions=[add, mul, power]))`       Notes* lambda is an expression, not a statement* lambda ’s body is a single expression, not a block of statements* scope lookup rules are the same as for the **def**   BenefitsLambda is an expression which returns functional objectLambda provides code proximityIs good as one-time jobCan be inlined into fuction calls  Final NotesBecause we can manipulate functions as objects, define functions inside functions and return functions, we can do a lot of interesting things like creating functions on-the-fly that enhance capabilities of other functions or classes, combining and splitting functions, memorizing function results to reuse to not to call function twice, etc.   SECTION 2 - Scopes, bounded and free variables.  Pure functionsFunction that has no side-effects* Output is dependent on the input only* Determenistic - always produce same results with same parameters not produce side effects * Objects are created or deleted, never mutated * May return another function   Bound and free variables<jupyter_code>b = 10  # global variable

def f(a):   # a is function parameter
    c = 3  # local variable
    return a + b + c

print(f(100)) 

b = 20

print(f(100))<jupyter_output><empty_output><jupyter_text>It is not a pure function, because of different results depending on context.Variable `a` is function parameter, variable `c` is `local variable` to function `scope`, but `b` is complicated - global variable defined outside function.<jupyter_code>def g(b):
    def f(a):
        c = 3
        return a + b + c
    return f

print(g(10)(100))  # here we invoke g(10) that returns FUNCTION and invoke the result with parameter a=100

print(g(20)(100))  # here we invoke g(20) that returns FUNCTION and invoke the result with parameter a=100<jupyter_output><empty_output><jupyter_text>This is a method called **Curriyng** which is transformation of function of many arguments to a sequence of functions taking only one argument.Like x = f(a, b, c):         g = l(a)        k = g(b)        d = k(c)Otherwise x = j(a)(b)(c)**NB** Currying is related to,  but not equivalent of partial function application  Partial<jupyter_code>def filter_(strng, black_list, white_list):
    
    if strng in white_list: return True
    
    if strng in black_list: return False<jupyter_output><empty_output><jupyter_text>The partial() is used for partial function application which “freezes” some portion of a function’s arguments and/or keywords resulting in a new object with a simplified signature.<jupyter_code>from functools import partial

filter_n = partial(filter_, black_list=bl, white_list=wl)<jupyter_output><empty_output><jupyter_text>You can now apply filter_n to any dataset to check if an item belongs to either white or black list Both functions (g and f) in this case are `pure`.Note the difference between variable `b` in the above examples. Usually, when defining function you wont be able to define function like this    def f(a):        c = 3        return a + b + cIn examples above python interpreter when defining function looks if variable was defined in some scope in levels above, traversing call stack back to the top. In first case, variable `b` inside function definition was  `bound` to global variable `b` (changing it leads to change a result of function output).Note that its value could be determined at once when function was defined.In second case, variable `b` inside function definition was `free variable` and you wont be able to determine its value until we invoked function `g`. Lets consider following tricky example<jupyter_code>b = 10

def f(a):
    c = 3
    return a + b + c
       
def g(a, b):
    return f(a)

g(100, 1)<jupyter_output><empty_output><jupyter_text>What the result would be?  In python whether variable in function is free or bound is determined when function is defined. Back to pure functions. True functional programming langages deals primary with pure functions (haskell has ONLY pure functions). Python is hybrid language that allows mixing approaches with procedural, imperative, declarative, OOP and functional style. This allows to write very readable code.Now, what is so important about pure functions?* testable. feed a parameter and expect a return result* values could be cached, precomputed, etc.* they can be lazy  (lazy function - function that is invoked only when needed)* since they dont alter the context they are very easy to parallelize (very important) In python to exploit benefits of functional programming style functions do not have to be pure, but you have to be careful with side effects, otherwise you may introduce `race conditions` or  `Heisenbugs` (bugs that appear only in rare cases), that are VERY hard to find and eliminate.  Functional programming drawbacks and The Python Way There are some disadvantages of pure functional approach, which are most clear in languages like Haskell, F etc:* Hard to code and understand* Functions limited to pure ones* I/O is hacked on top of language core (in haskell it's a mess)* Conditional execution is a mess (what happens if you need different business logic based on some value in db?)* Common patterns are overcomplicatedFunctional languages are good for calculations (like finding factorial of a number), data processing and so on, but poor in business cases.Python hybrid approach allows most of benefits of pure functional programming without significant drawbacks.  Decorators - example of hybrid approach If you have a function whose logic can be decomposed into two pieces, so that one is essential to your task and the other is auxiliary, so you can separate them. This brings you an ability generalize auxiliary task, and make logic relevant to your main task clear.Decorator is a function manager, which adds functionality to a function and which manages both function calls and function objects.You can encounter decorators: * Python built-in decorators (static and class method declaration, property creation, etc.). * Python toolkits (e.g. managing database or user-interface logic)* In most of the libraries you will be using**Note** We are speaking about function decorators. There is class decorators also.<jupyter_code>def get_user(username):
    conn = connect_to_db()
    print('connected to db!')
    try:
        user = conn.get_user(username)
    except UserNotExists:
        logging.error('Database error: user not exists')
        user = None
    conn.disconnect()
    return user

def update_user(username, data_to_update):
    conn = connect_to_db()
    print('connected to db!')
    try:
        user = conn.get_user(username)
    except UserNotExists:
        logging.error('Database error: user not exists')
        user = None
    if user:
        user.update(data_to_update)
    conn.disconnect()

def delete_user(username):
    conn = connect_to_db()
    print('connected to db!')
    try:
        user = conn.get_user(username)
    except UserNotExists:
        logging.error('Database error: user not exists')
        user = None
    if user:
        user.delete(user)
    conn.disconnect()<jupyter_output><empty_output><jupyter_text>To shorten this, we could write the following:<jupyter_code>def connect_and_get_user(username):
    conn = connect_to_db()
    print('connected to db!')
    try:
        user = conn.get_user(username)
    except UserNotExists:
        logging.error('Database error: user not exists')
        user = None
    return user, conn<jupyter_output><empty_output><jupyter_text>We'll still need to check if user was found and close connection:<jupyter_code>def get_user(username):
    user, conn = connect_and_get_user(username)
    conn.disconnect()
    return user
    
def delete_user(username):
    user, conn = connect_and_get_user(username)
    if user:
        user.delete()
    conn.disconnect()<jupyter_output><empty_output><jupyter_text>Functional approach<jupyter_code>def get_user(user):
    return user

def delete_user(user):
    user.delete()

def update_user(user, data_to_update):
    user.update(data_to_update)
def wrapper(f):
    def wrapped(username, *args, **kwargs):
        conn = connect_to_db()
        print('connected to db!')
        try:
            user = conn.get_user(username)
        except UserNotExists:
            logging.error('Database error: user not exists')
            user = None
            if user:
                user.update(data_to_update)
            conn.disconnect()
        return result
    return wrapped<jupyter_output><empty_output><jupyter_text>Substitute original functions with their wrapped versions<jupyter_code>delete_user = wrapper(delete_user) 
get_user = wrapper(get_user)
update_user = wrapper(update_user)<jupyter_output><empty_output><jupyter_text>Or in short:<jupyter_code>@wrapper
def change_password(user, password):
    user.change_password(password)<jupyter_output><empty_output><jupyter_text>Decorators can also be parameterized:    It will look like this: @save(filename='test.csv') to save result of a function call to file<jupyter_code>def save(filename):
    # parametrized decorator that saves result of function to file
    def wrapper(f):
        def substitute(a):
            result = f(a)
            with open(filename, 'w+') as output:
                output.write(str(result))
            return result
        return substitute
    return wrapper<jupyter_output><empty_output><jupyter_text>This can be used as follows<jupyter_code>@save(filename='test.csv)
def f(a):
    print(a)
    return a 

#is equivalent to 
def f(a):
    print(a)
    return a

f = save(filename='test.csv')(f)
#or, if we expand it with intermediate result
save_to_test_csv = save(filename='test.csv')  # made a simple decorator
f = save_to_test_csv(f)
    
`save` when invoked makes a unparametrized decorator and later we invoke it with argument (f).
this is not of course very readable, and theoretically you could replace two-layer function
of parametrized decorator with this:

def save(filename, f):
    def substitute(a):
        result = f(a)
        with open(filename, 'w+') as output:
            output.write(str(result))
        return result
            
    return substitute<jupyter_output><empty_output><jupyter_text>Decorator shortcut (@-syntax) allows only one parameter (function), and for that reason we **CURRY** it.   Creative decorator application* code deduplication* `registration` of functions (as routes in flask, for example)* cache and value memorisation (example, lru_cache decorator)* rpc (when function is not actually executed but is sent to some other place to be run, example - celery)  Functiona Programming Tools In order to support functional paradigm, Python includes a set of built-ins used for functionalprogramming—tools that apply functions to sequences and other iterables.* **Map** - call functions on an iterable’s items* **Filter** - filter out items based on a test function* **Reduce** -  apply functions to pairs of items and returnin results  Map<jupyter_code>c = [1, 2, 3, 4, 5]

new_c = []

for i in c:
    new_c.append(i + 1)

new_c
def increase(x): return x + 1
list(map(increase, c))<jupyter_output><empty_output><jupyter_text>List comprehension can be used instead of a Map (+/-)  Filter Application of test function to select iterables' items<jupyter_code>list(range(-10, 10))
list(filter((lambda x: x % 2 == 0), list(range(10))))<jupyter_output><empty_output><jupyter_text>Can be represented as for loop<jupyter_code>l = []
for i in range(10):
    if i % 2 == 0: l.append(i)

l
def test(x):
    if x % 2 == 0: return True
list(filter(test, list(range(10))))<jupyter_output><empty_output><jupyter_text>Reduce<jupyter_code>from functools import reduce
reduce((lambda x, y: x + y), [1, 2, 3, 4])
def my_reduce(func, seq):
    res = seq[0]
    for chunk in seq[1:]:
        res = func(res, chunk)
    return res
my_reduce((lambda x, y: x + y), [1, 2, 3, 4])<jupyter_output><empty_output><jupyter_text>Recursion<jupyter_code>def factor(x):
    if not x:
        return 1
    else: return x*factor(x-1)
l = []
def factor(x):
    l.append(x)
    print(l)
    if not x:
        return 1
    else: return x*factor(x-1)
factor(30)
reduce((lambda x, y: x*y), reversed(range(1, 30)))<jupyter_output><empty_output><jupyter_text>Why it is better to avoid recursion Although the recursion is very usefull to walk through a data structure with arbitrary shape, the recursion is not recommended way to solve problems in Python as it lacks compile time optimizations like tail recursion, cycle detections and so on. Obviously, due to absence of compile time. It is recommended to use techniques like queue  stacks, and others.  Comprehensions and Generators Comprehensions were borrowed from Haskell - pure functional language. **Map** and **Filter** were introduced earlier. The difference is that comprehensions apply an arbitrary expression to items in an iterable, rather than applying a function. Comprehensions are not limited to lists now. Sets, Dictionaries and value generator expressions.Formally one can write: `[ expression for target in iterable ]`You can surely write many nested loops, but don't get it far. Remember, KISS!If you need to translate code into statements, than it should be statements.But, remember, that **for** is twice as slow as **map** , and **map** is slower than comprehension, due to **map** and **comprehensions** are implemented in **C**, rahter than stepping Python Virtual Machine. This is changing though as Python version increases, so check it with _timeit_ module It worth mentioning that both **Map** and **Filter** can be expressed as list comprehension:**Map**: `[increase(x) for x in [1, 2, 3, 4,5]]`**Filter**: `[x for x in range(10) if x % 2 == 0]`<jupyter_code>[increase(x) for x in [1, 2, 3, 4, 5]]
[x for x in range(10) if x % 2 == 0]<jupyter_output><empty_output><jupyter_text>With list comprehensions, you can combine map and filter in single iteration<jupyter_code>[x**2 for x in range(10) if x % 2 ==0]<jupyter_output><empty_output><jupyter_text>Generators  Generator Functions Unlike normal functions that return a value and exit, generator functions automaticallysuspend and resume their execution and state around the point of value generation. They preserve state, scope (local). The syntax difference is in using **yield** instead of **return** operator.<jupyter_code>def gen_list(N):
    for i in range(N):
        yield i
k = gen_list(5)
next(k)
list(gen_list(5))<jupyter_output><empty_output><jupyter_text>Generator functions are usefull for reducing memory consumption and increasing performance  Generator Expressions<jupyter_code>[x**2 for x in range(10)]
(x**2 for x in range(10))
k = (x**2 for x in range(10))
next(k)
list(k)
def intergral(f, low, upper):<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08320810374575612, 'fraction_numerical': 0.008070351199421162, 'mean_word_length': 4.322274881516588, 'pattern_counts': {'":': 0, '<': 101, '<?xml version=': 0, '>': 107, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '250036', 'license': 'MIT', 'n_tokens_mistral': 5131, 'n_tokens_neox': 4918, 'n_words': 2180, 'path': 'lecture_03_functional_programming/materials/FP.ipynb', 'repo_name': 'revire/epam_python_autumn_2020', 'size': '17967'}