<jupyter_start><jupyter_text>Functional Programming  What, Why and HowWe will try to explain the first principles of functional programming by trying as much as possible to avoid big FP words.We will use python for this exercise, because it's easy to read; and it provides all the functional features required for understanding concepts described in this document.  A simple problem Count the number of words in a large fileYou have a file which may have millions of lines. we need to count the number of words on the file.  A simple solution<jupyter_code>def count_w(filename):
    count = 0
    offset = 0
    file = open(filenames)
    for line in file:
        for w in line.split():
            count += 1
    return count
# But This Doesnt Scale!<jupyter_output><empty_output><jupyter_text>When things don't scale,![title](img/parallel.jpg) Lets try the same example with Threads<jupyter_code>count = 0

def count_w_chunk(file_chunk):
    global count
    count = 0
    offset = 0
    file = open(filenames)
    for line in file:
        for w in line.split():
            count += 1

#actual driver code
def count_w(file):
    threads = [] # empty thread list
    chunks = file.make_chunks()
    # magic function parallelize
    for chunk in chunks:
        parallelize(count_w_chunks(chunk))
    print (count)<jupyter_output><empty_output><jupyter_text>But the code doesn't work! Race ConditionsLets take a simpler example<jupyter_code>count = 0
def increment(delta):
    global count
    temp = count
    temp += delta
    count = temp
    return count

# what will happen if increment(1) is run parallelly 1000 times ?<jupyter_output><empty_output><jupyter_text>The result will be less than 1000 most of the time.We will need some sort of locking/synchronization mechanism to avoid writing the same value to the variable over and over again.<jupyter_code># this will work, but sucks
count = 0
def increment(delta):
    global count
    with synchronized(count):
        count += delta
    return count<jupyter_output><empty_output><jupyter_text>The culprit in the above code is the assignment. Re-assignment to be precise. Once initialized, a `variable` can have different values at different times, depending on what we are `assigning` to it.If we had code that did not change the `state` of `count` variable, we would'nt need to synchronize the assignments.  Principle 1 Avoid shared state change wherever possible  Functions with Side EffectsWhen a function changes it `state` while it's being executed, and the `state` change is persisted between execution.Put simply but inaccurately, if a function might different values while invoking it with the same parameters NoteFunctions that changes `state` outside it's scope also cause side effects. For example,```codedef foo():    with open('filename', 'a') as f:        f.write('side effect')    return 0```The above function returns 0 all the time, but still changes the `state` of the file system  What is a function? We have functions in Mathematics ```mathf(x) = x^2 + 2x + 3```They do no state manipulation. f(25) is 678 no matter how many times you call it.These kind of functions are called `pure` functions.Read more about pure functions [here](https://en.wikipedia.org/wiki/Pure_function)While you are at it, read about [referential transparency](https://en.wikipedia.org/wiki/Referential_transparency)Its okay if you don't get it.<jupyter_code># Modeling a 'pure' function
def f(x):
    return x*x + 2*x + 3
f(9)<jupyter_output><empty_output><jupyter_text>Say NO to Mutations  Imagine a world without assignments* No mutations no race conditions* Free parallelism* Modular code* Mathematical reasonability* Your IDE can tell you if your code is right  some fun with fun-ctions I do not apologize for my puns<jupyter_code># frange(0,1,0.1) == [0, .1, .2, .3, .4 ... .9]
def frange(a,b,delta):
    i = a
    while i < b:
        yield i
        i += delta
        
# yes there are assignments in the above code, they are avoidable using recursion
# integration; approximation of area under the curve. for more accurate value, reduce the value of err
def integ(f, a,b):
    err = 0.01
    return sum( [f(x) * err for x in frange(a,b,err)]
    )

def square(x):
    return x*x

integ(square, 0, 2)
# functions returning functions

def add_value(value):
    def adder(x):
        return x + value
    return adder
add5 = add_value(5)
add5(22)<jupyter_output><empty_output><jupyter_text>A note about Lambda    lambda lets you write functions similar to mathematical expressions. It saves mind-space in remembering definitions, and sometimes makes statements more readable.  As a rule of thumb, use lambdas if the return value is a simple expression<jupyter_code>def square(x):
    return x*x

# is the same as
lambda x: x*x

# functions taking functions as argument

def fogger(f,g):
    def fog(x):
        return f(g(x))
    return fog
def add6(x):
    return x + 6

new_function = fogger (add6, add5)
new_function(20)<jupyter_output><empty_output><jupyter_text>Map Reduce & Filter<jupyter_code># map(funct, list) --> new list
# applies the funct on each element of the list and returns a new list
def square(x):
    return x*x
map(square, [1,2,3,4,5,6])
# lambdas are useful in one-liners like these
map(lambda x: x*x, [1,2,3,4,5,6])
# reduce(func, list) -- > new value
# reduces a list to a single value by combining elements (left to right) using func
reduce(lambda x,y: x + y , [1,2,3,4,5])
# reduce can also take an inital value to start with
reduce(lambda x,y: x + y , [1,2,3,4,5], 25)
reduce (lambda x, y: x / y, [4,2,1]) # ((4/2) / 1 )

help(reduce)
# filter (predicate, list) --> new list
filter(lambda x: len(x) <= 3,  ["str", "string", "srj", "sreeraj", "fp", "functional programming"])<jupyter_output><empty_output><jupyter_text>Back to our problem[Revisit](A-simple-problem) the question, if you forgotHow can we solve it in a paralell way, while avoiding mutations? Think in terms of data transformation, rather than stepsfile --> [chunk ..] --> [words] --> value<jupyter_code>def count_chunk(chunk):
    ## chunk is made of lines, lines are made of chunks
    words = [word for line in lines for word in line.split()]
    return len(words)

def count(file):
    chunks = make_chunks(file)
    reduce(sum, map(count_chunk, file.make_chunks()))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0914240755310779, 'fraction_numerical': 0.014004720692368215, 'mean_word_length': 4.092948717948718, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '217023', 'license': 'MIT', 'n_tokens_mistral': 1977, 'n_tokens_neox': 1896, 'n_words': 801, 'path': 'WhyFP.ipynb', 'repo_name': 'sreedom/fpExperiments', 'size': '6355'}