<jupyter_start><jupyter_text>Profiling PythonFamously, premature optimization has been called the source of all evil, at least when it comes to computer programming. The key part of this quote is the word premature. In order to optimize your code, you need to know what parts need optimization. This is where profiling comes in. With profiling, you can figure out just how much time is being used in various parts of your code, as well as how much memory is being used. With this information, you will be armed to figure what where to focus your energy to get the largest improvement. In this workshop, I will be assuming that you have some Python experience and ready to try and get the most out the code you have written.  Time ProfilingThe first step in getting better, faster code is to do some profiling to see where your time is being spent. The three broad categories of profiling are* course time measurements* function profiling* line profiling  Course Time MeasurementsThe most basic form of profiling is just seeing how long different chunks of code take to run. You can do this most simply by measuring the start and end times and finding the difference.<jupyter_code>import time
start = time.clock()
for a in range(1000000):
    b = a ^ a
end = time.clock()
print(end-start)<jupyter_output>0.38340199661512153<jupyter_text>A more organized way would be to have a timer object that you can easily reuse rather than sticking a number of start-end time commands. An example class might look like the code below (which was blatantly borrowed from the website https://www.huyng.com/posts/python-performance-analysis<jupyter_code>class Timer(object):
    def __init__(self, verbose=False):
        self.verbose = verbose

    def __enter__(self):
        self.start = time.time()
        return self

    def __exit__(self, *args):
        self.end = time.time()
        self.secs = self.end - self.start
        self.msecs = self.secs * 1000  # millisecs
        if self.verbose:
            print('elapsed time: %f ms' % self.msecs)<jupyter_output><empty_output><jupyter_text>You can then use it with the 'with' statement to time chunks of code.<jupyter_code>with Timer() as t:
    for a in range(1000000):
        b = a^a
print("=> elasped 1 million: %s s" % t.secs)

with Timer() as t:
    for a in range(10000000):
        b = a^a
print("=> elasped 10 million: %s s" % t.secs)<jupyter_output>=> elasped 1 million: 0.42665958404541016 s
=> elasped 10 million: 3.5925371646881104 s<jupyter_text>If you use the IPython shell, there are special functions called magics that let you interact with the Python interpreter. Two of these magics are %timeit and %%timeit. The first one times how long a single statement takes to run.<jupyter_code>%timeit range(1000000)
%%timeit
a = range(1000000)<jupyter_output>1000000 loops, best of 3: 1.07 µs per loop<jupyter_text>Function ProfilingWhile this might be good enough for simple code, it quickly becomes too coarse to do any real analysis with. Luckily, Python includes two profiling modules as part of a standard installation. The first a pure Python module named 'profile'. Because it is pure Python, it introduces quite a lot overhead. It is ideal if you want to alter the behavior of the profiler, however. The second profiler is named 'cProfile', which is a C implementation of the same behavior as 'profile'. It is therefore much faster.The easiest way to use it is to use the run method. We will start by creating our load function to make the code easier to read.<jupyter_code>def myload(num):
    for a in range(num):
        b = a^a<jupyter_output><empty_output><jupyter_text>Now we can profile it and see what happens.<jupyter_code>import cProfile

cProfile.run('myload(1000000)')<jupyter_output>4 function calls in 0.240 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.240    0.240    0.240    0.240 <ipython-input-5-421e6c049a13>:1(myload)
        1    0.000    0.000    0.240    0.240 <string>:1(<module>)
        1    0.000    0.000    0.240    0.240 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}<jupyter_text>As you can see, cProfile gives a breakdown of each function call, how many times it gets called, and various measures of the amount of time used. This is a good starting point for larger programs. The major issue with the run method is that you can only use a string that can be run with the exec function. You can also use cProfile directly (like the timer class above) to collect stats for sections of code.  Line ProfilerIf you need more detail, you can use another module named line_profiler. This one usually is not included in most Python installations. You will likely need to install it with* pip install line_profilerIt comes with a command line utility called kernprof that you can use to profile your scripts. You need to add a decorator to any functions you want to profile like<jupyter_code>@profile
myload(1000000)<jupyter_output><empty_output><jupyter_text>Then run it with* kernprof -l myscript.pyThis generates a binary output file with the profiling information. You can process the results with* python -m line_profiler myscript.py.lprofIf you want to use line_profiler directly within your script, you can import the LineProfiler class and use it to profile statements in your script.<jupyter_code>import line_profiler

profiler = line_profiler.LineProfiler()
profiler.add_function(myload(1000000))
profiler.print_stats()<jupyter_output>Timer unit: 4.72616e-07 s<jupyter_text>The above warning is because we are running this through Jupyter. If you were to run this as a regular script in IPython, it won't occur. It will also give you the line-by-line profiling information that looks likeFile: test.pyFunction: get_number at line 43Total time: 4.44195 sLine       Hits         Time  Per Hit   % Time  Line Contents==============================================================    43                                           def get_number():    44   5000001      2223313      0.4     50.1      for x in xrange(5000000):    45   5000000      2218638      0.4     49.9          yield xYou can also use it in a fashion similar to cProfile, using the run() method.<jupyter_code>profiler.run('myload(1000000)')
profiler.print_stats()<jupyter_output>Timer unit: 4.72616e-07 s<jupyter_text>Memory ProfilingThe other resource that you may need to optimize for is memory. Similarly to line_profilr is a module named memory_profiler. You can import is with* pip install memory_profilerThe most basic way to use it is again similar to line_profiler. It includes a decorator named @profile which you can add to your script to profile the functions in question. You can then run it with* python -m memory_profiler myscript.pySimilarly to line_profiler, memory_profiler includes a command line utility called mprof. You can run it with* mprof run myscript.py* mprof listIt can even generate a plot of memory usage with* mprof plotIn an interactive way, you can use memory_profiler directly in a few different ways. The easiest is similar to the coarse timing examples above.<jupyter_code>import memory_profiler

usage = memory_profiler.memory_usage()
print(usage)
b = list()
for a in range(10000000):
    b.append(a)
usage = memory_profiler.memory_usage()
print(usage)<jupyter_output>[34.93359375]
[421.328125]<jupyter_text>You can analyze the memory usage for a given function using the same function.<jupyter_code>def myload2(a):
    b = list()
    for c in range(a):
        b.append(c)

usage = memory_profiler.memory_usage((myload2,(1000000,)))
print(usage)<jupyter_output>[421.3671875, 421.4375, 434.71875, 448.64453125, 456.4296875, 421.6328125]<jupyter_text>While the above gives you information about raw memory usage, the issue in your code might be an inefficient use of memory by having more objects around than are strictly needed. You can use objgraph to get access to what is taking up chunks of memory. You can install it with* pip install objgraphThen you can start with a report of the most common objects in memory.<jupyter_code>import objgraph

objgraph.show_most_common_types()
print('Show growth of objects')
objgraph.show_growth()
myload2(1000000)
objgraph.show_growth()<jupyter_output>function                   7610
dict                       4990
tuple                      3633
weakref                    1891
list                       1369
wrapper_descriptor         1195
getset_descriptor          1060
builtin_function_or_method 961
method_descriptor          910
type                       890
Show growth of objects
function                       7608     +7608
dict                           4819     +4819
tuple                          3228     +3228
weakref                        1892     +1892
list                           1332     +1332
wrapper_descriptor             1195     +1195
getset_descriptor              1060     +1060
builtin_function_or_method      960      +960
method_descriptor               910      +910
type                            890      +890
Interactive        1        +1<jupyter_text>Optimizing PythonNow that you have some idea of what parts of your code are most problematic, you can now start to look at what you could do to get better performance out of it. One of the major efficiency problems with Python is also one of its greatest powers, the combination of untyped variables and object oriented programming. Because variables are untyped, Python ends up having to check what object it refers to each time it is ever accessed. Many of the optimization ideas are based on minimizing this issue. The following is simply a list of possible ideas in no particular order.  Use keys and sortSorting is something that can easily be done inefficiently. Luckily, Python has an extremely efficient sort method in lists.<jupyter_code>a = [1,74,5,2,8,3,9,10,2]
%timeit a.sort()
a<jupyter_output>The slowest run took 6.98 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 812 ns per loop<jupyter_text>As you can see, the sort is done in place. You can sort more complicated lists using keys. Say you had a list of tuples; you can sort based any of the values contained.<jupyter_code>import operator
b = [(1,2,3),(9,1,7),(3,4,5)]
%timeit b.sort(key=operator.itemgetter(0))
print(b)
%timeit b.sort(key=operator.itemgetter(1))
print(b)<jupyter_output>100000 loops, best of 3: 4.17 µs per loop
[(1, 2, 3), (3, 4, 5), (9, 1, 7)]
100000 loops, best of 3: 4.15 µs per loop
[(9, 1, 7), (1, 2, 3), (3, 4, 5)]<jupyter_text>Avoiding dotsWhenever you use object methods or module functions and give the full path to them, Python needs to resolve and check every step along the way. This happens everytime it is referenced. For example, making a list of words upper case involves:<jupyter_code>%%timeit
lowercase = ['this', 'is', 'a', 'lower', 'case', 'string']
uppercase = []
for word in lowercase:
    uppercase.append(str.upper(word))<jupyter_output>100000 loops, best of 3: 7.53 µs per loop<jupyter_text>You can speed up the references to str.upper and uppercase.append, as below:<jupyter_code>upper = str.upper
uppercase = []
append = uppercase.append
%%timeit
lowercase = ['this', 'is', 'a', 'lower', 'case', 'string']
for word in lowercase:
    append(upper(word))<jupyter_output>100000 loops, best of 3: 5.37 µs per loop<jupyter_text>Using different coding techniquesSometimes, using a different technique to get the same result will alter the amount of time used. Say you were creating a dictionary of characters and how many times they were used. As an example, we'll use the string 'abcd' and simply cycle over them. The first way to create the dictionary would be to create an empty one, and add new entries if the character isn't already being counted.<jupyter_code>%%timeit
n = 16
myDict = {}
for i in range(0, n):
    char = 'abcd'[i%4]
    if char not in myDict:
        myDict[char] = 0
    myDict[char] += 1<jupyter_output>100000 loops, best of 3: 14.7 µs per loop<jupyter_text>A different technique would be to use a try/except block instead.<jupyter_code>%%timeit
n = 16
myDict = {}
for i in range(0, n):
    char = 'abcd'[i%4]
    try:
        myDict[char] += 1
    except KeyError:
        myDict[char] = 1<jupyter_output>10000 loops, best of 3: 19.2 µs per loop<jupyter_text>For small data, the first is slightly faster. For larger data, the reverse is true.<jupyter_code>%%timeit
n = 1600
myDict = {}
for i in range(0, n):
    char = 'abcde'[i%5]
    if char not in myDict:
        myDict[char] = 0
    myDict[char] += 1
%%timeit
n = 1600
myDict = {}
for i in range(0, n):
    char = 'abcde'[i%5]
    try:
        myDict[char] += 1
    except KeyError:
        myDict[char] = 1<jupyter_output>1000 loops, best of 3: 1.08 ms per loop<jupyter_text>List ComprehensionsWhen you need to create lists of values, you can do the normal thing of using a for loop.<jupyter_code>%%timeit
a = []
for b in range(1000):
    a.append(b**2)<jupyter_output>1000 loops, best of 3: 1.19 ms per loop<jupyter_text>You can use comprehensions, instead, where you define the creation rule within the list definition. This may be faster in some cases, as below.<jupyter_code>%%timeit
a = [x**2 for x in range(1000)]<jupyter_output>1000 loops, best of 3: 1 ms per loop<jupyter_text>You can make very complex creation rules, as long as they can be defined with if and for statements.  Use NumpyIn scientific computations, numerical work is the bulk of what your code needs to handle. You can speed these types of operations up by having external libraries written in C do all of the heavy lifting. The main library is numpy. As an example, let's say we want to scale a matrix.<jupyter_code>%%timeit
w = 100
h = 100
matrix1 = [[1 for x in range(w)] for y in range(h)]
for x in range(h):
    for y in range(w):
        matrix1[x][y] = 5 * matrix1[x][y]
import numpy
%%timeit
w = 100
h = 100
matrix1 = numpy.ones((w, h))
matrix1 = 5 * matrix1<jupyter_output>The slowest run took 1388.21 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 3: 54.7 µs per loop<jupyter_text>Replace for with mapThe map function takes the explicit looping of a for loop and turns it into an implicit looping. This moves the bulk of the loop to the interpreter level.<jupyter_code>%%timeit
uppercase = []
lowercase = ['this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string']
for word in lowercase:
    uppercase.append(str.upper(word))
%%timeit
lowercase = ['this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string']
uppercase = map(str.upper, lowercase)<jupyter_output>1000000 loops, best of 3: 1.69 µs per loop<jupyter_text>Moving to local variablesIn cases where you can't use map, you can move your for loop to a function so that all of the intermediate variables are local. Local variables are accessed much more efficiently than global variables.<jupyter_code>def myupper(lowercase):
    upper = str.upper
    newlist = []
    append = newlist.append
    for word in lowercase:
        append(upper(word))
    return newlist
%%timeit
uppercase = []
lowercase = ['this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string']
for word in lowercase:
    uppercase.append(str.upper(word))
%%timeit
lowercase = ['this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string','this', 'is', 'a', 'lower', 'case', 'string']
uppercase = myupper(lowercase)<jupyter_output>10000 loops, best of 3: 17 µs per loop<jupyter_text>Minimizing function callsOne of the more expensive operations in Python is the function call. The example below is a function that is too general.<jupyter_code>x = 0
def doit1(i):
    global x
    x = x + i

list = range(100000)
%%timeit
for i in list:
    doit1(i)<jupyter_output>10 loops, best of 3: 59.9 ms per loop<jupyter_text>Moving the for loop into the function, and aggregating your data into a list, gives you:<jupyter_code>x = 0
def doit2(list):
    global x
    for i in list:
        x = x + i

list = range(100000)
%%timeit
doit2(list)<jupyter_output>10 loops, best of 3: 30.3 ms per loop<jupyter_text>Don't import ideas from other languagesThere are tricks that work well in other languages that do not port to Python. An example is doubling a value, which can be done with* a * 2* a<<1* a + aIn C, the middle one works fastest. What about Python?<jupyter_code>%%timeit
x=42
y = 2*x
%%timeit
x=42
y = x<<1
%%timeit
x=42
y = x + x<jupyter_output>The slowest run took 10.09 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 140 ns per loop<jupyter_text>The shift method actually is the worst version in Python.  CythonYou can move particularly problematic code into its own external C module of code. Cython lets you import the compiled C code back into your Python program. Scientific distributions, such as Anaconda, have it included.You can use it with IPython notebooks with<jupyter_code>%load_ext Cython<jupyter_output><empty_output><jupyter_text>The easiest way to take advantage of it is to use statically typed variables. This way, Python knows what the variables refer to and doesn't need to check.<jupyter_code>def sum1():
    a = 0
    for i in range(1000000):
        a += i
%timeit sum1
%%cython
def sum2():
    cdef int a = 0
    for i in range(10):
        a += i
%timeit sum2<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08686452475082132, 'fraction_numerical': 0.045102734005234145, 'mean_word_length': 3.888405008165487, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 100, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '344195', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 5889, 'n_tokens_neox': 5314, 'n_words': 2361, 'path': 'code/profiling-and-optimizing.ipynb', 'repo_name': 'bagustris/tutorial-python', 'size': '17959'}