<commit_before>### Possible Python Performance examples:
* General ideas:
    - Don't reinvent the wheel by writing your own versions of built-in functions.  
        + Example of using len() function vs writing your own function that counts the number of elements by incrementing a variable.
        + Example of using sum() vs for loop for adding elements
    Python 2.7.10 contains 76 built-in variables [source](https://docs.python.org/2/library/functions.html) - you won't need all of them (I don't remember when was the list time - if ever - I used _basestring()_, _execfile()_ or _reload()_), but it will take you just a moment to take a look at that list from time to time and keep it in the back of your head. If you want some more cool functions, there is the [itertools](https://docs.python.org/release/2.7.2/library/itertools.html) module that contains _fast, memory efficient tools that are useful by themselves or in combination_
    - _It's better to beg for forgiveness than to ask for permission_ give some examples where a code with exceptions handling is faster than code with __if__ statements
* Operators:
    - If checking for True/False the fastest way is __if element__ (vs __if element is True__ or __if element == True__)
* Structures:
    - Lists:
        + Sorting lists is faster if you sort in place (**list.sort() vs sorted(list)**)
        + List comprehension:
            * Example of list comprehension vs for loop (vs lambda) to filter a list of items
            * __in__ operator vs for loop to check if element is in a list
        + There are bad ways to check if a list is empty: **if list == []** or **if len(list) <= 0** but there is also a good way to do that **if not list**
    - Tuples:
    - Sets:
        + Sets vs lists:
            * Lookup time in a set is constant (so it's faster than in a list) but it costs you the overhead to convert list to a set (so in general if you are checking if one element belongs to a collection, use list, if you are checking for multiple elements, convert to a set)
            * Removing duplicates - you get it for free if you convert list to a set (but if the order is important use [OrderedSet](http://code.activestate.com/recipes/576694/))
            * Sets have a lot of useful functions for checking the intersection, union, difference, sub- and super-sets that you don't have in lists: Give a few examples.
    - Dictionaries:
        + Initializing a dictionary: __d = {} vs d = dict()__ (first one is faster, but compare it with different Python compilers)
        + To get an element from the dictionary use __dict.get(key)__ function instead of doing __if key in dict: dict[key]__
        + Example of dictionary comprehension (for example to get a dictionary {number: number**2}) vs for loop
* Strings:
    - example of string formatting vs string concatenation: __"Hello " + s + " !" vs "Hello %s !" % s__
* Lambdas:
    - There is usually no difference between using a lambda and a named function, so often named function is a better solution because it's more readable. Lambdas are nice as a short, simple one-liners that don't require a lot of explanation.
* Generators:
    - Example of finding the first element matching the criteria: comparing list comprehension (it's a totally wrong approach here, since we don't need the whole list) vs for loop (it's faster and seems like a good solution) vs generator (same performance as for loop, but this is a pythonic approach and you can get next elements basically for free)
* Python vs Java mocking:
    - [Reading file in Java](http://stackoverflow.com/questions/4716503/best-way-to-read-a-text-file) vs [Reading file in Python](http://stackoverflow.com/questions/8011797/open-read-and-close-a-file-in-1-line-of-code)


* Sources:
    - [<NAME> talking about how Pythonic > PEP-8 during PyCon 2015](https://www.youtube.com/watch?v=wf-BqAjZb8M)<commit_msg>Added more topic to ALL_topics:<commit_after>### Possible Python Performance improvements and best practices:
* General ideas:
    - Don't reinvent the wheel by writing your own versions of built-in functions.  
        + Example of using len() function vs writing your own function that counts the number of elements by incrementing a variable.
        + Example of using sum() vs for loop for adding elements
    Python 2.7.10 contains 76 built-in variables [source](https://docs.python.org/2/library/functions.html) - you won't need all of them (I don't remember when was the list time - if ever - I used _basestring()_, _execfile()_ or _reload()_), but it will take you just a moment to take a look at that list from time to time and keep it in the back of your head. If you want some more cool functions, there is the [itertools](https://docs.python.org/release/2.7.2/library/itertools.html) module that contains _fast, memory efficient tools that are useful by themselves or in combination_
    - _It's better to beg for forgiveness than to ask for permission_ give some examples where a code with exceptions handling is faster than code with __if__ statements
    - Mutable defaults: don't do stuff like __def foo(element, x=[])__, you will spend a lot of time debugging this error. Use **def foo(element, x=None):;if x in None:; x = []**
* Operators:
    - If checking for True/False the fastest way is __if element__ (vs __if element is True__ or __if element == True__)
* Structures:
    - Lists:
        + Sorting lists is faster if you sort in place (**list.sort() vs sorted(list)**)
        + When using **sorted()** function you can specify **cmp()** function that is used to compare two elements or **key()** (Python 2.4 and up)function that will return a comparison key for each list element. Use the **key()** because it's faster (the **cmp()** is actually removed in Python 3) - cmp has to be evaluated for each comparison, while key only once per element
        + List comprehension:
            * Example of list comprehension vs for loop (vs lambda) to filter a list of items
            * __in__ operator vs for loop to check if element is in a list
        + There are bad ways to check if a list is empty: **if list == []** or **if len(list) <= 0** but there is also a good way to do that **if not list**
    - Tuples:
    - Sets:
        + Sets vs lists:
            * Lookup time in a set is constant (so it's faster than in a list) but it costs you the overhead to convert list to a set (so in general if you are checking if one element belongs to a collection, use list, if you are checking for multiple elements, convert to a set)
            * Removing duplicates - you get it for free if you convert list to a set (but if the order is important use [OrderedSet](http://code.activestate.com/recipes/576694/))
            * Sets have a lot of useful functions for checking the intersection, union, difference, sub- and super-sets that you don't have in lists: Give a few examples.
    - Dictionaries:
        + Initializing a dictionary: __d = {} vs d = dict()__ (first one is faster, but compare it with different Python compilers)
        + To get an element from the dictionary use __dict.get(key)__ function instead of doing __if key in dict: dict[key]__
        + Example of dictionary comprehension (for example to get a dictionary {number: number**2}) vs for loop
* Strings:
    - If you need to create a string from a list of words, it's faster to use **s = "".join(list)** than **for x in list: s += x**. The same if you need to call a function on each element of list: this **slist = [some_function(elt) for elt in somelist]; s = "".join(slist)** is faster than the for loop.[^1]
    - Using string formatting is faster than string concatenation: **out = "<html>%s%s%s%s</html>" % (head, prologue, query, tail)** is better than **out = "<html>" + head + prologue + query + tail + "</html>"** (and using **out = "<html>%(head)s%(prologue)s%(query)s%(tail)s</html>" % locals()** is more clear).[^1]
* Loops:
    - You almost never need to use the **for** loop. Map/filter/reduce (functions are better solution. List comprehensions are the best solution.
    - If the body of a loop is simple, the interpreter overhead for the **for** loop itself can be quite big - in this case you can use **map()**. Map is _a for loop moved into C code_.[^1]
    - It's faster to perform an operation 1000 times inside a function, than calling the function 1000 times, for example:  
        x = 0  
        def do(num):  
        &nbsp;global x  
        &nbsp;x =+ 1  
        for num in range(1000):  
        &nbsp;do(num)  
    is slower than:  
        x = 0  
        def do():  
        &nbsp;global x  
        &nbsp;for num in range(1000):  
        &nbsp;&nbsp;x =+ 1  
        do().[^1]
* Lambdas:
    - There is usually no difference between using a lambda and a named function, so often named function is a better solution because it's more readable. Lambdas are nice as a short, simple one-liners that don't require a lot of explanation.
* Generators:
    - In short, generator expressions are like list comprehension but they don't generate the whole list at once (so they are faster if you don't need the whole list).
    - Example of finding the first element matching the criteria: comparing list comprehension (it's a totally wrong approach here, since we don't need the whole list) vs for loop (it's faster and seems like a good solution) vs generator (same performance as for loop, but this is a pythonic approach and you can get next elements basically for free)
* Functions:
    - each time you call function with dots inside (like **newlist.append** or **str.upper**, the function's reference is reevaluated. You can assign it to a variable like **append = newlist.append** and then use **append(element)** for a performance gain and a readability lost.[^1]
    - **range** vs **xrange**: in most cases xrange will have a better performance than range. However there are some situations where you actually want to use **range**:
        + when you actually need to initialize a list (for example to slice it), you would use **range**.
        + if you will iterate over the sequence multiple times, **range** will be faster, because **xrange** need to reconstruct the list every time.
        + **xrange** doesn't exist in Python 3. Remember to change it when porting (this can be easily done with automatic tools like **2to3**).[^2]
* Scope:
    - Python can access local variables faster than global variables, so if inside a function you assign **upper = str.upper** it will be faster than calling **str.upper()** but less readable.[^1]
    - Modules and namespaces:
        + If you have some modules that you will use only in some specific functions, it's faster to import than module inside function definition than in the global namespace (at the beginning of the file) - but you will lose some readability (it won't be obvious what modules are imported).[^1]
* Comparing with other languages:
    - Compare the 3 following ways to double the value of x: **x * 2**, **x << 1** and **x + x**. They perform the same operation, but the last version is faster (1,5 times faster) than the previous two. However, in C all those 3 operations would be translated to the same machine instruction- thus the execution time would be almost the same.
    - Python vs Java mocking:
        + [Reading file in Java](http://stackoverflow.com/questions/4716503/best-way-to-read-a-text-file) vs [Reading file in Python](http://stackoverflow.com/questions/8011797/open-read-and-close-a-file-in-1-line-of-code)
* Advanced profiling:
    - Always use profiling tools to find bottlenecks of your application. There is no point for you to try to optimize those parts of code that are already running fast or are called very rarely. Use module like **cProfile** (that replaced **hotshot**, that replaced **profile**) or **trace**.

* Sources:
    - [<NAME> talking about how Pythonic > PEP-8 during PyCon 2015](https://www.youtube.com/watch?v=wf-BqAjZb8M)

[^1]: https://wiki.python.org/moin/PythonSpeed/PerformanceTips
[^2]: http://stackoverflow.com/questions/135041/should-you-always-favor-xrange-over-range
{'fraction_non_alphanumeric': 0.0864410965670536, 'fraction_numerical': 0.010372931588046431, 'mean_word_length': 3.4563462949376373, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 13, 'https://': 7, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2428435', 'n_tokens_mistral': 3329, 'n_tokens_neox': 3160, 'n_words': 1718}